lfs: using workers in lfs prefetch This significantly speeds up lfs prefetch. With fast network we are seeing ~50% improvement of overall prefetch times Because of worker's API in posix we do lose finegrained progress update and only see progress when a file finished downloading. Test Plan: Run tests: ./run-tests.py -l test-lfs* .... # Ran 4 tests, 0 skipped, 0 failed. Run commands resulting in lfs prefetch e.g. hg sparse --enable-profile Differential Revision: https://phab.mercurial-scm.org/D1568

worker: make windows workers daemons The windows workers weren't daemons and were not correctly killed when ctrl-c'd from the terminal. Withi this change when the main thread is killed, all daemons get killed as well. I also reduced the time we give to workers to cleanup nicely to not have people ctrl-c'ing when they get inpatient. The output when threads clened up nicely: PS C:\<dir>> hg.exe sparse --disable-profile SparseProfiles/<profile>.sparse interrupted! The output when threads don't clenup in 1 sec: PS C:\<dir> hg.exe sparse --enable-profile SparseProfiles/<profile>.sparse failed to kill worker threads while handling an exception interrupted! Exception in thread Thread-4 (most likely raised during interpreter shutdown): PS C:\<dir>> Test Plan: Run hg command on windows (pull/update/sparse). Ctrl-C'd sparse --enable-profile command that was using threads and observed in proces explorer that all threads got killed. ran tests on CentOS Differential Revision: https://phab.mercurial-scm.org/D1564

workers: add config to enable/diable workers This adds config to disable/enable workers with default being enabled. Test Plan: enabled profile without updaing .hg/hgrc (the default should be to use workers) and ran hg sprase --enable-profile <profile>.sparse Watched in the proces explorer that hg started 12 new threads for materializing files (this is my worker.numcpus) value Added [worker] enabled = False to the .hg/hgrc and re ran the command. This time hg didn't spawn any new threads for matreializing of files Differential Revision: https://phab.mercurial-scm.org/D1460

workers: handling exceptions in windows workers This adds handling of exceptions from worker threads and resurfaces them as if the function ran without workers. If any of the threads throws, the main thread kills all running threads giving them 5 sec to handle the interruption and raises the first exception received. We don't have to join threads if is_alive() is false Test Plan: Ran multiple updates/enable/disable sparse profile and things worked well Ran test on CentOS- all tests passing on @ passed here Added a forged exception into the worker code and got it properly resurfaced and the rest of workers killed: P58642088 PS C:\open\<repo>> ..\facebook-hg-rpms\build\hg\hg.exe --config extensions.fsmonitor=! sparse --enable-profile <profile> updating [==> ] 1300/39166 1m57sException in thread Thread-3: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception Exception in thread Thread-2: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception <...> Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hgexe.py", line 41, in <module> dispatch.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 85, in run status = (dispatch(req) or 0) & 255 File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 173, in dispatch ret = _runcatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 324, in _runcatch return _callcatch(ui, _runcatchfunc) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 332, in _callcatch return scmutil.callcatch(ui, func) File "C:\open\facebook-hg-rpms\build\hg\mercurial\scmutil.py", line 154, in callcatch return func() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 314, in _runcatchfunc return _dispatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 951, in _dispatch cmdpats, cmdoptions) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 415, in runcommand return orig(lui, repo, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\undo.py", line 118, in _runcommandwrapper result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hgext\journal.py", line 84, in runcommand return orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 268, in _tracksparseprofiles res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 256, in _trackdirstatesizes res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\copytrace.py", line 144, in _runcommand return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbamend\hiddenoverride.py", line 119, in runcommand result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 712, in runcommand ret = _runcommand(ui, options, cmd, d) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 959, in _runcommand return cmdfunc() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 948, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "C:\open\facebook-hg-rpms\build\hg\mercurial\util.py", line 1183, in check return func(*args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 860, in sparse disableprofile=disableprofile, force=force) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 949, in _config len, _refresh(ui, repo, oldstatus, oldsparsematch, force)) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 1116, in _refresh mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 311, in applyupdates return orig(repo, actions, wctx, mctx, overwrite, labels=labels) File "C:\open\facebook-hg-rpms\build\hg\mercurial\merge.py", line 1464, in applyupdates for i, item in prog: File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 286, in _windowsworker raise t.exception Exception: Forged exception PS C:\open\ovrsource> Differential Revision: https://phab.mercurial-scm.org/D1459

workers: implemented worker on windows This change implements thread based worker on windows. The handling of exception from within threads will happen in separate diff. The worker is for now used in mercurial/merge.py and in lfs extension After multiple tests and milions of files materiealized, thousands lfs fetched it seems that neither merge.py nor lfs/blobstore.py is thread unsafe. I also looked through the code and besides the backgroundfilecloser (handled in base of this) things look good. The performance boost of this on windows is ~50% for sparse --enable-profile * Speedup of hg up/rebase - not exactly measured Test Plan: Ran 10s of hg sparse --enable-profile and --disable-profile operations on large profiles and verified that workers are running. Used sysinternals suite to see that all threads are spawned and run as they should Run various other operations on the repo including update and rebase Ran tests on CentOS and all tests that pass on @ pass here Differential Revision: https://phab.mercurial-scm.org/D1458

workers: don't use backgroundfilecloser in threads This disables background file closing when in not in main thread Test Plan: Ran pull, update, sparse commands and watched the closer threads created and destroyed in procexp.exe ran test on CentOS. No tests broken compared to the base Differential Revision: https://phab.mercurial-scm.org/D1457

debugdiscovery: correct and clean up command synopsis This does a few things: * Changes "-r" to "--rev", since "-r" is not a valid short form * Removes non-existent "-l" and "-b" options * Removes "..." after options, since we don't usually have that Differential Revision: https://phab.mercurial-scm.org/D1706

copies: group wdir-handling in one place I think this makes it both easier to follow and shorter. Differential Revision: https://phab.mercurial-scm.org/D1698

copies: extract method for getting non-wdir forward copies I may add an alternative way of getting copy metadata (from changelog, not filelog) but the chaining with the dirstate copy metadata will be the same, so it will probably help to have this extracted. Even if that doesn't happen, the next patch will show that we can simplify this a bit after this refactoring, so it seems worth it regardless. Differential Revision: https://phab.mercurial-scm.org/D1697

copies: consistently use """ for docstrings Differential Revision: https://phab.mercurial-scm.org/D1696

copies: always respect matcher arg to _forwardcopies() The function would ignore the matcher if the dirstate copies were requested. It doesn't matter in practice because all callers used the returned map only for looking up specific files from and those files had already been filtered by the matcher (AFACT). Still, it's a little confusing, so let's make it clearer by respecting the matcher in this case too. Differential Revision: https://phab.mercurial-scm.org/D1695

debugdiscovery: drop reference to non-existent --remote-head option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1693

debugdiscovery: drop reference to non-existent --serverlog option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1692

debugdiscovery: drop reference to invalid --branch option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1691

tests: avoid echo with backslash escapes Some shells' echo (e.g. Bash's) do not interpret backslash escapes by default, while others shells' do (e.g. Dash's). This led to a failure in test-run-tests.t. We could switch to the more compatible printf, but using a heredoc seems simpler, so that's what this patch does. For anyone who wants to test this for themselves, remove the "unset HGTEST_SHELL" line in helper-runtests.sh and pass e.g. "--shell=/bin/dash" to the test runner. Differential Revision: https://phab.mercurial-scm.org/D1699