worker: handle interrupt on windows After applying suggestions from https://phab.mercurial-scm.org/D1564 to catch all exceptions in the same way I actually broke the handling of KeyboardInterrupt on windows. The reason is that KeyboardInterrupt doesn't dervie from Exception, but BaseException: https://docs.python.org/2/library/exceptions.html starting from python 2.5 Test Plan: Run hg on windows and ctrl-c during a large update. No random exceptions from threads surface in the shell. Previously we'd nearly always get stack traces from some of threads Run tests ./run-tests.py [...] Failed test-convert-svn-encoding.t: output changed # Ran 622 tests, 41 skipped, 1 failed. python hash seed: 2962682116 The test failing seems to have nothing to do with the change and fails on base revision as well Differential Revision: https://phab.mercurial-scm.org/D1718

lfs: fix committing deleted files caused by e0a1b9ee93cd e0a1b9ee93cd (lfs: add a repo requirement for this extension once an lfs file is committed) introduced a regression that prevents committing file deletion. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D1717

tests: update expected output of svn encoding test The changes I see on the buildbot match the ones I see on my laptop, and all look reasonable. Differential Revision: https://phab.mercurial-scm.org/D1713

run-tests: use context managers for file descriptors I've seen the following error a few times recently when running the tests with `yes | ./run-tests.py --local -j9 -i`: Errored test-add.t: Traceback (most recent call last): File "./run-tests.py", line 821, in run self.runTest() File "./run-tests.py", line 910, in runTest if self._result.addOutputMismatch(self, ret, out, self._refout): File "./run-tests.py", line 1774, in addOutputMismatch rename(test.errpath, test.path) File "./run-tests.py", line 571, in rename os.remove(src) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\Users\\Matt\\projects\\hg\\tests\\test-add.t.err' This change doesn't fix the problem, but it seems like a simple enough improvement.

run-tests: add substitution patterns for common '\' path output on Windows The goal is to reduce the amount of hand tuning of new/changed tests that is required on Windows. Since the OS prints the proper paths everywhere else, this is limited to Windows. These are based on the check-code rules that were dropped in 5feb782c7a95. There are some minor tweaks, because those were trying to detect '/' paths without a '(glob)' at the end, whereas these detect '\' paths. Also, it looks like the 'no changes made to subrepo' one was broke, because the path to the subrepo has been getting output but was not in the pattern. End anchors are dropped because '(glob)' is no longer required, but '(feature !)' annotations are a possibility. The 'saved backup bundle' pattern dropped from run-tests.py was simply carrying over the first capture group. The replace() method runs prior to evaluating '\1', but it wasn't doing anything because of the 'r' prefix on '\\'. The 'not recording move' entry is new, because I stumbled upon it searching for some of these patterns. There are probably others.

debuginstall: add a line about re2 availability Using re2 engine can massively speed up regexp. We make it simpler to check if it is available in a given install.

tests: fix the check-code rule for testing non-existent files I missed this in feecfefeba25.

tests: convert the 'file://\$TESTTMP' rule to an automatic substitution The rule only triggered on non Windows platforms, even though Windows also required an adjustment. Automatic seems better. The aggressive globbing in test-subrepo-svn.t was found and rewritten by the substitution.

help: deprecate ui.slash in favor of slashpath template filter (issue5572) > For some reason, I thought someone (Mads?) said we had basically given > up on ui.slash, and some commands didn't support it. (from https://bz.mercurial-scm.org/show_bug.cgi?id=5572#c1) So the ui.slash option doesn't always work and is somewhat confusing. Let's make it clearer we won't improve the situation.

templatefilters: add slashpath() to convert path separator to slash Prepares for deprecating the ui.slash option, which isn't always respected.

check-code: remove unused variable 'winglobmsg' Follows up 5feb782c7a95.

phases: initialize number of loaded revisions to 0 As it isn't a revision number, an empty value should be 0, not -1.

phases: rename _phasemaxrev to _loadedrevslen to clarify it isn't max value "maxrev" sounds like max(0:tip), but it is actually len(0:tip).

lfs: add an experimental config to override User-Agent for the blob transfer This will allow developers to test against various server implementations. I didn't put it under [devel] because it's possible that some user needs to use it in the field.

lfs: add git to the User-Agent header for blob transfers As we were trying to transition off of the non production lfs-test-server for further experimenting, one of the problems we ran into was interoperability. A coworker setup gitbucket[1] to act as the blob server, tested with git, and passed it off to me. But push failed with a message saying "abort: LFS server returns invalid JSON:", and then proceeded to dump a huge HTML page to the screen. It turns out that it is assuming that git is the only thing that wants to do a blob transfer, and everything else is a web browser wanting HTML. It's only a single data point, but I suspect other things may be doing this too. RFC7231 gives an example [2] of listing multiple products in decreasing order of significance. Since the standard provides for this, and since it works with the one problematic server I found, I'm just enabling this by default for a better UX. There's nothing significant about the version of git chosen, other than it is the current version. [1] https://github.com/gitbucket/gitbucket/ [2] https://tools.ietf.org/html/rfc7231#page-46

outgoing: respect ":pushurl" paths (issue5365) Make 'hg outgoing' respect "paths.default:pushurl" in addition to "paths.default-push". 'hg outgoing' has always meant "what will happen if I run 'hg push'?" and it's still documented that way: Show changesets not found in the specified destination repository or the default push location. These are the changesets that would be pushed if a push was requested. If the user uses the now-deprecated "paths.default-push" path, it continues to work that way. However, as described at https://bz.mercurial-scm.org/show_bug.cgi?id=5365, it doesn't behave the same with "paths.default:pushurl". Why does it matter? Similar to the bugzilla reporter, I have a read-only mirror of a non-Mercurial repository: upstream -> imported mirror -> user clone ^-----------------------/ Users push directly to upstream, and that content is then imported into the mirror. However, those repositories are not the same; it's possible that the mirroring has either broken completely, or an import process is running and not yet complete. In those cases, 'hg outgoing' will list changesets that have already been pushed. Mozilla's desired behavior described in bug 5365 can be accomplished through other means (e.g. 'hg outgoing default'), preserving the consistency and meaning of 'hg outgoing'.

tests: test-pathconflicts-merge.t requires symlinks Once we're ready to turn this functionality on more widely, we might want to write a symlink-free version of the test that can run on Windows, but for now we'll just leave it disabled there. Differential Revision: https://phab.mercurial-scm.org/D1710

sshpeer: allow for additional environment passing to ssh exe We already have the ability to customize the ssh command line arguments, let's add the ability to customize its environment as well. Example use-case is ssh.exe from Git on Windows. If `HOME` enviroment variable is present and has some non-empty value, ssh.exe will try to access that location for some stuff (for example, it seems for resolving `~` in `.ssh/config`). Git for Windows seems to sometimess set this variable to the value of `/home/username` which probably works under Git Bash, but does not work in a native `cmd.exe` or `powershell`. Whatever the root cause, setting `HOME` to be an empty string heals things. Therefore, some distributors might want to set `sshenv.HOME=` in the configuration (seems less intrusive that forcing everyone to tweak their env). Test Plan: - rt Differential Revision: https://phab.mercurial-scm.org/D1683

unamend: allow unamending if allowunstable is set I don't see why unamend should be disallowed when allowunstable is set. By switching to rewriteutil.precheck() we fix that and get more consistent error messages (and some additional ones). Differential Revision: https://phab.mercurial-scm.org/D1682

rebase: add ui.log calls for whether IMM used, whether rebasing WCP Make it a bit easy to get metrics from these. Differential Revision: https://phab.mercurial-scm.org/D1681

lfs: using workers in lfs prefetch This significantly speeds up lfs prefetch. With fast network we are seeing ~50% improvement of overall prefetch times Because of worker's API in posix we do lose finegrained progress update and only see progress when a file finished downloading. Test Plan: Run tests: ./run-tests.py -l test-lfs* .... # Ran 4 tests, 0 skipped, 0 failed. Run commands resulting in lfs prefetch e.g. hg sparse --enable-profile Differential Revision: https://phab.mercurial-scm.org/D1568

worker: make windows workers daemons The windows workers weren't daemons and were not correctly killed when ctrl-c'd from the terminal. Withi this change when the main thread is killed, all daemons get killed as well. I also reduced the time we give to workers to cleanup nicely to not have people ctrl-c'ing when they get inpatient. The output when threads clened up nicely: PS C:\<dir>> hg.exe sparse --disable-profile SparseProfiles/<profile>.sparse interrupted! The output when threads don't clenup in 1 sec: PS C:\<dir> hg.exe sparse --enable-profile SparseProfiles/<profile>.sparse failed to kill worker threads while handling an exception interrupted! Exception in thread Thread-4 (most likely raised during interpreter shutdown): PS C:\<dir>> Test Plan: Run hg command on windows (pull/update/sparse). Ctrl-C'd sparse --enable-profile command that was using threads and observed in proces explorer that all threads got killed. ran tests on CentOS Differential Revision: https://phab.mercurial-scm.org/D1564

workers: add config to enable/diable workers This adds config to disable/enable workers with default being enabled. Test Plan: enabled profile without updaing .hg/hgrc (the default should be to use workers) and ran hg sprase --enable-profile <profile>.sparse Watched in the proces explorer that hg started 12 new threads for materializing files (this is my worker.numcpus) value Added [worker] enabled = False to the .hg/hgrc and re ran the command. This time hg didn't spawn any new threads for matreializing of files Differential Revision: https://phab.mercurial-scm.org/D1460

workers: handling exceptions in windows workers This adds handling of exceptions from worker threads and resurfaces them as if the function ran without workers. If any of the threads throws, the main thread kills all running threads giving them 5 sec to handle the interruption and raises the first exception received. We don't have to join threads if is_alive() is false Test Plan: Ran multiple updates/enable/disable sparse profile and things worked well Ran test on CentOS- all tests passing on @ passed here Added a forged exception into the worker code and got it properly resurfaced and the rest of workers killed: P58642088 PS C:\open\<repo>> ..\facebook-hg-rpms\build\hg\hg.exe --config extensions.fsmonitor=! sparse --enable-profile <profile> updating [==> ] 1300/39166 1m57sException in thread Thread-3: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception Exception in thread Thread-2: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception <...> Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hgexe.py", line 41, in <module> dispatch.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 85, in run status = (dispatch(req) or 0) & 255 File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 173, in dispatch ret = _runcatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 324, in _runcatch return _callcatch(ui, _runcatchfunc) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 332, in _callcatch return scmutil.callcatch(ui, func) File "C:\open\facebook-hg-rpms\build\hg\mercurial\scmutil.py", line 154, in callcatch return func() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 314, in _runcatchfunc return _dispatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 951, in _dispatch cmdpats, cmdoptions) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 415, in runcommand return orig(lui, repo, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\undo.py", line 118, in _runcommandwrapper result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hgext\journal.py", line 84, in runcommand return orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 268, in _tracksparseprofiles res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 256, in _trackdirstatesizes res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\copytrace.py", line 144, in _runcommand return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbamend\hiddenoverride.py", line 119, in runcommand result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 712, in runcommand ret = _runcommand(ui, options, cmd, d) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 959, in _runcommand return cmdfunc() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 948, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "C:\open\facebook-hg-rpms\build\hg\mercurial\util.py", line 1183, in check return func(*args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 860, in sparse disableprofile=disableprofile, force=force) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 949, in _config len, _refresh(ui, repo, oldstatus, oldsparsematch, force)) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 1116, in _refresh mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 311, in applyupdates return orig(repo, actions, wctx, mctx, overwrite, labels=labels) File "C:\open\facebook-hg-rpms\build\hg\mercurial\merge.py", line 1464, in applyupdates for i, item in prog: File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 286, in _windowsworker raise t.exception Exception: Forged exception PS C:\open\ovrsource> Differential Revision: https://phab.mercurial-scm.org/D1459

workers: implemented worker on windows This change implements thread based worker on windows. The handling of exception from within threads will happen in separate diff. The worker is for now used in mercurial/merge.py and in lfs extension After multiple tests and milions of files materiealized, thousands lfs fetched it seems that neither merge.py nor lfs/blobstore.py is thread unsafe. I also looked through the code and besides the backgroundfilecloser (handled in base of this) things look good. The performance boost of this on windows is ~50% for sparse --enable-profile * Speedup of hg up/rebase - not exactly measured Test Plan: Ran 10s of hg sparse --enable-profile and --disable-profile operations on large profiles and verified that workers are running. Used sysinternals suite to see that all threads are spawned and run as they should Run various other operations on the repo including update and rebase Ran tests on CentOS and all tests that pass on @ pass here Differential Revision: https://phab.mercurial-scm.org/D1458

workers: don't use backgroundfilecloser in threads This disables background file closing when in not in main thread Test Plan: Ran pull, update, sparse commands and watched the closer threads created and destroyed in procexp.exe ran test on CentOS. No tests broken compared to the base Differential Revision: https://phab.mercurial-scm.org/D1457

debugdiscovery: correct and clean up command synopsis This does a few things: * Changes "-r" to "--rev", since "-r" is not a valid short form * Removes non-existent "-l" and "-b" options * Removes "..." after options, since we don't usually have that Differential Revision: https://phab.mercurial-scm.org/D1706

copies: group wdir-handling in one place I think this makes it both easier to follow and shorter. Differential Revision: https://phab.mercurial-scm.org/D1698