lfs: add git to the User-Agent header for blob transfers As we were trying to transition off of the non production lfs-test-server for further experimenting, one of the problems we ran into was interoperability. A coworker setup gitbucket[1] to act as the blob server, tested with git, and passed it off to me. But push failed with a message saying "abort: LFS server returns invalid JSON:", and then proceeded to dump a huge HTML page to the screen. It turns out that it is assuming that git is the only thing that wants to do a blob transfer, and everything else is a web browser wanting HTML. It's only a single data point, but I suspect other things may be doing this too. RFC7231 gives an example [2] of listing multiple products in decreasing order of significance. Since the standard provides for this, and since it works with the one problematic server I found, I'm just enabling this by default for a better UX. There's nothing significant about the version of git chosen, other than it is the current version. [1] https://github.com/gitbucket/gitbucket/ [2] https://tools.ietf.org/html/rfc7231#page-46

outgoing: respect ":pushurl" paths (issue5365) Make 'hg outgoing' respect "paths.default:pushurl" in addition to "paths.default-push". 'hg outgoing' has always meant "what will happen if I run 'hg push'?" and it's still documented that way: Show changesets not found in the specified destination repository or the default push location. These are the changesets that would be pushed if a push was requested. If the user uses the now-deprecated "paths.default-push" path, it continues to work that way. However, as described at https://bz.mercurial-scm.org/show_bug.cgi?id=5365, it doesn't behave the same with "paths.default:pushurl". Why does it matter? Similar to the bugzilla reporter, I have a read-only mirror of a non-Mercurial repository: upstream -> imported mirror -> user clone ^-----------------------/ Users push directly to upstream, and that content is then imported into the mirror. However, those repositories are not the same; it's possible that the mirroring has either broken completely, or an import process is running and not yet complete. In those cases, 'hg outgoing' will list changesets that have already been pushed. Mozilla's desired behavior described in bug 5365 can be accomplished through other means (e.g. 'hg outgoing default'), preserving the consistency and meaning of 'hg outgoing'.

tests: test-pathconflicts-merge.t requires symlinks Once we're ready to turn this functionality on more widely, we might want to write a symlink-free version of the test that can run on Windows, but for now we'll just leave it disabled there. Differential Revision: https://phab.mercurial-scm.org/D1710

sshpeer: allow for additional environment passing to ssh exe We already have the ability to customize the ssh command line arguments, let's add the ability to customize its environment as well. Example use-case is ssh.exe from Git on Windows. If `HOME` enviroment variable is present and has some non-empty value, ssh.exe will try to access that location for some stuff (for example, it seems for resolving `~` in `.ssh/config`). Git for Windows seems to sometimess set this variable to the value of `/home/username` which probably works under Git Bash, but does not work in a native `cmd.exe` or `powershell`. Whatever the root cause, setting `HOME` to be an empty string heals things. Therefore, some distributors might want to set `sshenv.HOME=` in the configuration (seems less intrusive that forcing everyone to tweak their env). Test Plan: - rt Differential Revision: https://phab.mercurial-scm.org/D1683

unamend: allow unamending if allowunstable is set I don't see why unamend should be disallowed when allowunstable is set. By switching to rewriteutil.precheck() we fix that and get more consistent error messages (and some additional ones). Differential Revision: https://phab.mercurial-scm.org/D1682

rebase: add ui.log calls for whether IMM used, whether rebasing WCP Make it a bit easy to get metrics from these. Differential Revision: https://phab.mercurial-scm.org/D1681

lfs: using workers in lfs prefetch This significantly speeds up lfs prefetch. With fast network we are seeing ~50% improvement of overall prefetch times Because of worker's API in posix we do lose finegrained progress update and only see progress when a file finished downloading. Test Plan: Run tests: ./run-tests.py -l test-lfs* .... # Ran 4 tests, 0 skipped, 0 failed. Run commands resulting in lfs prefetch e.g. hg sparse --enable-profile Differential Revision: https://phab.mercurial-scm.org/D1568

worker: make windows workers daemons The windows workers weren't daemons and were not correctly killed when ctrl-c'd from the terminal. Withi this change when the main thread is killed, all daemons get killed as well. I also reduced the time we give to workers to cleanup nicely to not have people ctrl-c'ing when they get inpatient. The output when threads clened up nicely: PS C:\<dir>> hg.exe sparse --disable-profile SparseProfiles/<profile>.sparse interrupted! The output when threads don't clenup in 1 sec: PS C:\<dir> hg.exe sparse --enable-profile SparseProfiles/<profile>.sparse failed to kill worker threads while handling an exception interrupted! Exception in thread Thread-4 (most likely raised during interpreter shutdown): PS C:\<dir>> Test Plan: Run hg command on windows (pull/update/sparse). Ctrl-C'd sparse --enable-profile command that was using threads and observed in proces explorer that all threads got killed. ran tests on CentOS Differential Revision: https://phab.mercurial-scm.org/D1564

workers: add config to enable/diable workers This adds config to disable/enable workers with default being enabled. Test Plan: enabled profile without updaing .hg/hgrc (the default should be to use workers) and ran hg sprase --enable-profile <profile>.sparse Watched in the proces explorer that hg started 12 new threads for materializing files (this is my worker.numcpus) value Added [worker] enabled = False to the .hg/hgrc and re ran the command. This time hg didn't spawn any new threads for matreializing of files Differential Revision: https://phab.mercurial-scm.org/D1460

diff: disable diff.noprefix option for diffstat (issue5759) We could use patch.diffhunks() instead of patch.diff() to get filenames without parsing patch content, but that isn't always possible because we sometimes feed raw patch data to patch.diffstat().

hgweb: disable diff.noprefix option for diffstat Copied from cf1e15f91c90.

workers: handling exceptions in windows workers This adds handling of exceptions from worker threads and resurfaces them as if the function ran without workers. If any of the threads throws, the main thread kills all running threads giving them 5 sec to handle the interruption and raises the first exception received. We don't have to join threads if is_alive() is false Test Plan: Ran multiple updates/enable/disable sparse profile and things worked well Ran test on CentOS- all tests passing on @ passed here Added a forged exception into the worker code and got it properly resurfaced and the rest of workers killed: P58642088 PS C:\open\<repo>> ..\facebook-hg-rpms\build\hg\hg.exe --config extensions.fsmonitor=! sparse --enable-profile <profile> updating [==> ] 1300/39166 1m57sException in thread Thread-3: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception Exception in thread Thread-2: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception <...> Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hgexe.py", line 41, in <module> dispatch.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 85, in run status = (dispatch(req) or 0) & 255 File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 173, in dispatch ret = _runcatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 324, in _runcatch return _callcatch(ui, _runcatchfunc) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 332, in _callcatch return scmutil.callcatch(ui, func) File "C:\open\facebook-hg-rpms\build\hg\mercurial\scmutil.py", line 154, in callcatch return func() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 314, in _runcatchfunc return _dispatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 951, in _dispatch cmdpats, cmdoptions) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 415, in runcommand return orig(lui, repo, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\undo.py", line 118, in _runcommandwrapper result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hgext\journal.py", line 84, in runcommand return orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 268, in _tracksparseprofiles res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 256, in _trackdirstatesizes res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\copytrace.py", line 144, in _runcommand return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbamend\hiddenoverride.py", line 119, in runcommand result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 712, in runcommand ret = _runcommand(ui, options, cmd, d) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 959, in _runcommand return cmdfunc() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 948, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "C:\open\facebook-hg-rpms\build\hg\mercurial\util.py", line 1183, in check return func(*args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 860, in sparse disableprofile=disableprofile, force=force) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 949, in _config len, _refresh(ui, repo, oldstatus, oldsparsematch, force)) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 1116, in _refresh mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 311, in applyupdates return orig(repo, actions, wctx, mctx, overwrite, labels=labels) File "C:\open\facebook-hg-rpms\build\hg\mercurial\merge.py", line 1464, in applyupdates for i, item in prog: File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 286, in _windowsworker raise t.exception Exception: Forged exception PS C:\open\ovrsource> Differential Revision: https://phab.mercurial-scm.org/D1459

workers: implemented worker on windows This change implements thread based worker on windows. The handling of exception from within threads will happen in separate diff. The worker is for now used in mercurial/merge.py and in lfs extension After multiple tests and milions of files materiealized, thousands lfs fetched it seems that neither merge.py nor lfs/blobstore.py is thread unsafe. I also looked through the code and besides the backgroundfilecloser (handled in base of this) things look good. The performance boost of this on windows is ~50% for sparse --enable-profile * Speedup of hg up/rebase - not exactly measured Test Plan: Ran 10s of hg sparse --enable-profile and --disable-profile operations on large profiles and verified that workers are running. Used sysinternals suite to see that all threads are spawned and run as they should Run various other operations on the repo including update and rebase Ran tests on CentOS and all tests that pass on @ pass here Differential Revision: https://phab.mercurial-scm.org/D1458

workers: don't use backgroundfilecloser in threads This disables background file closing when in not in main thread Test Plan: Ran pull, update, sparse commands and watched the closer threads created and destroyed in procexp.exe ran test on CentOS. No tests broken compared to the base Differential Revision: https://phab.mercurial-scm.org/D1457

debugssl: convert port number to int (issue5757) It doesn't use util.getport(), which may resolve service name to port number.

debugdiscovery: correct and clean up command synopsis This does a few things: * Changes "-r" to "--rev", since "-r" is not a valid short form * Removes non-existent "-l" and "-b" options * Removes "..." after options, since we don't usually have that Differential Revision: https://phab.mercurial-scm.org/D1706

copies: group wdir-handling in one place I think this makes it both easier to follow and shorter. Differential Revision: https://phab.mercurial-scm.org/D1698

copies: extract method for getting non-wdir forward copies I may add an alternative way of getting copy metadata (from changelog, not filelog) but the chaining with the dirstate copy metadata will be the same, so it will probably help to have this extracted. Even if that doesn't happen, the next patch will show that we can simplify this a bit after this refactoring, so it seems worth it regardless. Differential Revision: https://phab.mercurial-scm.org/D1697

copies: consistently use """ for docstrings Differential Revision: https://phab.mercurial-scm.org/D1696

copies: always respect matcher arg to _forwardcopies() The function would ignore the matcher if the dirstate copies were requested. It doesn't matter in practice because all callers used the returned map only for looking up specific files from and those files had already been filtered by the matcher (AFACT). Still, it's a little confusing, so let's make it clearer by respecting the matcher in this case too. Differential Revision: https://phab.mercurial-scm.org/D1695

debugdiscovery: drop reference to non-existent --remote-head option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1693

debugdiscovery: drop reference to non-existent --serverlog option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1692

debugdiscovery: drop reference to invalid --branch option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1691

tests: avoid echo with backslash escapes Some shells' echo (e.g. Bash's) do not interpret backslash escapes by default, while others shells' do (e.g. Dash's). This led to a failure in test-run-tests.t. We could switch to the more compatible printf, but using a heredoc seems simpler, so that's what this patch does. For anyone who wants to test this for themselves, remove the "unset HGTEST_SHELL" line in helper-runtests.sh and pass e.g. "--shell=/bin/dash" to the test runner. Differential Revision: https://phab.mercurial-scm.org/D1699

hgweb: stop using HTML comments in <script> Once upon a time, in 1995, there were browsers that didn't understand <script> tags and they would simply show the code inside as text. This started a tradition of wrapping everything inside <script> in <!-- HTML comments -->. Nowadays, it's not only not needed, but can be considered harmful[1]: - within XHTML documents, the source will actually be hidden from all browsers and rendered useless - `--` is not allowed within HTML comments, so any decrement operations in script are invalid [1]: http://www.javascripttoolbox.com/bestpractices/#comments

merge with stable

statprof: remove a line of deadcode Differential Revision: https://phab.mercurial-scm.org/D1689

extdata: abort if external command exits with non-zero status (BC) Per the last discussion, this is more reliable and consistent way than suppressing an error. For grep, erroring out might be inconvenient, but for curl, non-zero exit status should be detected. The latter wouldn't be possible if non-zero status is ignored. https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-October/105727.html

hgweb: update graph function docstring

rebase: fix for hgsubversion 5c25fe7fb1e broke something in the hgsubversion test path, causing it raise an abort (Abort: nothing to merge) during a perfectly good rebase. I tracked it down to this change. It's probably not hgsubversion related. I suspect that using the same `wctx` from before the initial update causes problems with the wctx's cached manifest property. I noticed we also sometimes stick random gunk on the wctx object in other places (like in `copies.py`) so it's probably best to reset it for now. The line I added before was actually useless since we don't pass wctx to the initial `merge.update`, so it defaults to `repo[None]`. So I just removed it. Differential Revision: https://phab.mercurial-scm.org/D1679