lfs: fix committing deleted files caused by e0a1b9ee93cd e0a1b9ee93cd (lfs: add a repo requirement for this extension once an lfs file is committed) introduced a regression that prevents committing file deletion. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D1717

tests: update expected output of svn encoding test The changes I see on the buildbot match the ones I see on my laptop, and all look reasonable. Differential Revision: https://phab.mercurial-scm.org/D1713

run-tests: use context managers for file descriptors I've seen the following error a few times recently when running the tests with `yes | ./run-tests.py --local -j9 -i`: Errored test-add.t: Traceback (most recent call last): File "./run-tests.py", line 821, in run self.runTest() File "./run-tests.py", line 910, in runTest if self._result.addOutputMismatch(self, ret, out, self._refout): File "./run-tests.py", line 1774, in addOutputMismatch rename(test.errpath, test.path) File "./run-tests.py", line 571, in rename os.remove(src) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\Users\\Matt\\projects\\hg\\tests\\test-add.t.err' This change doesn't fix the problem, but it seems like a simple enough improvement.

run-tests: add substitution patterns for common '\' path output on Windows The goal is to reduce the amount of hand tuning of new/changed tests that is required on Windows. Since the OS prints the proper paths everywhere else, this is limited to Windows. These are based on the check-code rules that were dropped in 5feb782c7a95. There are some minor tweaks, because those were trying to detect '/' paths without a '(glob)' at the end, whereas these detect '\' paths. Also, it looks like the 'no changes made to subrepo' one was broke, because the path to the subrepo has been getting output but was not in the pattern. End anchors are dropped because '(glob)' is no longer required, but '(feature !)' annotations are a possibility. The 'saved backup bundle' pattern dropped from run-tests.py was simply carrying over the first capture group. The replace() method runs prior to evaluating '\1', but it wasn't doing anything because of the 'r' prefix on '\\'. The 'not recording move' entry is new, because I stumbled upon it searching for some of these patterns. There are probably others.

debuginstall: add a line about re2 availability Using re2 engine can massively speed up regexp. We make it simpler to check if it is available in a given install.

tests: fix the check-code rule for testing non-existent files I missed this in feecfefeba25.

tests: convert the 'file://\$TESTTMP' rule to an automatic substitution The rule only triggered on non Windows platforms, even though Windows also required an adjustment. Automatic seems better. The aggressive globbing in test-subrepo-svn.t was found and rewritten by the substitution.

help: deprecate ui.slash in favor of slashpath template filter (issue5572) > For some reason, I thought someone (Mads?) said we had basically given > up on ui.slash, and some commands didn't support it. (from https://bz.mercurial-scm.org/show_bug.cgi?id=5572#c1) So the ui.slash option doesn't always work and is somewhat confusing. Let's make it clearer we won't improve the situation.

templatefilters: add slashpath() to convert path separator to slash Prepares for deprecating the ui.slash option, which isn't always respected.

check-code: remove unused variable 'winglobmsg' Follows up 5feb782c7a95.

phases: initialize number of loaded revisions to 0 As it isn't a revision number, an empty value should be 0, not -1.

phases: rename _phasemaxrev to _loadedrevslen to clarify it isn't max value "maxrev" sounds like max(0:tip), but it is actually len(0:tip).

lfs: add an experimental config to override User-Agent for the blob transfer This will allow developers to test against various server implementations. I didn't put it under [devel] because it's possible that some user needs to use it in the field.

lfs: add git to the User-Agent header for blob transfers As we were trying to transition off of the non production lfs-test-server for further experimenting, one of the problems we ran into was interoperability. A coworker setup gitbucket[1] to act as the blob server, tested with git, and passed it off to me. But push failed with a message saying "abort: LFS server returns invalid JSON:", and then proceeded to dump a huge HTML page to the screen. It turns out that it is assuming that git is the only thing that wants to do a blob transfer, and everything else is a web browser wanting HTML. It's only a single data point, but I suspect other things may be doing this too. RFC7231 gives an example [2] of listing multiple products in decreasing order of significance. Since the standard provides for this, and since it works with the one problematic server I found, I'm just enabling this by default for a better UX. There's nothing significant about the version of git chosen, other than it is the current version. [1] https://github.com/gitbucket/gitbucket/ [2] https://tools.ietf.org/html/rfc7231#page-46

outgoing: respect ":pushurl" paths (issue5365) Make 'hg outgoing' respect "paths.default:pushurl" in addition to "paths.default-push". 'hg outgoing' has always meant "what will happen if I run 'hg push'?" and it's still documented that way: Show changesets not found in the specified destination repository or the default push location. These are the changesets that would be pushed if a push was requested. If the user uses the now-deprecated "paths.default-push" path, it continues to work that way. However, as described at https://bz.mercurial-scm.org/show_bug.cgi?id=5365, it doesn't behave the same with "paths.default:pushurl". Why does it matter? Similar to the bugzilla reporter, I have a read-only mirror of a non-Mercurial repository: upstream -> imported mirror -> user clone ^-----------------------/ Users push directly to upstream, and that content is then imported into the mirror. However, those repositories are not the same; it's possible that the mirroring has either broken completely, or an import process is running and not yet complete. In those cases, 'hg outgoing' will list changesets that have already been pushed. Mozilla's desired behavior described in bug 5365 can be accomplished through other means (e.g. 'hg outgoing default'), preserving the consistency and meaning of 'hg outgoing'.

tests: test-pathconflicts-merge.t requires symlinks Once we're ready to turn this functionality on more widely, we might want to write a symlink-free version of the test that can run on Windows, but for now we'll just leave it disabled there. Differential Revision: https://phab.mercurial-scm.org/D1710

sshpeer: allow for additional environment passing to ssh exe We already have the ability to customize the ssh command line arguments, let's add the ability to customize its environment as well. Example use-case is ssh.exe from Git on Windows. If `HOME` enviroment variable is present and has some non-empty value, ssh.exe will try to access that location for some stuff (for example, it seems for resolving `~` in `.ssh/config`). Git for Windows seems to sometimess set this variable to the value of `/home/username` which probably works under Git Bash, but does not work in a native `cmd.exe` or `powershell`. Whatever the root cause, setting `HOME` to be an empty string heals things. Therefore, some distributors might want to set `sshenv.HOME=` in the configuration (seems less intrusive that forcing everyone to tweak their env). Test Plan: - rt Differential Revision: https://phab.mercurial-scm.org/D1683

unamend: allow unamending if allowunstable is set I don't see why unamend should be disallowed when allowunstable is set. By switching to rewriteutil.precheck() we fix that and get more consistent error messages (and some additional ones). Differential Revision: https://phab.mercurial-scm.org/D1682

rebase: add ui.log calls for whether IMM used, whether rebasing WCP Make it a bit easy to get metrics from these. Differential Revision: https://phab.mercurial-scm.org/D1681

lfs: using workers in lfs prefetch This significantly speeds up lfs prefetch. With fast network we are seeing ~50% improvement of overall prefetch times Because of worker's API in posix we do lose finegrained progress update and only see progress when a file finished downloading. Test Plan: Run tests: ./run-tests.py -l test-lfs* .... # Ran 4 tests, 0 skipped, 0 failed. Run commands resulting in lfs prefetch e.g. hg sparse --enable-profile Differential Revision: https://phab.mercurial-scm.org/D1568

worker: make windows workers daemons The windows workers weren't daemons and were not correctly killed when ctrl-c'd from the terminal. Withi this change when the main thread is killed, all daemons get killed as well. I also reduced the time we give to workers to cleanup nicely to not have people ctrl-c'ing when they get inpatient. The output when threads clened up nicely: PS C:\<dir>> hg.exe sparse --disable-profile SparseProfiles/<profile>.sparse interrupted! The output when threads don't clenup in 1 sec: PS C:\<dir> hg.exe sparse --enable-profile SparseProfiles/<profile>.sparse failed to kill worker threads while handling an exception interrupted! Exception in thread Thread-4 (most likely raised during interpreter shutdown): PS C:\<dir>> Test Plan: Run hg command on windows (pull/update/sparse). Ctrl-C'd sparse --enable-profile command that was using threads and observed in proces explorer that all threads got killed. ran tests on CentOS Differential Revision: https://phab.mercurial-scm.org/D1564

workers: add config to enable/diable workers This adds config to disable/enable workers with default being enabled. Test Plan: enabled profile without updaing .hg/hgrc (the default should be to use workers) and ran hg sprase --enable-profile <profile>.sparse Watched in the proces explorer that hg started 12 new threads for materializing files (this is my worker.numcpus) value Added [worker] enabled = False to the .hg/hgrc and re ran the command. This time hg didn't spawn any new threads for matreializing of files Differential Revision: https://phab.mercurial-scm.org/D1460

diff: disable diff.noprefix option for diffstat (issue5759) We could use patch.diffhunks() instead of patch.diff() to get filenames without parsing patch content, but that isn't always possible because we sometimes feed raw patch data to patch.diffstat().

hgweb: disable diff.noprefix option for diffstat Copied from cf1e15f91c90.

workers: handling exceptions in windows workers This adds handling of exceptions from worker threads and resurfaces them as if the function ran without workers. If any of the threads throws, the main thread kills all running threads giving them 5 sec to handle the interruption and raises the first exception received. We don't have to join threads if is_alive() is false Test Plan: Ran multiple updates/enable/disable sparse profile and things worked well Ran test on CentOS- all tests passing on @ passed here Added a forged exception into the worker code and got it properly resurfaced and the rest of workers killed: P58642088 PS C:\open\<repo>> ..\facebook-hg-rpms\build\hg\hg.exe --config extensions.fsmonitor=! sparse --enable-profile <profile> updating [==> ] 1300/39166 1m57sException in thread Thread-3: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception Exception in thread Thread-2: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception <...> Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hgexe.py", line 41, in <module> dispatch.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 85, in run status = (dispatch(req) or 0) & 255 File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 173, in dispatch ret = _runcatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 324, in _runcatch return _callcatch(ui, _runcatchfunc) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 332, in _callcatch return scmutil.callcatch(ui, func) File "C:\open\facebook-hg-rpms\build\hg\mercurial\scmutil.py", line 154, in callcatch return func() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 314, in _runcatchfunc return _dispatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 951, in _dispatch cmdpats, cmdoptions) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 415, in runcommand return orig(lui, repo, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\undo.py", line 118, in _runcommandwrapper result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hgext\journal.py", line 84, in runcommand return orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 268, in _tracksparseprofiles res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 256, in _trackdirstatesizes res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\copytrace.py", line 144, in _runcommand return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbamend\hiddenoverride.py", line 119, in runcommand result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 712, in runcommand ret = _runcommand(ui, options, cmd, d) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 959, in _runcommand return cmdfunc() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 948, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "C:\open\facebook-hg-rpms\build\hg\mercurial\util.py", line 1183, in check return func(*args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 860, in sparse disableprofile=disableprofile, force=force) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 949, in _config len, _refresh(ui, repo, oldstatus, oldsparsematch, force)) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 1116, in _refresh mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 311, in applyupdates return orig(repo, actions, wctx, mctx, overwrite, labels=labels) File "C:\open\facebook-hg-rpms\build\hg\mercurial\merge.py", line 1464, in applyupdates for i, item in prog: File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 286, in _windowsworker raise t.exception Exception: Forged exception PS C:\open\ovrsource> Differential Revision: https://phab.mercurial-scm.org/D1459

workers: implemented worker on windows This change implements thread based worker on windows. The handling of exception from within threads will happen in separate diff. The worker is for now used in mercurial/merge.py and in lfs extension After multiple tests and milions of files materiealized, thousands lfs fetched it seems that neither merge.py nor lfs/blobstore.py is thread unsafe. I also looked through the code and besides the backgroundfilecloser (handled in base of this) things look good. The performance boost of this on windows is ~50% for sparse --enable-profile * Speedup of hg up/rebase - not exactly measured Test Plan: Ran 10s of hg sparse --enable-profile and --disable-profile operations on large profiles and verified that workers are running. Used sysinternals suite to see that all threads are spawned and run as they should Run various other operations on the repo including update and rebase Ran tests on CentOS and all tests that pass on @ pass here Differential Revision: https://phab.mercurial-scm.org/D1458

workers: don't use backgroundfilecloser in threads This disables background file closing when in not in main thread Test Plan: Ran pull, update, sparse commands and watched the closer threads created and destroyed in procexp.exe ran test on CentOS. No tests broken compared to the base Differential Revision: https://phab.mercurial-scm.org/D1457

debugssl: convert port number to int (issue5757) It doesn't use util.getport(), which may resolve service name to port number.

debugdiscovery: correct and clean up command synopsis This does a few things: * Changes "-r" to "--rev", since "-r" is not a valid short form * Removes non-existent "-l" and "-b" options * Removes "..." after options, since we don't usually have that Differential Revision: https://phab.mercurial-scm.org/D1706

copies: group wdir-handling in one place I think this makes it both easier to follow and shorter. Differential Revision: https://phab.mercurial-scm.org/D1698

copies: extract method for getting non-wdir forward copies I may add an alternative way of getting copy metadata (from changelog, not filelog) but the chaining with the dirstate copy metadata will be the same, so it will probably help to have this extracted. Even if that doesn't happen, the next patch will show that we can simplify this a bit after this refactoring, so it seems worth it regardless. Differential Revision: https://phab.mercurial-scm.org/D1697

copies: consistently use """ for docstrings Differential Revision: https://phab.mercurial-scm.org/D1696

copies: always respect matcher arg to _forwardcopies() The function would ignore the matcher if the dirstate copies were requested. It doesn't matter in practice because all callers used the returned map only for looking up specific files from and those files had already been filtered by the matcher (AFACT). Still, it's a little confusing, so let's make it clearer by respecting the matcher in this case too. Differential Revision: https://phab.mercurial-scm.org/D1695

debugdiscovery: drop reference to non-existent --remote-head option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1693

debugdiscovery: drop reference to non-existent --serverlog option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1692

debugdiscovery: drop reference to invalid --branch option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1691

tests: avoid echo with backslash escapes Some shells' echo (e.g. Bash's) do not interpret backslash escapes by default, while others shells' do (e.g. Dash's). This led to a failure in test-run-tests.t. We could switch to the more compatible printf, but using a heredoc seems simpler, so that's what this patch does. For anyone who wants to test this for themselves, remove the "unset HGTEST_SHELL" line in helper-runtests.sh and pass e.g. "--shell=/bin/dash" to the test runner. Differential Revision: https://phab.mercurial-scm.org/D1699

hgweb: stop using HTML comments in <script> Once upon a time, in 1995, there were browsers that didn't understand <script> tags and they would simply show the code inside as text. This started a tradition of wrapping everything inside <script> in <!-- HTML comments -->. Nowadays, it's not only not needed, but can be considered harmful[1]: - within XHTML documents, the source will actually be hidden from all browsers and rendered useless - `--` is not allowed within HTML comments, so any decrement operations in script are invalid [1]: http://www.javascripttoolbox.com/bestpractices/#comments

merge with stable

statprof: remove a line of deadcode Differential Revision: https://phab.mercurial-scm.org/D1689

extdata: abort if external command exits with non-zero status (BC) Per the last discussion, this is more reliable and consistent way than suppressing an error. For grep, erroring out might be inconvenient, but for curl, non-zero exit status should be detected. The latter wouldn't be possible if non-zero status is ignored. https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-October/105727.html

hgweb: update graph function docstring

rebase: fix for hgsubversion 5c25fe7fb1e broke something in the hgsubversion test path, causing it raise an abort (Abort: nothing to merge) during a perfectly good rebase. I tracked it down to this change. It's probably not hgsubversion related. I suspect that using the same `wctx` from before the initial update causes problems with the wctx's cached manifest property. I noticed we also sometimes stick random gunk on the wctx object in other places (like in `copies.py`) so it's probably best to reset it for now. The line I added before was actually useless since we don't pass wctx to the initial `merge.update`, so it defaults to `repo[None]`. So I just removed it. Differential Revision: https://phab.mercurial-scm.org/D1679

hgweb: render next pages on /graph incrementally Previously, when user scrolled down to see the next page on /graph, all hgweb did was re-render everything that would be visible (by simply incrementing revcount). It was not efficient at all, and this patch makes /graph page behave similarly to the regular /log: every new page only consists of new changesets, no duplication, and only jsdata is based on the full set of changesets required to build accurate graph. This is achieved by adding "?graphtop=<node>" to the next page URL template, effectively remembering where the graph started, and using that value to create the new `tree` that covers the whole visible graph. That variable is then used to produce jsdata for redrawing graph client-side. nextentry is used for the same purpose as on /log page (to format the next page URL), but it's not a part of the graph.

hgweb: split graphdata() into jsdata() and nodes() nodes keyword passed to the template can be any iterator, but jsdata needs to be a list because it gets JSONified.

hgweb: calculate <canvas> width and height client-side hgweb determines and passes to templates some variables related to graph appearance, like bg_height, canvaswidth and canvasheight. bg_height was and still is used for graph.scale() call in graph.tmpl, and the two latter variables were used in <canvas> element as width and height properties, and they were set before JS code got to run. Setting these properties server-side doesn't make a lot of sense, because a graph that has been scaled should calculate things like width and height on its own when being rendered. Let's move (re)sizing <canvas> to JavaScript (to Graph.render function) and stop parsing HTML with regular expressions just to know new width and height. That extra loop that only counts cols is required because <canvas> can't be resized after or in the process of rendering (or it gets cleared). Incidentally, SVG doesn't have this problem and I'm hoping to switch graph to using it in future. There also was truecanvasheight, but according to hg grep --all it was never used, see d490edc71146.

hgweb: filter graphmod.colored() output before iterating over it Consumers in this function use output of graphmod.colored(), but only want items with type == CHANGESET, so let's filter it early. This is primarily just a refactoring, but it also fixes a potential small bug with `rows = len(tree)` (this variable is used for "Rows shown" line in raw-graph) if there are items of other types.

py3: introduce pycompat.ziplist as zip is a generator on Python 3 Similar to map, the return value of zip was also turned into a generator on Python 3. We have multiple occurences in our codebase where we retrieve the values of zip multiple times, so let's introduce pycompat.ziplist similar to pycompat.maplist. Differential Revision: https://phab.mercurial-scm.org/D1675