dispatch: handle late KeyboardInterrupt occurred in run() User can press Ctrl+C while flushing streams in dispatch.run(). In such case, I think exiting with 255 is better than printing Python traceback and exiting with 1.

dispatch: indent run() function I'll add KeyboardInterrupt handling there.

debuginstall: don't translate encoding messages While Microsoft trained us to read mojibake text, it doesn't make sense to print mojibake messages when debugging the exact issue.

merge with stable

patch: refactor content diffing part in separate fn so extensions can wrap Right now extdiff uses it's own logic using archival to diff two versions of file using external diff tools. This makes the extdiff functionality non-extensible. This series is an attempt to refactor core patch and diff functionality so that extdiff can wrap and reuse it. This will help us in using external diffing tools at more places and not just extdiff command only then. Differential Revision: https://phab.mercurial-scm.org/D8685

tests: check that procutil.std{out,err}.write() returns correct result On Windows, we currently don’t fully test the case when the stream is connected to a TTY, but we test the child process side by connecting them to NUL, which is recognized as a TTY by Python. To make the large write test a bit more useful besides checking that it doesn’t crash, we can check that the write() method returns the correct result.

tests: add tests for when stdout or stderr is connected to `os.devnull` The original motivation was that creating PTYs on Windows is not possible, but `NUL` is recognized as a TTY, so we can have at least some test coverage for the TTY case. I think it doesn’t hurt to run the test cases on all systems.

procutil: ensure that procutil.std{out,err}.write() writes all bytes Python 3 offers different kind of streams and it’s not guaranteed for all of them that calling write() writes all bytes. When Python is started in unbuffered mode, sys.std{out,err}.buffer are instances of io.FileIO, whose write() can write less bytes for platform-specific reasons (e.g. Linux has a 0x7ffff000 bytes maximum and could write less if interrupted by a signal; when writing to Windows consoles, it’s limited to 32767 bytes to avoid the "not enough space" error). This can lead to silent loss of data, both when using sys.std{out,err}.buffer (which may in fact not be a buffered stream) and when using the text streams sys.std{out,err} (I’ve created a CPython bug report for that: https://bugs.python.org/issue41221). Python may fix the problem at some point. For now, we implement our own wrapper for procutil.std{out,err} that calls the raw stream’s write() method until all bytes have been written. We don’t use sys.std{out,err} for larger writes, so I think it’s not worth the effort to patch them.

procutil: move assignments This should probably be part of the previous patch, but folding it results in a less useful word diff, so I decided to keep it separate for review.

procutil: distribute code for stdout It makes sense to have the distinction between Python 2 and 3 at the top level, as we have to fight a different kind of battle on each: On Python 3, we get consistent behavior on all platforms, but need to create correctly-behaving binary streams. On Python 2, we have to account for platform differences.

overlayworkingctx: remove unused `nofilechanges()` and `_compact()` methods Differential Revision: https://phab.mercurial-scm.org/D8733

rebase: correctly check for empty commit in in-memory mode The new code has a small overhead in the empty commit case, as a `memctx` object is always created, but I don’t think it’s justified here to duplicate code to optimize a relatively unlikely code path. Differential Revision: https://phab.mercurial-scm.org/D8732

memctx: make `parents()` return list of one element if it’s not a merge It is part of the contract of `parents()` that non-merges return a list of one element. `self._parents` is always a list of two elements in `memctx`. Differential Revision: https://phab.mercurial-scm.org/D8731

commit: factor out empty commit check to `basectx.isempty()` This enables reuse in other places, e.g. those dealing with `memctx`. Differential Revision: https://phab.mercurial-scm.org/D8729

overlayworkingctx: rename misleadingly named `isempty()` method This method is only about whether there are file changes, not about whether the commit will be empty or not. One user of the method was incorrectly assuming the latter meaning, leading to the bug for which a test case was added in D8727. I’ve added a FIXME to the code. The original motivation for the rename was that I want to add `committablectx.isempty()`, that properly checks if a commit will be empty, using the exact same logic as in `repo.commit()`, and I wanted to avoid a name clash. Differential Revision: https://phab.mercurial-scm.org/D8728

rebase: add test case that shows that inmemory rebase does not preserve merges Differential Revision: https://phab.mercurial-scm.org/D8727

rebase: remove now unnecessary logic to allow empty commit when branch changes This was a workaround for a bug in the empty commit check in repo.commit(), where the parent branch name was incorrectly compared with the wdir branch name instead of the branch name passed via `extra`. The bug was fixed in D8724. The workaround was introduced in b2415e94b2f5. Differential Revision: https://phab.mercurial-scm.org/D8725

commit: check branch change on to-be-committed changeset This makes a difference when a branch name is passed via the `extra` argument, which will be used as the branch name for the created changeset. The empty commit check should use that branch name instead of whatever was set on the working directory. Besides fixing the bug for which a test case was extended, it enables to remove a workaround in the rebase extension, which will be done in a follow-up patch (D8725). Differential Revision: https://phab.mercurial-scm.org/D8724

merge with stable

procutil: move comment closer to relevant code

procutil: use mercurial.windows.winstdout only on Python 2 and TTYs Python 3 already works around the bug. The workaround is only needed when writing to consoles. If stdout is a console, sys.stdout.isatty() is true.

procutil: split if condition This prepares the code for subsequent changes where we need to differentiate between the two cases.

procutil: explain better why line buffering is not possible The sentence “On Python 3, buffered binary streams can't be set line-buffered.” was imprecise, as all streams are just Python classes and we can implement our own (which we did).

tests: make subprocess handling reusable for different tests in test-stdio.py

procutil: back out 8403cc54bc83 (make ....procutil.stderr unbuffered) Changeset 8403cc54bc83 introduced code that opens a second file object referring to the stderr file descriptor. This broke tests on Windows. The reason is that on Windows, sys.stderr is buffered and procutil.stderr closed the file descriptor when it got garbage collected before sys.stderr had the chance to flush buffered data. `procutil.stdout` had the same problem for a long time, but we didn’t realize, as in CI test runs, stdout is not a TTY and in this case no second file object is opened.

test-clone-uncompressed: use config to silence the lock warning The ui.timeout.warn option can solve the issue6237 in a cleaner way than what was done in 752da6863e39. That way, possible other unexpected error output will not be silenced. Differential Revision: https://phab.mercurial-scm.org/D8726

hooklib: fix detection of successors for changeset_obsoleted Provide a hook for obsutil.getobsolete to be used with either a transaction or the changes item of the transaction, since hooks only have access to the latter. Use that to find the correct list of revisions with obsmarkers, even new ones, and then filter out revisions with known successors. Move the processing from pretxnclose to txnclose as the transaction access itself is no longer necessary. This is more in line with notify and ensures that sanity checks can abort the transaction first. Differential Revision: https://phab.mercurial-scm.org/D8575

fix: prefetch file contents This prevents the worker subprocesses from contacting the server individually, which is either inefficient, or leads to problems if the connection is shared among them. Differential Revision: https://phab.mercurial-scm.org/D8723

fix: obtain base paths before starting workers This moves calculation of base paths to before work is dispatched. While this does mean that copy tracing will be serialized instead of parallel, it is necessary to be able to prefetch the base contents in a batch, which will likely be more efficient. Differential Revision: https://phab.mercurial-scm.org/D8722

scmutil: allowing different files to be prefetched per revision The old API takes a list of revision separate from the file matcher, and thus provides no way to fetch different sets of files from each revision. In preparation for adding one such usage, I'm changing the API to take a list of (revision, file matcher) tuples instead. Differential Revision: https://phab.mercurial-scm.org/D8721

tests: update test-devel-warnings.t output with chg

tests: make names in test-stdio.py more distinctive This way, more tests can be added without name clashes.

tests: terminate subprocess in test-stdio.py in case of exception If an error happened while reading the output of the subprocess, the pipe / TTY buffer can fill up and prevent that the subprocess ends. Therefore we should terminate the subprocess in case of an exception.

tests: proof test-stdio.py against buffer fill-up With the previous code, it could in theory happen that the pipe / PTY buffer of the child stdout / stderr fills up and the process never finishes. To prevent that, we read all of the stream before waiting for the end of the process. To ensure that the stream reaches EOF when the child finishes, we must close the parent "copy" of the child stdout / stderr.

tests: stablize test-serve.t on Windows I forget the reason that the subprocess on Windows doesn't print this, but all other instances of this are similarly conditionalized, so I didn't think too hard about it. Also, the server needs to be killed so it doesn't prevent the next run from working, especially since the port isn't randomized. Differential Revision: https://phab.mercurial-scm.org/D8720

commitctx: document the manifest writing function Same spirit as for `_filecommit` lets document the input and output before making any change or improvement. This is part of a larger refactoring/cleanup of the commitctx code to clarify and augment the logic gathering metadata useful for copy tracing. The current code is a tad too long and entangled to make such update easy. We start with easy and small cleanup. Differential Revision: https://phab.mercurial-scm.org/D8705

commitctx: reorder some conditional for efficiency in _filecommit Checking if a dict is empty will be faster than comparing text. I don't expect it to be a huge performance win, but still a good (but gratuitous) cleanup to do while we are at it. This is part of a larger refactoring/cleanup of the commitctx code to clarify and augment the logic gathering metadata useful for copy tracing. The current code is a tad too long and entangled to make such update easy. We start with easy and small cleanup. Differential Revision: https://phab.mercurial-scm.org/D8701

commitctx: document _filecommit input and output This is the beginning of a larger refactoring/cleanup of the commitctx code to clarify and augment the logic gathering metadata useful for copy tracing. The current code is a tad too long and entangled to make such update easy. We start with easy and small cleanup. Differential Revision: https://phab.mercurial-scm.org/D8699

phases: replace magic number by constant Differential Revision: https://phab.mercurial-scm.org/D8695

clonebundles: optional memory-requirement attribution The new REQUIREDRAM option allows a client to skip bundles it isn't expected to handle well, e.g. without swapping. This allows a fallback path to be provided e.g. using zstd level 10 instead of 22. Differential Revision: https://phab.mercurial-scm.org/D8645

util: provide a helper function to estimate RAM size For POSIX systems, it uses sysconf. For Windows, it uses the win32 API directly. Differential Revision: https://phab.mercurial-scm.org/D8644

tests: allow timing difference in output Differential Revision: https://phab.mercurial-scm.org/D8676

documentation: add some internals documentation about bid merge This is an important algorithm that was only documented on the wiki so far. Some update to the algorithm (and associated doc) is to expected in the future since the bid merge algorithm is bug-ridden when it comes to file deletion comes to play. Differential Revision: https://phab.mercurial-scm.org/D8711

cmdserver: document message-encodings and channel output options While writing the previous patch, I noticed these options are undocumented. In my testing, a separate status/error message channel works well in GUI frontend as we no longer have to sort out data and message from mixed outputs. So let's mark it as not experimental.

cmdserver: add option to not exit from message loop on SIGINT Sending SIGINT to server is the only way to interrupt a command running in command-server process. SIGINT will be caught at dispatch.dispatch() if we're lucky. Otherwise it will terminate the serer process. This is fundamentally unreliable as signals are delivered asynchronously. "cmdserver.shutdown-on-interrupt=False" mitigate the issue by making the server basically block SIGINT.

pycompat: remove pycompat.{stdin,stdout,stderr} All users have been changed to use procutil.{stdin,stdout,stderr}, which provide consistent behavior across platforms and Python versions.

pycompat: change users of pycompat.{stdin,stdout,stderr} to use procutil.std* On Python 3, pycompat.{stdin,stdout,stderr} are usually block-buffered even if connected to a TTY. procutil.{stdin,stdout,stderr} provide consistent behavior across platforms and Python versions.

hgignore: add mypy cache files Differential Revision: https://phab.mercurial-scm.org/D8692

revlog: avoid hard-coded hash sizes Differential Revision: https://phab.mercurial-scm.org/D8683

test: restore expected output in test-persistent-nodemap.t The merge 78cafd48b9b2 had a small error and wrongly dropped the output updated from one side of the merge. This changeset re-install it. Differential Revision: https://phab.mercurial-scm.org/D8678

heptapod-ci: build rhg before tests run Differential Revision: https://phab.mercurial-scm.org/D8682

rhg: add a limited `rhg root` subcommand Clap has been choosen for argument parsing for the following reasons: - it's a wildly used and maintained crate - it can deal with OS encoding making it suitable for any encoding - it supports nonambiguous prefix matching as already available in hg - it will soon allow for structopts-style declarative pattern instead of the currently used builder pattern Differential Revision: https://phab.mercurial-scm.org/D8613

rhg: add RootCommand using hg-core FindRoot operation to prepare `hg root` The println macro is not used to avoid string usage. Dealing only with bytes allows us to be compatible with any encoding and not just UTF8. Later on, format macro will be made to have more readable code. Differential Revision: https://phab.mercurial-scm.org/D8612

merge with stable

procutil: make mercurial.utils.procutil.stderr unbuffered For most Mercurial code, it doesn’t make a difference, as the ui object flushes stderr explicitly (after the change, we could get rid of the explicit flush). One example where it makes a observable difference is mercurial.util.timed(). Without the patch, the time is not immediately shown on Python 3. With the patch, it’s shown immediately on all Python versions and platforms.

tests: generalize common test case code in test-stdio.py

procutil: make stdout line-buffered on Windows if connected to TTY Windows doesn’t support line buffering. Previously, we worked around that by setting the stream unbuffered. Instead, we can use our own line buffering we already use on Python 3.

procutil: split import and reassignment Besides making the code clearer, it will reduce the diff in the next patch.

procutil: move assignments closer to reassignments Doing reassignments is an anti-pattern IMHO, but I see how it makes sense here. When first looking at this code after jumping here with ctags, I missed the fact that stdout was reassigned. To make the code clearer, the assignments should be as close as possible to the reassignments.

procutil: factor out conditional creation of LineBufferedWrapper At the same time, document the logic and generalize it to work on all Python versions.

procutil: define LineBufferedWrapper on all Python versions There’s nothing Python 3-only about LineBufferedWrapper. In the future, we may want to use it on Windows, to work around missing line-buffering support.

tests: add tests for buffering behavior of mercurial.utils.procutil.stdout

cleanup: use slightly more meaningful name for temporary variable Not that it makes a big difference, but using `p` instead of `x` is clearer to me.

cleanup: use any() instead of checking truthiness of temporary list It was not immediately obvious to me, when first seeing this, why a list was created. It needed a second look to understand that the purpose was to check whether the condition is true for any of the parents. Using any() for that is clearer.

chg: suppress OSError in _restoreio() and add some logging (issue6330) According to issue6330, running chg on heavy loaded systems can lead to following error: ``` Traceback (most recent call last): File "path-to-hg/mercurial/commandserver.py", line 650, in _acceptnewconnection self._runworker(conn) File "path-to-hg/mercurial/commandserver.py", line 701, in _runworker prereposetups=[self._reposetup], File "path-to-hg/mercurial/commandserver.py", line 470, in _serverequest sv.cleanup() File "path-to-hg/mercurial/chgserver.py", line 381, in cleanup self._restoreio() File "path-to-hg/mercurial/chgserver.py", line 444, in _restoreio os.dup2(fd, fp.fileno()) OSError: [Errno 16] Device or resource busy ``` [man dup2] indicates that, on Linux, EBUSY comes from a race condition between open() and dup2(). However it's not clear why open() race occurred for newfd=stdin/out/err. We suppress the OSError in _restoreio() since the forked worker process will finish anyway and add some logging. Thanks to Mitchell Plamann for a detailed bug description and Yuya Nishihara for suggesting the fix.

ui: fix Python 2.7 support for ui.timestamp-output Differential Revision: https://phab.mercurial-scm.org/D8675

merge with stable

ui: add option to timestamp status and diagnostic messages Differential Revision: https://phab.mercurial-scm.org/D8640

merge with stable

compat: back out a25343d16ebe (initialize LC_CTYPE locale on all Python ...) As Yuya Nishihara pointed out, setting LC_CTYPE changes the behavior of some str methods on Python 2.

curses: back out d2227d4c9e6b (do not initialize LC_ALL to user settings) The changeset was based on a25343d16ebe, which will be backed out, too. Another fix for the problem will be resubmitted to the stable branch.

test: redirect stderr so warning messages don't change output (issue6237) clone and commit race for the lock, and if commit has to wait more than a second it prints a warning to stderr. Since this is somewhat expected here, silence it. Differential Revision: https://phab.mercurial-scm.org/D8664

locks: expect repo lock, not wlock, when writing to .hg/strip-backup/ There should be no need for a working copy lock when creating (or reading) bundles in `.hg/strip-backup/` since they don't affect the working copy. I noticed this because we have an extension that tries to strip some revisions while holding only a repo lock. I guess we have no such cases in core, which seems a bit surprising. Maybe we always take a wlock at a higher level so the working copy is not updated while the target commit is being stripped. Differential Revision: https://phab.mercurial-scm.org/D8666

graft: leverage cmdutil.check_incompatible_arguments() for --abort/--stop Differential Revision: https://phab.mercurial-scm.org/D8669

graft: leverage cmdutil.check_incompatible_arguments() for --no-commit Differential Revision: https://phab.mercurial-scm.org/D8668

graft: leverage cmdutil.check_at_most_one_arg() for --abort/--stop/--continue Differential Revision: https://phab.mercurial-scm.org/D8667

version: sort extensions by name in verbose mode External extensions can be assigned any name, but presumably most enabled extensions will be internal ones and having them sorted makes it easier to find specific ones if the list is long. The lists in `hg help extensions` are already sorted. Differential Revision: https://phab.mercurial-scm.org/D8671

crecord: stop trying to import wcurses The original import of crecord in 2008 already said "I have no idea if wcurses works with crecord...". The last reference to a Python package called wcurses is https://web.archive.org/web/20101025073658/http://adamv.com/dev/python/curses/. However, the Python package from there is called "curses" and not "wcurses". I didn’t find any evidence that it ever worked.

debian: support building a single deb for multiple py3 versions Around transitions from one python minor version to another (such as 3.7 to 3.8), the current packaging can be slightly problematic - it produces a `control` file that requires that the version of `python3` that's installed be exactly the one that was used on the build machine for the `mercurial` package, by containing a line like: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.8), python3 (>= 3.7~), python3:any (>= 3.5~) This is because it "knows" we only built for v3.7, which is the current default on my system. By building the native components for multiple versions, we can make it produce a line like this, which is compatible with 3.7 AND 3.8: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.9), python3 (>= 3.7~), python3:any (>= 3.5~) This isn't *normally* required, so I'm not making it the default. For those that receive their python3 and mercurial packages from their distro, and/or don't have to worry about a situation where the team that manages the python3 installation isn't the same as the team that manages the mercurial installation, this is probably not necessary. I chose the names `DEB_HG_*` because `DEB_*` is passed through `debuild` automatically (otherwise we'd have to explicitly allow the options through, which is a nuisance), and the `HG` part is to make it clear that this isn't a "standard" debian option that other packages might respect. Test Plan: 1. "nothing changed": - built a deb without these changes - built a deb with these changes but everything at the default - used diffoscope to compare, all differences were due to timestamps 2. "explicit is the same as implicit" (single version) - built a deb with everything at the default - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 3. "explicit is the same as implicit" (multi version) - built a deb with DEB_HG_MULTI_VERSION=1 - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 4. (single version, 3.7) doesn't work with python3.8 - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` crashes 5. (multi version, 3.7 + 3.8) - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` works Differential Revision: https://phab.mercurial-scm.org/D8642

merge with stable

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Luckily this is already done at interpreter startup on modern Python versions and, since recently, by Mercurial in the pycompat module in all other cases.

compat: initialize LC_CTYPE locale on all Python versions and platforms Previously, the LC_CTYPE locale was not initialized according to user settings on all Python versions (e.g. never on Python 2) and platforms (e.g. not on some Python < 3.8 on Windows). This broke e.g. non-ASCII filenames passed to the Subversion bindings on Python 2, resulting in error messages like "file:///tmp/a%C3%A4 does not look like a Subversion repository to libsvn version 1.14.0". The following command could be used to test this functionality. Adding it to the test suite would be pointless, as the locale is always set to "C" during test runs. @command(b'check_initial_codeset', norepo=True) def check_initial_codeset(ui): codeset1 = locale.nl_langinfo(locale.CODESET) locale.setlocale(locale.LC_ALL, '') codeset2 = locale.nl_langinfo(locale.CODESET) assert codeset1 == codeset2

merge with stable

merge: don't grab wlock when merging in memory I noticed this because we have an internal extension that does an in-memory rebase while holding only a repo lock, which resulted in a developer warning about the working copy lock being taken after the repo lock. Differential Revision: https://phab.mercurial-scm.org/D8665

pycompat: use os.fsencode() to re-encode sys.argv Historically, the previous code made sense, as Py_EncodeLocale() and fs.fsencode() could possibly use different encodings. However, this is not the case anymore for Python 3.2, which uses the locale encoding as the filesystem encoding (this is not true for later Python versions, but see below). See https://vstinner.github.io/painful-history-python-filesystem-encoding.html for a source and more background information. Using os.fsencode() is safer, as the documentation for sys.argv says that it can be used to get the original bytes. When doing further changes, the Python developers will take care that this continues to work. One concrete case where os.fsencode() is more correct is when enabling Python's UTF-8 mode. Py_DecodeLocale() will use UTF-8 in this case. Our previous code would have encoded it using the locale encoding (which might be different), whereas os.fsencode() will encode it with UTF-8. Since we don’t claim to support the UTF-8 mode, this is not really a bug and the patch can go to the default branch. It might be a good idea to not commit this to the stable branch, as it could in theory introduce regressions.

merge with stable

perf: make `hg perfwrite` more flexible The more flexible command was used recently while finding a solution for a buffering bug (eventually fixed in f9734b2d59cc (the changeset description uses a different benchmark)). In comparison to the previous version, the new version is much more flexible. While using it, the focus was on testing small writes. For this reason, by default it calls ui.write() 100 times with a single byte plus one newline byte, for 100 lines. To get the previous behavior, run `hg perfwrite --nlines=100000 --nitems=1 --item='Testing write performance' --batch-line`.

chg: fix typo

copies: implement __repr__ on branch_copies for debugging Differential Revision: https://phab.mercurial-scm.org/D8650

phases: remove useless lookup of repo[rev].rev() in _retractboundary changectx object is no longer needed thanks to a54ee130210a.

ui: replace `self._ferr` with identical `dest` Originally, it was part of a larger change that was abandoned. IMHO it makes the code slightly cleaner and saves one attribute access, so I decided to send it anyway instead of throwing it away.

absorb: make it clear what happens when no input Differential Revision: https://phab.mercurial-scm.org/D8643

tests: add hghave rule 'setprocname' to check if osutil.setprocname and use it setprocname is not present when we build a pure version. This leads to output changes in test-chg.t This should make test suite green on Python 2 pure build with chg. Differential Revision: https://phab.mercurial-scm.org/D8638

util: flush stderr explicitly after using warnings.warn() Due to some unknown reasons, when using chg with python3, the warnings.warn() output is not flushed. Fixes test-devel-warnings.t on py3 with chg.

phases: improve performance of _retractboundary The old version repeatedly converts nodes to revisions, which is a moderately expensive operation. Mapping all new changes once to revisions and back at the end reduces the time spend in _retractboundary during the unbundling of NetBSD's src from 67s to 17s. Differential Revision: https://phab.mercurial-scm.org/D8641

tests: use proctutil.stdout.write() instead of print() in test-extension.t I was debugging this test failure on python3 + chg. I get the following hunk as test failure: ``` @@ -206,6 +206,18 @@ Check normal command's load order of ext 4) bar uipopulate 5) foo reposetup 5) bar reposetup + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 5) foo reposetup (chg !) + 5) bar reposetup (chg !) 0:c24b9ac61126 ``` After hours of debugging and head scracthing, I figured out that something is wrong with output flushing. I initially switched the print() statements to ui.warn() but thanks to Yuya who suggested using procutil.stdout.write() instead.

fuzz: tell manifest fuzzer about longer node hashes Differential Revision: https://phab.mercurial-scm.org/D8374

absorb: preserve changesets which were already empty Most commands in Mercurial (commit, rebase, absorb itself) don’t create empty changesets or drop them if they become empty. If there’s a changeset that’s empty, it must be a deliberate choice of the user. At least it shouldn’t be absorb’s responsibility to prune them. The fact that changesets that became empty during absorb are pruned, is unaffected by this. This case was found while writing patches which make it possible to configure absorb and rebase to not drop empty changesets. Even without having such config set, I think it’s valuable to preserve changesets which were already empty.

absorb: preserve branch-closing changesets even if empty This makes the behavior consistent with 'hg commit', which allows to create otherwise empty changesets if they close the branch. A lost branch closure can inadvertently re-open a branch, so it should be preserved.

absorb: preserve branch-changing changesets even if empty This makes the behavior consistent with 'hg commit', which allows to create otherwise empty changesets if the branch changes compared to the parent. A branch change can denote important information, so it should be preserved.

rhg: add Command trait for subcommands implemented by rhg Normalizes the interface of the cli's commands Differential Revision: https://phab.mercurial-scm.org/D8611 Differential Revision: https://phab.mercurial-scm.org/D8648

rhg: add rhg crate The goal of rhg is to speedup some of hg's commands when possible by bypassing python entirely for the time being. It is by no means a replacement for hg as it will not support extentions or configuration and implement only a subset of hg's commands and options. Only use rhg if you understand what the tradeoffs are. Differential Revision: https://phab.mercurial-scm.org/D8610

hg-core: add FindRoot operation to find repository root path Differential Revision: https://phab.mercurial-scm.org/D8609

hg-core: add Operation interface for high-level hg operations A distinction is made between operations and commands. An operation is a high-level function of mercurial whereas a command is what is exposed by the cli. A single command can use several operations to achieve its goal. Differential Revision: https://phab.mercurial-scm.org/D8608

rust: do a clippy pass This is the result of running `cargo clippy` on hg-core/hg-cpython and fixing the lints that do not require too much code churn (and would warrant a separate commit/complete refactor) and only come from our code (a lot of warnings in hg-cpython come from `rust-cpython`). Most of those were good lints, two of them was the linter not being smart enough (or compiler to get up to `clippy`'s level depending on how you see it). Maybe in the future we could have `clippy` be part of the CI. Differential Revision: https://phab.mercurial-scm.org/D8635

fuzz: add config knob for PYTHON_CONFIG_FLAGS I'll clean this up once we get oss-fuzz to use Python 3.8 instead of 2.7, but for now we need a way to evolve the flags passed to python-config in lockstep with the Python version. Yuck. Differential Revision: https://phab.mercurial-scm.org/D8637

pyutil: this has taken so long to fix, I'm using 3.8 now Differential Revision: https://phab.mercurial-scm.org/D8636

merge with stable

tests: remove unused creation of file and outdated text It was forgotten to remove this in fb0de0bcd297.

tests: skip pyflakes for mercurial/thirdparty/ The current version of pyflakes (2.2.0) correctly detects one issue: mercurial/thirdparty/selectors2.py:335:40 '...'.format(...) has unused arguments at position(s): 1 But we're not interested in fixing lint errors in third-party code, so we need to exclude at least selectors2.py. And in the discussion for this patch it was decided to just skip the entire thirdparty directory. Differential Revision: https://phab.mercurial-scm.org/D8619

git: add debug logging when there's a mismatch in the cached heads list The dag rebuild can be expensive, so let's try and avoid bugs where it transparently rebuilds all the time for no reason. This would have prevented the issue fixed in D8622. Differential Revision: https://phab.mercurial-scm.org/D8625

py3: make stdout line-buffered if connected to a TTY Status messages that are to be shown on the terminal should be written to the file descriptor before anything further is done, to keep the user updated. One common way to achieve this is to make stdout line-buffered if it is connected to a TTY. This is done on Python 2 (except on Windows, where libc, which the CPython 2 streams depend on, does not properly support this). Python 3 rolls it own I/O streams. On Python 3, buffered binary streams can't be set line-buffered. The previous code (added in 227ba1afcb65) incorrectly assumed that on Python 3, pycompat.stdout (sys.stdout.buffer) is already line-buffered. However the interpreter initializes it with a block-buffered stream or an unbuffered stream (when the -u option or the PYTHONUNBUFFERED environment variable is set), never with a line-buffered stream. One example where the current behavior is unacceptable is when running `hg pull https://www.mercurial-scm.org/repo/hg` on Python 3, where the line "pulling from https://www.mercurial-scm.org/repo/hg" does not appear on the terminal before the hg process blocks while waiting for the server. Various approaches to fix this problem are possible, including: 1. Weaken the contract of procutil.stdout to not give any guarantees about buffering behavior. In this case, users of procutil.stdout need to be changed to do enough flushes. In particular, 1. either ui must insert enough flushes for ui.write() and friends, or 2. ui.write() and friends get split into flushing and fully buffered methods, or 3. users of ui.write() and friends must flush explicitly. 2. Make stdout unbuffered. 3. Make stdout line-buffered. Since Python 3 does not natively support that for binary streams, we must implement it ourselves. (2.) is problematic because using unbuffered I/O changes the performance characteristics significantly compared to line-buffered (which is used on Python 2) and this would be a regression. (1.2.) and (1.3) are a substantial amount of work. It’s unclear whether the added complexity would be justified, given that raw performance doesn’t matter that much when writing to a terminal much faster than the user could read it. (1.1.) pushes complexity into the ui class instead of separating the concern of how stdout is buffered. Other users of procutil.stdout would still need to take care of the flushes. This patch implements (3.). The general performance considerations are very similar to (1.1.). The extra method invocation and method forwarding add a little more overhead if the class is used. In exchange, it doesn’t add overhead if not used. For the benchmarks, I compared the previous implementation (incorrect on Python 3), (1.1.), (3.) and (2.). The command was chosen so that the streams were configured as if they were writing to a TTY, but actually write to a pager, which is also the default: HGRCPATH=/dev/null python3 ./hg --cwd ~/vcs/mozilla-central --time --pager yes --config pager.pager='cat > /dev/null' status --all previous: time: real 7.880 secs (user 7.290+0.050 sys 0.580+0.170) time: real 7.830 secs (user 7.220+0.070 sys 0.590+0.140) time: real 7.800 secs (user 7.210+0.050 sys 0.570+0.170) (1.1.) using Yuya Nishihara’s patch: time: real 9.860 secs (user 8.670+0.350 sys 1.160+0.830) time: real 9.540 secs (user 8.430+0.370 sys 1.100+0.770) time: real 9.830 secs (user 8.630+0.370 sys 1.180+0.840) (3.) using this patch: time: real 9.580 secs (user 8.480+0.350 sys 1.090+0.770) time: real 9.670 secs (user 8.480+0.330 sys 1.170+0.860) time: real 9.640 secs (user 8.500+0.350 sys 1.130+0.810) (2.) using a previous patch by me: time: real 10.480 secs (user 8.850+0.720 sys 1.590+1.500) time: real 10.490 secs (user 8.750+0.750 sys 1.710+1.470) time: real 10.240 secs (user 8.600+0.700 sys 1.590+1.510) As expected, there’s no difference on Python 2, as exactly the same code paths are used: previous: time: real 6.950 secs (user 5.870+0.330 sys 1.070+0.770) time: real 7.040 secs (user 6.040+0.360 sys 0.980+0.750) time: real 7.070 secs (user 5.950+0.360 sys 1.100+0.760) this patch: time: real 7.010 secs (user 5.900+0.390 sys 1.070+0.730) time: real 7.000 secs (user 5.850+0.350 sys 1.120+0.760) time: real 7.000 secs (user 5.790+0.380 sys 1.170+0.710)

simplemerge: rewrite flag merging loop as expression I feel binary operations are more readable.

simplemerge: leverage pycompat function to convert byte string to set

simplemerge: fix function name that tests if ctx is not null revision

git: decode node IDs back into Python strings (issue6349) db.text_factory = bytes, so the database contains only strings. The object IDs we get from pygit2 are Python strings. b'foo' != 'foo' This change allows the "don't reindex" optimization to work by allowing the "cur_cache_heads == cache_heads" comparison a few lines down to succeed. Differential Revision: https://phab.mercurial-scm.org/D8622

phabricator: make it clear what happen when no response Differential Revision: https://phab.mercurial-scm.org/D8621

tests: make it clear what happen when no response entered Differential Revision: https://phab.mercurial-scm.org/D8620

localrepo: handle ValueError during repository opening Python 3.8 can raise ValueError on attempt of an I/O operation against an illegal path. This was causing test-remotefilelog-gc.t to fail on Python 3.8. This commit teaches repository opening to handle ValueError and re-raise an Abort on failure. An arguably better solution would be to implement this logic in the vfs layer. But that seems like a bag of worms and I don't want to go down that rabbit hole. Until users report uncaught ValueError exceptions in the wild, I think it is fine to patch this at the only occurrence our test harness is finding it. Differential Revision: https://phab.mercurial-scm.org/D7944

metadata: filter the `removed` set to only contains relevant data The `files` entry can be bogus and contains too many entries. This can badly combines with the computation of `removed` inflating the set size. The can lead to the changesets centric rename computation to process much more data than needed, slowing it down (and increasing space taken by data storage). In practice newer commits already that reduced set, this applies this "fix" to older changeset. Differential Revision: https://phab.mercurial-scm.org/D8589

files: extract code for extra filtering of the `removed` entry into copies We want to reduce the set of `removed` files that to the set of files actually removed. That `removed` set is used as of the changeset centric algorithm, having smaller sets means less processing and faster computation. In this changeset we extract the code to be a function of it own. We will make use of it in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D8588

metadata: move computation related to files touched in a dedicated module This was suggested by Yuya Nishihara a while back. Since I am about to add more metadata related computation, lets create a new repositories. Differential Revision: https://phab.mercurial-scm.org/D8587

merge: move an inspection of the dirstate from record to calculate phase The intent is clearly to have `calculateupdates()` figure out what actions need to be taken and `recordupdates()` to make necessary modifications to the dirstate. However, in the `ACTION_PATH_CONFLICT_RESOLVE` case, there was one little inspection of copy information done in `recordupdates()`. This patch moves that to `calculateupdates()`. That will help with the next patch, which makes `merge.update()` work better with `overlayworkingctx` (copies should be recorded there too, even though we skip the `recordupdates()` step). Differential Revision: https://phab.mercurial-scm.org/D8615

tests: adjust to the new format in pyflakes output According to the pyflakes' NEWS.rst, the default output format changed recently: 2.2.0 (2020-04-08) - Include column information in error messages So the lines now read: contrib/perf.py:149:15 undefined name 'xrange' mercurial/hgweb/server.py:427:13 undefined name 'reload' mercurial/util.py:2862:24 undefined name 'file' Differential Revision: https://phab.mercurial-scm.org/D8618

tests: consistently use pyflakes as a Python module We check availability of pyflakes as a module, and also running it for real as a module. Only fair to test filterpyflakes.py working correctly when using pyflakes as a module too. Differential Revision: https://phab.mercurial-scm.org/D8617

heptapod-ci: also run tests for chg on python 2 I am not aware of any regular effort to run test with chg. So let's at least do it here. Differential Revision: https://phab.mercurial-scm.org/D8178

rust-dependencies: upgrade `micro-timer` dependency I wanted to to a tour of dependencies to upgrade, but only `micro-timer` has a new release which does not print when the function panics, which should be less misleading. Differential Revision: https://phab.mercurial-scm.org/D8605

context: fix creation of ProgrammingError to not use non-existent field Differential Revision: https://phab.mercurial-scm.org/D8606

help: explain in `hg help flags` that unambiguous prefixes are allowed I used `hg commit --amend` as an example because that's the most frequently used flag I could think of that doesn't yet have a short form. Differential Revision: https://phab.mercurial-scm.org/D8607

git: add coverage for manifest.diff() so we don't regress

git: implement diff manifest method This makes 'hg diff' work.

git: properly visit child tree objects when resolving a path

git: don't yield paths for directories when walking

git: correctly check for type of object when walking

git: implement stub prefetch_parents dirstate method A recent change (35b255e474d9) introduced this new required dirstate method but didn't update the git extension.

rust: leverage .expect() in place of .unwrap() + inline comment For a better error indication in case we made a mistake.

rust: fix false comment about mpsc::Sender We need Sync to share the Sender reference across threads.

merge with stable

relnotes: advertize the possibility to use rust I think the rust work may have been mentioned in the release notes, but if so only in passing, and not as an invitation to try it out. I think the next version is a decent time to do this, because the rust doesn't come with performance regressions AFAIK, speeds up status noticeably when it applies, which is the case for most invocations of status, and doesn't have the undesirable restriction of regex around empty patterns anymore. I am cheating a bit, because I'm giving numbers for `hg status` in mozilla-central, but they have one hgignore pattern that uses lookaround, ".vscode/(?!extensions\.json|tasks\.json", which I took out as it would cause a fallback to python when unknown files are requested. But it seems that they could express their hgignore differently if they were so inclined. Not sure if there are limitation other than linux-only that I am not thinking of but would be worth mentioning upfront, to avoid disappointing users? Differential Revision: https://phab.mercurial-scm.org/D8604

rust: add a pointer for profiling to the README As figuring out how to get useful profiles is not obvious. Differential Revision: https://phab.mercurial-scm.org/D8603

rust: update the mention of hgcli in rust/README.rst This may not be exactly right, but it's better than before. Differential Revision: https://phab.mercurial-scm.org/D8602

sslutil: fix comment to use inclusive or instead of exclusive or The incorrect "either" was introduced by one of my recent patches.

sslutil: propagate return value ssl.PROTOCOL_SSLv23 from protocolsettings() Also, protocolsettings() was renamed to commonssloptions() to reflect that only the options are returned.

sslutil: stop storing protocol and options for SSLContext in settings dict Call protocolsettings() where its return values are needed.

sslutil: rename 'minimumprotocolui' -> 'minimumprotocol' Before, both 'minimumprotocolui' and 'minimumprotocol' were used, but meaning the same.

sslutil: properly detect which TLS versions are supported by the ssl module For the record, I contacted the CPython developers to remark that unconditionally defining ssl.PROTOCOL_TLSv1_1 / ssl.PROTOCOL_TLSv1_2 is problematic: https://github.com/python/cpython/commit/6e8cda91d92da72800d891b2fc2073ecbc134d98#r39569316

sslutil: remove dead code (that failed if only TLS 1.0 is available) We ensure in setup.py that TLS 1.1 or TLS 1.2 is present.

config: remove unused hostsecurity.disabletls10warning config

sslutil: remove dead code (that downgraded default minimum TLS version) We ensure in setup.py that TLS 1.1 or TLS 1.2 is present.

sslutil: remove comment referring to unsupported legacy stacks

setup: require that Python has TLS 1.1 or TLS 1.2 This ensures that Mercurial never downgrades the minimum TLS version from TLS 1.1+ to TLS 1.0+ and enables us to remove that compatibility code. It is reasonable to expect that distributions having Python 2.7.9+ or having backported modern features to the ssl module (which we require) have a OpenSSL version supporting TLS 1.1 or TLS 1.2, as this is the main reason why distributions would want to backport these features. TLS 1.1 and TLS 1.2 are often either both enabled or both not enabled. However, both can be disabled independently, at least on current Python / OpenSSL versions. For the record, I contacted the CPython developers to remark that unconditionally defining ssl.PROTOCOL_TLSv1_1 / ssl.PROTOCOL_TLSv1_2 is problematic: https://github.com/python/cpython/commit/6e8cda91d92da72800d891b2fc2073ecbc134d98#r39569316

sslutil: check for OpenSSL without TLS 1.0 support in one case It can only happen if supportedprotocols gets fixed to contain only correct items (see the FIXME above in the file).

sslutil: don't set minimum TLS version to 1.0 if 1.2 but not 1.1 is available This case isn't very likely, but possible, especially if supportedprotocols gets fixed to contain only correct items (see the FIXME above in the file).

sslutil: add FIXME about supportedprotocols possibly containing too many items

sslutil: fix names of variables containing minimum protocol strings When working in this module, I found it very confusing that "protocol" as a variable name could mean either "minimum protocol string" or an exact version (as a string or ssl.PROTOCOL_* value). This patch prefixes variables of the former type with "minimum".

sslutil: stop returning argument as third return value of protocolsettings() The third return value was always the same as the argument.

relnotes: note that we now require modern SSL/TLS features in Python

tests: stop checking for optional, now impossible output In 7dd63a8cb1ee, the code that could output that line was removed.

rust: remove one more occurrence of re2 Differential Revision: https://phab.mercurial-scm.org/D8601

scmutil: clarify getuipathfn comment Differential Revision: https://phab.mercurial-scm.org/D8600

githelp: add some minimal help for pickaxe functionality Due to a conversation in work chat, I realized this is actually pretty well-hidden in Mercurial. Differential Revision: https://phab.mercurial-scm.org/D8590

rust: remove duplicate import Differential Revision: https://phab.mercurial-scm.org/D8456

tests: remove "sslcontext" check Now that we require the presence of ssl.SSLContext in setup.py, the check would always return `True`.

sslutil: eliminate `_canloaddefaultcerts` by constant-folding code using it

tests: remove "defaultcacerts" check `sslutil._canloaddefaultcerts` is always true (and will be removed).

sslutil: eliminate `modernssl` by constant-folding code using it

hgweb: avoid using `sslutil.modernssl` `sslutil.modernssl` is going to be removed. Since the point of using this attribute was to check the importability of the `sslutil`, a different attribute can be used. `sslutil.wrapserversocket` is used because it’s anyway used a few lines below.

sslutil: remove comments referring to removed SSLContext emulation class

sslutil: remove code checking for presence of ssl.SSLContext Now that we require the presence of ssl.SSLContext in setup.py, we can remove this code.

setup: require a Python version with modern SSL features This increases the minimum security baseline of Mercurial and enables us to remove compatibility code for supporting older, less secure Python versions.

sslutil: set `_canloaddefaultcerts` to `True` if `ssl.SSLContext` is present The `load_default_certs()` method was already present when `ssl.SSLContext` was backported to Python 2.7 (https://hg.python.org/cpython/rev/221a1f9155e2).

filemerge: add __bytes__ for absentfilectx This will at _least_ aid some upcoming debugging. Differential Revision: https://phab.mercurial-scm.org/D8592

mergestate: move staticmethod _filectxorabsent to module level I suspect this was a static method just because it made merge.py feel less messy, but now we have a mergestate package so we can do better. Differential Revision: https://phab.mercurial-scm.org/D8591

rust: remove support for `re2` With the performance issues with `regex` figured out and fixed in previous patches and `regex` newly gaining support for empty alternations, there is no reason to keep `re2` around anymore. It's only *marginally* faster at creating the regex which saves at most a couple of ms, but gets beaten by `regex` in every other aspect. This removes the Rust/C/C++ bridge (hooray!), the `with-re2` feature, the conditional code that goes with it, the documentation and relevant part of the debug/module output. Differential Revision: https://phab.mercurial-scm.org/D8594

rust-dependencies: update `regex` to 1.3.9 Version `1.3.8` introduces support for empty alternations, which makes previously disallowed patterns usable in `regex`. From a user's perspective, this means that glob patterns like `*.py{,c}` will no longer generate an "invalid" regex and will use the Rust path. `1.3.9` is a bugfix release, might as well update to the latest one. Differential Revision: https://phab.mercurial-scm.org/D8593

cleanup: remove compatibility code for Python < 2.7.4 The minimum supported Python version was recently raised to 2.7.4.

cleanup: eliminate procutil.quotecommand() After some compatibility code was removed, the function was the identity function on all platforms.

cleanup: remove compatibility code for Python < 2.7.1 The minimum supported Python version was recently raised to 2.7.4.

grep: reduce the cost of pathauditor checks when grepping working copy Running `time hg grep zxczxczxczxczxc -l` on mozilla-central: before: real 0m20,000s user 0m15,796s sys 0m4,189s after: real 0m10,903s user 0m8,964s sys 0m1,916s if vfs didn't call pathauditor at all: real 0m7,781s user 0m5,968s sys 0m1,790s Differential Revision: https://phab.mercurial-scm.org/D8582

grep: test that paths get audited Differential Revision: https://phab.mercurial-scm.org/D8581

grep: add test coverage of behavior on symlinks Differential Revision: https://phab.mercurial-scm.org/D8580

help: fix description of revlog version 2 Differential Revision: https://phab.mercurial-scm.org/D8576

files: speed up `hg files` when no flags change display It's not the first time I see slowness from this command slow down tools built on top of hg. The majority of the time is spent merely printing the result before this change, which is clearly not how it should be (especially since the computation of the result also looks slow). Running `hg files` in mozilla-central: parent revision: 1,260s this commit: 0,683s this commit without batching ui.write: 0,931s this commit replacing the body of the loop with `pass`: 0,566s This looks like a prime candidate for a rust fast path, but until then, it seems reasonable to optimize the python. Differential Revision: https://phab.mercurial-scm.org/D8586

sshpeer: make client print (likely) server errors on stderr (BC) so `hg clone -q` or `hg pull -q` don't print `abort: no suitable response from remote hg!` with no indication of what went wrong. There are other errors still silenced by -q (like failing to push due to a server hook), but the current change covers a good fraction of the problem (all errors setting up the ssh connection, no such remote repository, no access to the repository). Differential Revision: https://phab.mercurial-scm.org/D8584

sshpeer: add test showing that -q silences remote errors Differential Revision: https://phab.mercurial-scm.org/D8583

scmutil: speed up relativization of paths when it's a no-op Running commands from the root is commmon, in particular for automation. Running `hg files > /tmp/a` from the root of mozilla-central on linux: before: real 0m1,510s user 0m1,387s sys 0m0,090s after: real 0m1,266s user 0m1,165s sys 0m0,073s (there are 280k paths, so this was costing ~1us per path somehow) Differential Revision: https://phab.mercurial-scm.org/D8585

context: implement mergestate() method This will let us have the mergestate storage be controlled by the context. In particular, for working contexts we should use the existing mergestate, but for overlay contexts it's inappropriate to drop files in .hg/merge. Differential Revision: https://phab.mercurial-scm.org/D8551

mergestate: split out merge state handling code from main merge module There's already some pretty reasonable encapsulation here, but I want to make the mergestate storage a property of the context so memctx instances can do a reasonable thing. This is the first step in a reshuffle to make that easier. Differential Revision: https://phab.mercurial-scm.org/D8550

tests: add coverage for repo.changelog.children() in the git extension Differential Revision: https://phab.mercurial-scm.org/D8548

tests: add coverage for repo.changelog.findmissing() in test-git-interop.t This at least does a basic test of the method. It's not super-complete, but it's better than the nothing we'd otherwise have. Differential Revision: https://phab.mercurial-scm.org/D8547

relnotes: add API change note per request in D8502 Differential Revision: https://phab.mercurial-scm.org/D8549

merge with stable

grep: grep the working copy faster `hg grep qqqq` in the mercurial repo: before: 0,859s after: 0,233s `hg grep somethingwithnomatch` in mozilla-central: before: 51s after: 19s This is probably also a tiny bug fix, because the code was looking up a node for filename `pfn` on a filelog for filename `fn`, which are most of the time the same filename, but don't have to be. Ignoring performance and the bug fix, the code should have the same behavior. Differential Revision: https://phab.mercurial-scm.org/D8545

grep: stop computing information for --diff when unnecessary This is one reason why `hg grep pattern` essentially does `hg cat -r . 'set:**'` inside. There is no speed improvement in this commit, because the rest of the code still greps data from filelog instead of working copy when possible. Differential Revision: https://phab.mercurial-scm.org/D8544

grep: don't go in an infinite loop when given empty regex Differential Revision: https://phab.mercurial-scm.org/D8543

grep: improve test coverage Differential Revision: https://phab.mercurial-scm.org/D8542

phabricator: avoid passing None to pycompat.fsdecode Differential Revision: https://phab.mercurial-scm.org/D8525

setup: stop asking cargo to spam It prints this kind of stuff by default: Compiling regex v1.3.6 Compiling rand v0.7.3 Compiling crossbeam-deque v0.7.3 which is way better than we ask for before this change, which is screen after screen of compilation commands and irrelevant warnings in crates we use. Differential Revision: https://phab.mercurial-scm.org/D8537

git: implement some changelog methods Differential Revision: https://phab.mercurial-scm.org/D8540

git: avoid looking-up parents for the null commit Differential Revision: https://phab.mercurial-scm.org/D8541

git: fix probable missing return Differential Revision: https://phab.mercurial-scm.org/D8539

rust: fix warning about unnecessary mut If there's a reason to use mut (like compability with older compilers), then we should stick `#[allow(unused_mut)]` on the declaration. Differential Revision: https://phab.mercurial-scm.org/D8538

upgrade: support upgrade and downgrade from persistent nodemap The requirements is now recognised and dealt with and the associated files properly handled. The persistent nodemap should be ready for usage in the field now. Differential Revision: https://phab.mercurial-scm.org/D8431

status: also support for `traversedir` callback in the Rust fast-path Repeating the performance numbers from the `hg-core` change: Running `hg clean/purge` on Netbeans' repo (100k files): ``` | No-op | 30% unknown -------------------------- Rust | 1.0s | 1.67s C | 2.0s | 2.87s ``` Differential Revision: https://phab.mercurial-scm.org/D8520

rust-hg-cpython: update status bridge with the new `traversedir` support Differential Revision: https://phab.mercurial-scm.org/D8519

rust-status: collect traversed directories if required Some commands (`hg purge` notably) register the `traversedir` callback on their matcher to run said callback every time a directory is traversed. This is the first of three patches, further broadening Rust support for status. Unfortunately, there is no way around collecting a full `Vec` (or any other owned datastructure, like a radix tree) and pushing it back up the Python layer since keeping the Python callback in a closure would mean giving up multithreading because of the GIL, which is obviously unacceptable. Performance is still a lot better than the Python+C path. Running `hg clean/purge` on Netbeans' repo (100k files): ``` | No-op | 30% unknown -------------------------- Rust | 1.0s | 1.67s C | 2.0s | 2.87s ``` Differential Revision: https://phab.mercurial-scm.org/D8518

rust-status: don't dispatch unknown file when traversing if not listing unknowns This usually isn't a (functional) problem since we ignore the unknown files anyway, but when specifically using `hg purge`, unknown files were iterated over regardless of the option being true. This is both more correct and more efficient. Differential Revision: https://phab.mercurial-scm.org/D8517

status: update comment to reflect the more recent situation This is a gratuitous cleanup. Differential Revision: https://phab.mercurial-scm.org/D8516

hooks: provide access to transaction changes for internal hooks External hooks are skipped here as the environment often has a size limit in the low MBs and that can easily be reached by larger transactions. Differential Revision: https://phab.mercurial-scm.org/D8490

rust-regex: add test for verbatim regex syntax This makes sure it's not modified. Differential Revision: https://phab.mercurial-scm.org/D8508

rust-regex: prevent nonsensical `.*.*` pattern from happening Differential Revision: https://phab.mercurial-scm.org/D8507

rust-regex: fix issues with regex anchoring and performance It turns out that the way I tried to work around `regex`'s behavior difference with `re2` and Python's `re` was 1) buggy and 2) much more complicated than needed. In a few words: `regex` adds `.*` on either side of patterns when no start or end anchor is present. My previous workaround put `^` or `$` for every pattern, which is wrong even without the other 2 bugs on top of it. Using `^(?:<patterns>)` right at the end of the `regex` path fixes the issue. I've opened an issue to get a build option instead: https://github.com/rust-lang/regex/issues/675 Differential Revision: https://phab.mercurial-scm.org/D8506

diff: avoid going from contexts to nodes and back This will allow us to pass in-memory contexts that may not have a valid node to the diffing logic. Differential Revision: https://phab.mercurial-scm.org/D8503

cleanup: avoid extra node/ctx conversions in logcmdutil.diffordiffstat I'm about to write some code that wants to pass a memctx to diffordiffstat, but this feels like a meritorious cleanup anyway, since the first thing this method does is turn nodes into contexts, and most callers have a context handy. Differential Revision: https://phab.mercurial-scm.org/D8502

pyoxidizer: formatting bazel definitions This meets the Bazel style guide: https://docs.bazel.build/versions/master/skylark/bzl-style.html and was mostly done automatically with buildifier. Differential Revision: https://phab.mercurial-scm.org/D8521

revisions: parse "x123" as "nodeid starting with 123" without prefixhexnode `experimental.revisions.prefixhexnode` makes it so the template function `shortest()` uses an "x" prefix to disambiguate between short nodeids and revnums. That config has so far also been used for enabling parsing of "x123" unambiguously as a nodeid. That makes it a little annoying for people who have prefixhexnode=yes to share such nodeids with people who have prefixhexnode=no ("x123" will be considered invalid for them). There seems to be little harm in allowing that parsing for everyone. We still let e.g. bookmark names like "x123" take precedence over the nodeid, so that's not a concern. The only thing I can think of is that people get used to the "x" prefix being valid, making it impossible for us to change to a different prefix if we wanted to do that when graduating the feature. Differential Revision: https://phab.mercurial-scm.org/D8514

status: use cmdutil.check_at_most_one_arg() for checking --rev/--change There are apparently no tests for this either. Differential Revision: https://phab.mercurial-scm.org/D8511

diff: use cmdutil.check_at_most_one_arg() for checking --rev/--change The same check was done in extdiff as well, so I fixed that too. There are apparently no tests for this. Differential Revision: https://phab.mercurial-scm.org/D8510

copy: give better error message when no source paths found with --at-rev The new error message matches what we show when marking copies in the working copy. Differential Revision: https://phab.mercurial-scm.org/D8496

tests: show poor error message for `hg cp -A --at-rev . non-existent dst` Differential Revision: https://phab.mercurial-scm.org/D8495

copy: to find copy source, walk parent of revision we're marking copies in As shown in the previous patch, `hg cp --after --at-rev . src dst` fails if `src` is not in `.`. It seems obvious that you should always walk the *parent* of the revision you're marking copies in, but that's not how it was done for the working copy, and I didn't think to change it when marking copies in a non-working-copy commit. This patch fixes that by walking the parent commit instead, but only if we're marking copies for a non-working-copy commit. We need to leave the working-copy code unchanged because it depends on the weird behavior of `workingctx.walk()`. With these changes, there's very little overlap between the working-copy version and the non-working-copy version of `walkpats()`, but I've refrained from cleaning that up on the stable branch. Differential Revision: https://phab.mercurial-scm.org/D8494

tests: show that `hg cp -A --at-rev .` doesn't work for renames I clearly forgot to implement (and test) support for marking of renames when I added support for marking of copies :( Differential Revision: https://phab.mercurial-scm.org/D8493

rust-matchers: add TODO about incomplete `Display` for `IncludeMatcher` This is purely for future reference, I don't think this is a problem right now, since the `Display` is *only* used to ease debugging and has no real users. Differential Revision: https://phab.mercurial-scm.org/D8492

rust-filepatterns: match exact `rootglob`s with a `HashSet`, not in the regex This optimization yields some very interesting results in `rootglob`-heavy repositories. I build a test repository of the following structure: ``` root /<uuid>/build/empty_file ... repeat for 4000 entries ``` and a `.hgignore` containing the corresponding 4000 `rootglob` entries pointing to all `build/` folders. Rust+c `hg status` goes from 350ms down to 110ms. Differential Revision: https://phab.mercurial-scm.org/D8491

dirstate: force _checkexec to return a bool posix.checkexec can return True, False, or None. The rust status implementation expects a boolean, so make sure _checkexec returns a boolean. Differential Revision: https://phab.mercurial-scm.org/D8432

locking: wait for locks in `hg cp` and `hg mv` I traced this back to revision 1822 (64df422), and there's no explanation why we would prefer to error out instead of waiting for the locks. Differential Revision: https://phab.mercurial-scm.org/D8464

nodemap: teach `hg debugformat` about the persistent nodemap option We have a new requirement, we should display it. Differential Revision: https://phab.mercurial-scm.org/D8430

upgrade: support the --quiet flag The command is currently very verbose with a various bit of output being time sensitive or randomized. The make invocation bulky and hard to match in the test. We move various message from `ui.write` to `ui.status` in order for the `--quiet` flag to have effect and helps the situation. As a side benefit, we can replace the various redirection to > /dev/null with the --quiet flag. Differential Revision: https://phab.mercurial-scm.org/D8429

upgrade: clearly list optimisations This makes the command operation clearer. And this will be necessary to have a short version of this information with the next changesets, teaching `hg debugupgraderepo` about `--quiet`. Differential Revision: https://phab.mercurial-scm.org/D8428

nodemap: move the mode option to storage.revlog.nodemap.mode The option stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8422

nodemap: move the option for mmap usage to storage.revlog.nodemap.mmap The option is stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8421

nodemap: move and update the commend about persistence being experimental The comment was at the wrong place (on the developer option instead of the activation switch). So we move it at the right location and update it. Differential Revision: https://phab.mercurial-scm.org/D8420

nodemap: move the main switch to the `format` section The config to enable persistent nodemap is now `format.use-persistent-nodemap`. However the option remain marked as experimental because it only improve performance for people using the rust extensions. Differential Revision: https://phab.mercurial-scm.org/D8419

nodemap: drop the 'exp-' prefix for internal opener option The feature is now in a descent shape and we can consider having it "less" experimental. We won't be able to make it "totally" non-experimental, because its benefit rely on rust, which is totally experimental. Differential Revision: https://phab.mercurial-scm.org/D8418

nodemap: gate the feature behind a new requirement Now that the feature is working smoothly, a question was still open, should we gate the feature behind a new requirement or just treat it as a cache to be warmed by those who can and ignored by other. The advantage of using the cache approach is a transparent upgrade/downgrade story, making the feature easier to move to. However having out of date cache can come with a significant performance hit for process who expect an up to date cache but found none. In this case the file needs to be stored under `.hg/cache`. The "requirement" approach guarantee that the persistent nodemap is up to date. However, it comes with a less flexible activation story since an explicite upgrade is required. In this case the file can be stored in `.hg/store`. This wiki page is relevant to this questions: https://www.mercurial-scm.org/wiki/ComputedIndexPlan So which one should we take? Another element came into plan, the persistent nodemap use the `add` method of the transaction, it is used to keep track of a file content before a transaction in case we need to rollback it back. It turns out that the `transaction.add` API does not support file stored anywhere than `.hg/store`. Making it support file stored elsewhere is possible, require a change in on disk transaction format. Updating on disk file requires… introducing a new requirements. As a result, we pick the second option "gating the persistent nodemap behind a new requirements". Differential Revision: https://phab.mercurial-scm.org/D8417

nodemap: move on disk file to version 1 The current format contains the information we need, lets freeze it before the release. Differential Revision: https://phab.mercurial-scm.org/D8416

nodemap: add a new mode value, "strict" When "strict" is set, operation will refuse to use the slow path and abort if they would. This is useful for testing, benchmarking and server operation. Differential Revision: https://phab.mercurial-scm.org/D8415

nodemap: add a new mode option, with an optional "warn" value When "warn" is set, user will get notified when the slow code, used for compatibility is used. This can help people to detect situation were using that feature will give them a slowdown instead of a speedup. Differential Revision: https://phab.mercurial-scm.org/D8414

nodemap: also warm manifest nodemap with other caches The `hg debugupdatecache` command now also warm the persistent nodemap for the manifest (when applicable). Differential Revision: https://phab.mercurial-scm.org/D8411

nodemap: also use persistent nodemap for manifest The manifest as a different usage pattern than the changelog. First, while the lookup in changelog are not garanteed to match, the lookup in the manifest nodemap come from changelog and will exist in the manifest. In addition, looking up a manifest almost always result in unpacking a manifest an operation that rarely come cheap. Nevertheless, using a persistent nodemap provide a significant gain for some operations. For our measurementw, we use `hg cat --rev REV FILE` on the our reference mozilla-try. On this repository the persistent nodemap cache is about 29 MB in side for a total store side of 11,988 MB File with large history (file: b2g/config/gaia.json, revision: 195a1146daa0) no optimisation: 0.358s using mmap for index: 0.297s (-0.061s) persistent nodemap for changelog only: 0.275s (-0.024s) persistent nodemap for manifest too: 0.258s (-0.017s) File with small history (file: .hgignore, revision: 195a1146daa0) no optimisation: 0.377s using mmap for index: 0.296s (-0.061s) persistent nodemap for changelog only: 0.274s (-0.022s) persistent nodemap for manifest too: 0.257s (-0.017s) Same file but using a revision (8ba995b74e18) with a smaller manifest (3944829 bytes vs 10 bytes) no optimisation: 0.192s (-0.185s) using mmap for index: 0.131s (-0.061s) persistent nodemap for changelog only: 0.106s (-0.025s) persistent nodemap for manifest too: 0.087s (-0.019s) Differential Revision: https://phab.mercurial-scm.org/D8410