commitctx: document _filecommit input and output This is the beginning of a larger refactoring/cleanup of the commitctx code to clarify and augment the logic gathering metadata useful for copy tracing. The current code is a tad too long and entangled to make such update easy. We start with easy and small cleanup. Differential Revision: https://phab.mercurial-scm.org/D8699

phases: replace magic number by constant Differential Revision: https://phab.mercurial-scm.org/D8695

clonebundles: optional memory-requirement attribution The new REQUIREDRAM option allows a client to skip bundles it isn't expected to handle well, e.g. without swapping. This allows a fallback path to be provided e.g. using zstd level 10 instead of 22. Differential Revision: https://phab.mercurial-scm.org/D8645

util: provide a helper function to estimate RAM size For POSIX systems, it uses sysconf. For Windows, it uses the win32 API directly. Differential Revision: https://phab.mercurial-scm.org/D8644

tests: allow timing difference in output Differential Revision: https://phab.mercurial-scm.org/D8676

documentation: add some internals documentation about bid merge This is an important algorithm that was only documented on the wiki so far. Some update to the algorithm (and associated doc) is to expected in the future since the bid merge algorithm is bug-ridden when it comes to file deletion comes to play. Differential Revision: https://phab.mercurial-scm.org/D8711

cmdserver: document message-encodings and channel output options While writing the previous patch, I noticed these options are undocumented. In my testing, a separate status/error message channel works well in GUI frontend as we no longer have to sort out data and message from mixed outputs. So let's mark it as not experimental.

cmdserver: add option to not exit from message loop on SIGINT Sending SIGINT to server is the only way to interrupt a command running in command-server process. SIGINT will be caught at dispatch.dispatch() if we're lucky. Otherwise it will terminate the serer process. This is fundamentally unreliable as signals are delivered asynchronously. "cmdserver.shutdown-on-interrupt=False" mitigate the issue by making the server basically block SIGINT.

pycompat: remove pycompat.{stdin,stdout,stderr} All users have been changed to use procutil.{stdin,stdout,stderr}, which provide consistent behavior across platforms and Python versions.

pycompat: change users of pycompat.{stdin,stdout,stderr} to use procutil.std* On Python 3, pycompat.{stdin,stdout,stderr} are usually block-buffered even if connected to a TTY. procutil.{stdin,stdout,stderr} provide consistent behavior across platforms and Python versions.

hgignore: add mypy cache files Differential Revision: https://phab.mercurial-scm.org/D8692

revlog: avoid hard-coded hash sizes Differential Revision: https://phab.mercurial-scm.org/D8683

test: restore expected output in test-persistent-nodemap.t The merge 78cafd48b9b2 had a small error and wrongly dropped the output updated from one side of the merge. This changeset re-install it. Differential Revision: https://phab.mercurial-scm.org/D8678

heptapod-ci: build rhg before tests run Differential Revision: https://phab.mercurial-scm.org/D8682

rhg: add a limited `rhg root` subcommand Clap has been choosen for argument parsing for the following reasons: - it's a wildly used and maintained crate - it can deal with OS encoding making it suitable for any encoding - it supports nonambiguous prefix matching as already available in hg - it will soon allow for structopts-style declarative pattern instead of the currently used builder pattern Differential Revision: https://phab.mercurial-scm.org/D8613

rhg: add RootCommand using hg-core FindRoot operation to prepare `hg root` The println macro is not used to avoid string usage. Dealing only with bytes allows us to be compatible with any encoding and not just UTF8. Later on, format macro will be made to have more readable code. Differential Revision: https://phab.mercurial-scm.org/D8612

merge with stable

procutil: make mercurial.utils.procutil.stderr unbuffered For most Mercurial code, it doesn’t make a difference, as the ui object flushes stderr explicitly (after the change, we could get rid of the explicit flush). One example where it makes a observable difference is mercurial.util.timed(). Without the patch, the time is not immediately shown on Python 3. With the patch, it’s shown immediately on all Python versions and platforms.

tests: generalize common test case code in test-stdio.py

procutil: make stdout line-buffered on Windows if connected to TTY Windows doesn’t support line buffering. Previously, we worked around that by setting the stream unbuffered. Instead, we can use our own line buffering we already use on Python 3.

procutil: split import and reassignment Besides making the code clearer, it will reduce the diff in the next patch.

procutil: move assignments closer to reassignments Doing reassignments is an anti-pattern IMHO, but I see how it makes sense here. When first looking at this code after jumping here with ctags, I missed the fact that stdout was reassigned. To make the code clearer, the assignments should be as close as possible to the reassignments.

procutil: factor out conditional creation of LineBufferedWrapper At the same time, document the logic and generalize it to work on all Python versions.

procutil: define LineBufferedWrapper on all Python versions There’s nothing Python 3-only about LineBufferedWrapper. In the future, we may want to use it on Windows, to work around missing line-buffering support.

tests: add tests for buffering behavior of mercurial.utils.procutil.stdout

cleanup: use slightly more meaningful name for temporary variable Not that it makes a big difference, but using `p` instead of `x` is clearer to me.

cleanup: use any() instead of checking truthiness of temporary list It was not immediately obvious to me, when first seeing this, why a list was created. It needed a second look to understand that the purpose was to check whether the condition is true for any of the parents. Using any() for that is clearer.

chg: suppress OSError in _restoreio() and add some logging (issue6330) According to issue6330, running chg on heavy loaded systems can lead to following error: ``` Traceback (most recent call last): File "path-to-hg/mercurial/commandserver.py", line 650, in _acceptnewconnection self._runworker(conn) File "path-to-hg/mercurial/commandserver.py", line 701, in _runworker prereposetups=[self._reposetup], File "path-to-hg/mercurial/commandserver.py", line 470, in _serverequest sv.cleanup() File "path-to-hg/mercurial/chgserver.py", line 381, in cleanup self._restoreio() File "path-to-hg/mercurial/chgserver.py", line 444, in _restoreio os.dup2(fd, fp.fileno()) OSError: [Errno 16] Device or resource busy ``` [man dup2] indicates that, on Linux, EBUSY comes from a race condition between open() and dup2(). However it's not clear why open() race occurred for newfd=stdin/out/err. We suppress the OSError in _restoreio() since the forked worker process will finish anyway and add some logging. Thanks to Mitchell Plamann for a detailed bug description and Yuya Nishihara for suggesting the fix.

ui: fix Python 2.7 support for ui.timestamp-output Differential Revision: https://phab.mercurial-scm.org/D8675

merge with stable

ui: add option to timestamp status and diagnostic messages Differential Revision: https://phab.mercurial-scm.org/D8640

merge with stable

compat: back out a25343d16ebe (initialize LC_CTYPE locale on all Python ...) As Yuya Nishihara pointed out, setting LC_CTYPE changes the behavior of some str methods on Python 2.

curses: back out d2227d4c9e6b (do not initialize LC_ALL to user settings) The changeset was based on a25343d16ebe, which will be backed out, too. Another fix for the problem will be resubmitted to the stable branch.

test: redirect stderr so warning messages don't change output (issue6237) clone and commit race for the lock, and if commit has to wait more than a second it prints a warning to stderr. Since this is somewhat expected here, silence it. Differential Revision: https://phab.mercurial-scm.org/D8664

locks: expect repo lock, not wlock, when writing to .hg/strip-backup/ There should be no need for a working copy lock when creating (or reading) bundles in `.hg/strip-backup/` since they don't affect the working copy. I noticed this because we have an extension that tries to strip some revisions while holding only a repo lock. I guess we have no such cases in core, which seems a bit surprising. Maybe we always take a wlock at a higher level so the working copy is not updated while the target commit is being stripped. Differential Revision: https://phab.mercurial-scm.org/D8666

graft: leverage cmdutil.check_incompatible_arguments() for --abort/--stop Differential Revision: https://phab.mercurial-scm.org/D8669

graft: leverage cmdutil.check_incompatible_arguments() for --no-commit Differential Revision: https://phab.mercurial-scm.org/D8668

graft: leverage cmdutil.check_at_most_one_arg() for --abort/--stop/--continue Differential Revision: https://phab.mercurial-scm.org/D8667

version: sort extensions by name in verbose mode External extensions can be assigned any name, but presumably most enabled extensions will be internal ones and having them sorted makes it easier to find specific ones if the list is long. The lists in `hg help extensions` are already sorted. Differential Revision: https://phab.mercurial-scm.org/D8671

crecord: stop trying to import wcurses The original import of crecord in 2008 already said "I have no idea if wcurses works with crecord...". The last reference to a Python package called wcurses is https://web.archive.org/web/20101025073658/http://adamv.com/dev/python/curses/. However, the Python package from there is called "curses" and not "wcurses". I didn’t find any evidence that it ever worked.

debian: support building a single deb for multiple py3 versions Around transitions from one python minor version to another (such as 3.7 to 3.8), the current packaging can be slightly problematic - it produces a `control` file that requires that the version of `python3` that's installed be exactly the one that was used on the build machine for the `mercurial` package, by containing a line like: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.8), python3 (>= 3.7~), python3:any (>= 3.5~) This is because it "knows" we only built for v3.7, which is the current default on my system. By building the native components for multiple versions, we can make it produce a line like this, which is compatible with 3.7 AND 3.8: Depends: sensible-utils, libc6 (>= 2.14), python3 (<< 3.9), python3 (>= 3.7~), python3:any (>= 3.5~) This isn't *normally* required, so I'm not making it the default. For those that receive their python3 and mercurial packages from their distro, and/or don't have to worry about a situation where the team that manages the python3 installation isn't the same as the team that manages the mercurial installation, this is probably not necessary. I chose the names `DEB_HG_*` because `DEB_*` is passed through `debuild` automatically (otherwise we'd have to explicitly allow the options through, which is a nuisance), and the `HG` part is to make it clear that this isn't a "standard" debian option that other packages might respect. Test Plan: 1. "nothing changed": - built a deb without these changes - built a deb with these changes but everything at the default - used diffoscope to compare, all differences were due to timestamps 2. "explicit is the same as implicit" (single version) - built a deb with everything at the default - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 3. "explicit is the same as implicit" (multi version) - built a deb with DEB_HG_MULTI_VERSION=1 - built a deb with DEB_HG_PYTHON_VERSIONS=3.7 - used diffoscope to compare, all differences were due to timestamps 4. (single version, 3.7) doesn't work with python3.8 - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` crashes 5. (multi version, 3.7 + 3.8) - `/usr/bin/python3.7 /usr/bin/hg debuginstall` works - `/usr/bin/python3.8 /usr/bin/hg debuginstall` works Differential Revision: https://phab.mercurial-scm.org/D8642

merge with stable

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Luckily this is already done at interpreter startup on modern Python versions and, since recently, by Mercurial in the pycompat module in all other cases.

compat: initialize LC_CTYPE locale on all Python versions and platforms Previously, the LC_CTYPE locale was not initialized according to user settings on all Python versions (e.g. never on Python 2) and platforms (e.g. not on some Python < 3.8 on Windows). This broke e.g. non-ASCII filenames passed to the Subversion bindings on Python 2, resulting in error messages like "file:///tmp/a%C3%A4 does not look like a Subversion repository to libsvn version 1.14.0". The following command could be used to test this functionality. Adding it to the test suite would be pointless, as the locale is always set to "C" during test runs. @command(b'check_initial_codeset', norepo=True) def check_initial_codeset(ui): codeset1 = locale.nl_langinfo(locale.CODESET) locale.setlocale(locale.LC_ALL, '') codeset2 = locale.nl_langinfo(locale.CODESET) assert codeset1 == codeset2

merge with stable

merge: don't grab wlock when merging in memory I noticed this because we have an internal extension that does an in-memory rebase while holding only a repo lock, which resulted in a developer warning about the working copy lock being taken after the repo lock. Differential Revision: https://phab.mercurial-scm.org/D8665

pycompat: use os.fsencode() to re-encode sys.argv Historically, the previous code made sense, as Py_EncodeLocale() and fs.fsencode() could possibly use different encodings. However, this is not the case anymore for Python 3.2, which uses the locale encoding as the filesystem encoding (this is not true for later Python versions, but see below). See https://vstinner.github.io/painful-history-python-filesystem-encoding.html for a source and more background information. Using os.fsencode() is safer, as the documentation for sys.argv says that it can be used to get the original bytes. When doing further changes, the Python developers will take care that this continues to work. One concrete case where os.fsencode() is more correct is when enabling Python's UTF-8 mode. Py_DecodeLocale() will use UTF-8 in this case. Our previous code would have encoded it using the locale encoding (which might be different), whereas os.fsencode() will encode it with UTF-8. Since we don’t claim to support the UTF-8 mode, this is not really a bug and the patch can go to the default branch. It might be a good idea to not commit this to the stable branch, as it could in theory introduce regressions.

merge with stable

perf: make `hg perfwrite` more flexible The more flexible command was used recently while finding a solution for a buffering bug (eventually fixed in f9734b2d59cc (the changeset description uses a different benchmark)). In comparison to the previous version, the new version is much more flexible. While using it, the focus was on testing small writes. For this reason, by default it calls ui.write() 100 times with a single byte plus one newline byte, for 100 lines. To get the previous behavior, run `hg perfwrite --nlines=100000 --nitems=1 --item='Testing write performance' --batch-line`.

chg: fix typo

copies: implement __repr__ on branch_copies for debugging Differential Revision: https://phab.mercurial-scm.org/D8650

phases: remove useless lookup of repo[rev].rev() in _retractboundary changectx object is no longer needed thanks to a54ee130210a.

ui: replace `self._ferr` with identical `dest` Originally, it was part of a larger change that was abandoned. IMHO it makes the code slightly cleaner and saves one attribute access, so I decided to send it anyway instead of throwing it away.

absorb: make it clear what happens when no input Differential Revision: https://phab.mercurial-scm.org/D8643

tests: add hghave rule 'setprocname' to check if osutil.setprocname and use it setprocname is not present when we build a pure version. This leads to output changes in test-chg.t This should make test suite green on Python 2 pure build with chg. Differential Revision: https://phab.mercurial-scm.org/D8638

util: flush stderr explicitly after using warnings.warn() Due to some unknown reasons, when using chg with python3, the warnings.warn() output is not flushed. Fixes test-devel-warnings.t on py3 with chg.

phases: improve performance of _retractboundary The old version repeatedly converts nodes to revisions, which is a moderately expensive operation. Mapping all new changes once to revisions and back at the end reduces the time spend in _retractboundary during the unbundling of NetBSD's src from 67s to 17s. Differential Revision: https://phab.mercurial-scm.org/D8641

tests: use proctutil.stdout.write() instead of print() in test-extension.t I was debugging this test failure on python3 + chg. I get the following hunk as test failure: ``` @@ -206,6 +206,18 @@ Check normal command's load order of ext 4) bar uipopulate 5) foo reposetup 5) bar reposetup + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 4) foo uipopulate (chg !) + 4) bar uipopulate (chg !) + 5) foo reposetup (chg !) + 5) bar reposetup (chg !) 0:c24b9ac61126 ``` After hours of debugging and head scracthing, I figured out that something is wrong with output flushing. I initially switched the print() statements to ui.warn() but thanks to Yuya who suggested using procutil.stdout.write() instead.

fuzz: tell manifest fuzzer about longer node hashes Differential Revision: https://phab.mercurial-scm.org/D8374

absorb: preserve changesets which were already empty Most commands in Mercurial (commit, rebase, absorb itself) don’t create empty changesets or drop them if they become empty. If there’s a changeset that’s empty, it must be a deliberate choice of the user. At least it shouldn’t be absorb’s responsibility to prune them. The fact that changesets that became empty during absorb are pruned, is unaffected by this. This case was found while writing patches which make it possible to configure absorb and rebase to not drop empty changesets. Even without having such config set, I think it’s valuable to preserve changesets which were already empty.

absorb: preserve branch-closing changesets even if empty This makes the behavior consistent with 'hg commit', which allows to create otherwise empty changesets if they close the branch. A lost branch closure can inadvertently re-open a branch, so it should be preserved.

absorb: preserve branch-changing changesets even if empty This makes the behavior consistent with 'hg commit', which allows to create otherwise empty changesets if the branch changes compared to the parent. A branch change can denote important information, so it should be preserved.

rhg: add Command trait for subcommands implemented by rhg Normalizes the interface of the cli's commands Differential Revision: https://phab.mercurial-scm.org/D8611 Differential Revision: https://phab.mercurial-scm.org/D8648