extensions: make `hg nonexistent` not crash with PyOxidizer When running `hg nonexistent`, we try to look for extensions that provide that command. We do that by looking for files in the `hgext.__file__` directory. However, PyOxidizer doesn't provide a `__file__`, so we crash when running with PyOxidizer. We should be able to look for the command in built-in extensions, but we seem to already have code for skipping the scan when running in a frozen binary, so I just modified that code instead. By the way, it also seems like we should be able to search for extensions in the `hgext3rd` module, but we don't do that yet either (before or after this patch). Differential Revision: https://phab.mercurial-scm.org/D8750

procutil: avoid use of deprecated tempfile.mktemp() In the previous version, I used tempfile.mktemp() because it seemed to be the only way to open a file from two processes (the Python documentation says the file backing NamedTemporaryFile can’t be opened a second time on Windows). However, it’s possible when passing the O_TEMPORARY flag to the second open. Source: https://stackoverflow.com/a/15235559/6366251

procutil: make _make_write_all() function private Because this function isn’t meant for general use (e.g., it’s Python 2-only), make in a module-private function by prefixing it with `_`.

dispatch: handle late KeyboardInterrupt occurred in run() User can press Ctrl+C while flushing streams in dispatch.run(). In such case, I think exiting with 255 is better than printing Python traceback and exiting with 1.

dispatch: indent run() function I'll add KeyboardInterrupt handling there.

debuginstall: don't translate encoding messages While Microsoft trained us to read mojibake text, it doesn't make sense to print mojibake messages when debugging the exact issue.

merge with stable

patch: refactor content diffing part in separate fn so extensions can wrap Right now extdiff uses it's own logic using archival to diff two versions of file using external diff tools. This makes the extdiff functionality non-extensible. This series is an attempt to refactor core patch and diff functionality so that extdiff can wrap and reuse it. This will help us in using external diffing tools at more places and not just extdiff command only then. Differential Revision: https://phab.mercurial-scm.org/D8685

tests: check that procutil.std{out,err}.write() returns correct result On Windows, we currently don’t fully test the case when the stream is connected to a TTY, but we test the child process side by connecting them to NUL, which is recognized as a TTY by Python. To make the large write test a bit more useful besides checking that it doesn’t crash, we can check that the write() method returns the correct result.

tests: add tests for when stdout or stderr is connected to `os.devnull` The original motivation was that creating PTYs on Windows is not possible, but `NUL` is recognized as a TTY, so we can have at least some test coverage for the TTY case. I think it doesn’t hurt to run the test cases on all systems.

procutil: ensure that procutil.std{out,err}.write() writes all bytes Python 3 offers different kind of streams and it’s not guaranteed for all of them that calling write() writes all bytes. When Python is started in unbuffered mode, sys.std{out,err}.buffer are instances of io.FileIO, whose write() can write less bytes for platform-specific reasons (e.g. Linux has a 0x7ffff000 bytes maximum and could write less if interrupted by a signal; when writing to Windows consoles, it’s limited to 32767 bytes to avoid the "not enough space" error). This can lead to silent loss of data, both when using sys.std{out,err}.buffer (which may in fact not be a buffered stream) and when using the text streams sys.std{out,err} (I’ve created a CPython bug report for that: https://bugs.python.org/issue41221). Python may fix the problem at some point. For now, we implement our own wrapper for procutil.std{out,err} that calls the raw stream’s write() method until all bytes have been written. We don’t use sys.std{out,err} for larger writes, so I think it’s not worth the effort to patch them.

procutil: move assignments This should probably be part of the previous patch, but folding it results in a less useful word diff, so I decided to keep it separate for review.

procutil: distribute code for stdout It makes sense to have the distinction between Python 2 and 3 at the top level, as we have to fight a different kind of battle on each: On Python 3, we get consistent behavior on all platforms, but need to create correctly-behaving binary streams. On Python 2, we have to account for platform differences.

convert: use bytes for value in extra dict The keys and values of the extra dict are expected to be bytes. Before this fix, there was a crash in mercurial.changelog.encodeextra().

overlayworkingctx: remove unused `nofilechanges()` and `_compact()` methods Differential Revision: https://phab.mercurial-scm.org/D8733

rebase: correctly check for empty commit in in-memory mode The new code has a small overhead in the empty commit case, as a `memctx` object is always created, but I don’t think it’s justified here to duplicate code to optimize a relatively unlikely code path. Differential Revision: https://phab.mercurial-scm.org/D8732

memctx: make `parents()` return list of one element if it’s not a merge It is part of the contract of `parents()` that non-merges return a list of one element. `self._parents` is always a list of two elements in `memctx`. Differential Revision: https://phab.mercurial-scm.org/D8731

commit: factor out empty commit check to `basectx.isempty()` This enables reuse in other places, e.g. those dealing with `memctx`. Differential Revision: https://phab.mercurial-scm.org/D8729

overlayworkingctx: rename misleadingly named `isempty()` method This method is only about whether there are file changes, not about whether the commit will be empty or not. One user of the method was incorrectly assuming the latter meaning, leading to the bug for which a test case was added in D8727. I’ve added a FIXME to the code. The original motivation for the rename was that I want to add `committablectx.isempty()`, that properly checks if a commit will be empty, using the exact same logic as in `repo.commit()`, and I wanted to avoid a name clash. Differential Revision: https://phab.mercurial-scm.org/D8728

rebase: add test case that shows that inmemory rebase does not preserve merges Differential Revision: https://phab.mercurial-scm.org/D8727

rebase: remove now unnecessary logic to allow empty commit when branch changes This was a workaround for a bug in the empty commit check in repo.commit(), where the parent branch name was incorrectly compared with the wdir branch name instead of the branch name passed via `extra`. The bug was fixed in D8724. The workaround was introduced in b2415e94b2f5. Differential Revision: https://phab.mercurial-scm.org/D8725

commit: check branch change on to-be-committed changeset This makes a difference when a branch name is passed via the `extra` argument, which will be used as the branch name for the created changeset. The empty commit check should use that branch name instead of whatever was set on the working directory. Besides fixing the bug for which a test case was extended, it enables to remove a workaround in the rebase extension, which will be done in a follow-up patch (D8725). Differential Revision: https://phab.mercurial-scm.org/D8724

merge with stable

procutil: move comment closer to relevant code

procutil: use mercurial.windows.winstdout only on Python 2 and TTYs Python 3 already works around the bug. The workaround is only needed when writing to consoles. If stdout is a console, sys.stdout.isatty() is true.

procutil: split if condition This prepares the code for subsequent changes where we need to differentiate between the two cases.

procutil: explain better why line buffering is not possible The sentence “On Python 3, buffered binary streams can't be set line-buffered.” was imprecise, as all streams are just Python classes and we can implement our own (which we did).

tests: make subprocess handling reusable for different tests in test-stdio.py

procutil: back out 8403cc54bc83 (make ....procutil.stderr unbuffered) Changeset 8403cc54bc83 introduced code that opens a second file object referring to the stderr file descriptor. This broke tests on Windows. The reason is that on Windows, sys.stderr is buffered and procutil.stderr closed the file descriptor when it got garbage collected before sys.stderr had the chance to flush buffered data. `procutil.stdout` had the same problem for a long time, but we didn’t realize, as in CI test runs, stdout is not a TTY and in this case no second file object is opened.

test-clone-uncompressed: use config to silence the lock warning The ui.timeout.warn option can solve the issue6237 in a cleaner way than what was done in 752da6863e39. That way, possible other unexpected error output will not be silenced. Differential Revision: https://phab.mercurial-scm.org/D8726

hooklib: fix detection of successors for changeset_obsoleted Provide a hook for obsutil.getobsolete to be used with either a transaction or the changes item of the transaction, since hooks only have access to the latter. Use that to find the correct list of revisions with obsmarkers, even new ones, and then filter out revisions with known successors. Move the processing from pretxnclose to txnclose as the transaction access itself is no longer necessary. This is more in line with notify and ensures that sanity checks can abort the transaction first. Differential Revision: https://phab.mercurial-scm.org/D8575

fix: prefetch file contents This prevents the worker subprocesses from contacting the server individually, which is either inefficient, or leads to problems if the connection is shared among them. Differential Revision: https://phab.mercurial-scm.org/D8723

fix: obtain base paths before starting workers This moves calculation of base paths to before work is dispatched. While this does mean that copy tracing will be serialized instead of parallel, it is necessary to be able to prefetch the base contents in a batch, which will likely be more efficient. Differential Revision: https://phab.mercurial-scm.org/D8722

scmutil: allowing different files to be prefetched per revision The old API takes a list of revision separate from the file matcher, and thus provides no way to fetch different sets of files from each revision. In preparation for adding one such usage, I'm changing the API to take a list of (revision, file matcher) tuples instead. Differential Revision: https://phab.mercurial-scm.org/D8721

tests: update test-devel-warnings.t output with chg

tests: make names in test-stdio.py more distinctive This way, more tests can be added without name clashes.

tests: terminate subprocess in test-stdio.py in case of exception If an error happened while reading the output of the subprocess, the pipe / TTY buffer can fill up and prevent that the subprocess ends. Therefore we should terminate the subprocess in case of an exception.

tests: proof test-stdio.py against buffer fill-up With the previous code, it could in theory happen that the pipe / PTY buffer of the child stdout / stderr fills up and the process never finishes. To prevent that, we read all of the stream before waiting for the end of the process. To ensure that the stream reaches EOF when the child finishes, we must close the parent "copy" of the child stdout / stderr.

tests: stablize test-serve.t on Windows I forget the reason that the subprocess on Windows doesn't print this, but all other instances of this are similarly conditionalized, so I didn't think too hard about it. Also, the server needs to be killed so it doesn't prevent the next run from working, especially since the port isn't randomized. Differential Revision: https://phab.mercurial-scm.org/D8720

commitctx: document the manifest writing function Same spirit as for `_filecommit` lets document the input and output before making any change or improvement. This is part of a larger refactoring/cleanup of the commitctx code to clarify and augment the logic gathering metadata useful for copy tracing. The current code is a tad too long and entangled to make such update easy. We start with easy and small cleanup. Differential Revision: https://phab.mercurial-scm.org/D8705

commitctx: reorder some conditional for efficiency in _filecommit Checking if a dict is empty will be faster than comparing text. I don't expect it to be a huge performance win, but still a good (but gratuitous) cleanup to do while we are at it. This is part of a larger refactoring/cleanup of the commitctx code to clarify and augment the logic gathering metadata useful for copy tracing. The current code is a tad too long and entangled to make such update easy. We start with easy and small cleanup. Differential Revision: https://phab.mercurial-scm.org/D8701

commitctx: document _filecommit input and output This is the beginning of a larger refactoring/cleanup of the commitctx code to clarify and augment the logic gathering metadata useful for copy tracing. The current code is a tad too long and entangled to make such update easy. We start with easy and small cleanup. Differential Revision: https://phab.mercurial-scm.org/D8699

phases: replace magic number by constant Differential Revision: https://phab.mercurial-scm.org/D8695

clonebundles: optional memory-requirement attribution The new REQUIREDRAM option allows a client to skip bundles it isn't expected to handle well, e.g. without swapping. This allows a fallback path to be provided e.g. using zstd level 10 instead of 22. Differential Revision: https://phab.mercurial-scm.org/D8645

util: provide a helper function to estimate RAM size For POSIX systems, it uses sysconf. For Windows, it uses the win32 API directly. Differential Revision: https://phab.mercurial-scm.org/D8644

tests: allow timing difference in output Differential Revision: https://phab.mercurial-scm.org/D8676

documentation: add some internals documentation about bid merge This is an important algorithm that was only documented on the wiki so far. Some update to the algorithm (and associated doc) is to expected in the future since the bid merge algorithm is bug-ridden when it comes to file deletion comes to play. Differential Revision: https://phab.mercurial-scm.org/D8711

cmdserver: document message-encodings and channel output options While writing the previous patch, I noticed these options are undocumented. In my testing, a separate status/error message channel works well in GUI frontend as we no longer have to sort out data and message from mixed outputs. So let's mark it as not experimental.

cmdserver: add option to not exit from message loop on SIGINT Sending SIGINT to server is the only way to interrupt a command running in command-server process. SIGINT will be caught at dispatch.dispatch() if we're lucky. Otherwise it will terminate the serer process. This is fundamentally unreliable as signals are delivered asynchronously. "cmdserver.shutdown-on-interrupt=False" mitigate the issue by making the server basically block SIGINT.

pycompat: remove pycompat.{stdin,stdout,stderr} All users have been changed to use procutil.{stdin,stdout,stderr}, which provide consistent behavior across platforms and Python versions.

pycompat: change users of pycompat.{stdin,stdout,stderr} to use procutil.std* On Python 3, pycompat.{stdin,stdout,stderr} are usually block-buffered even if connected to a TTY. procutil.{stdin,stdout,stderr} provide consistent behavior across platforms and Python versions.

hgignore: add mypy cache files Differential Revision: https://phab.mercurial-scm.org/D8692

revlog: avoid hard-coded hash sizes Differential Revision: https://phab.mercurial-scm.org/D8683

test: restore expected output in test-persistent-nodemap.t The merge 78cafd48b9b2 had a small error and wrongly dropped the output updated from one side of the merge. This changeset re-install it. Differential Revision: https://phab.mercurial-scm.org/D8678

heptapod-ci: build rhg before tests run Differential Revision: https://phab.mercurial-scm.org/D8682

rhg: add a limited `rhg root` subcommand Clap has been choosen for argument parsing for the following reasons: - it's a wildly used and maintained crate - it can deal with OS encoding making it suitable for any encoding - it supports nonambiguous prefix matching as already available in hg - it will soon allow for structopts-style declarative pattern instead of the currently used builder pattern Differential Revision: https://phab.mercurial-scm.org/D8613

rhg: add RootCommand using hg-core FindRoot operation to prepare `hg root` The println macro is not used to avoid string usage. Dealing only with bytes allows us to be compatible with any encoding and not just UTF8. Later on, format macro will be made to have more readable code. Differential Revision: https://phab.mercurial-scm.org/D8612

archival: abort if compression method is unavailable `tarfile.CompressionError` is documented to be the "exception for unavailable compression methods". Also, make tests conditional on whether the lzma module is available or not.

demandimport: ignore `lzma` module for demandimport This makes importing the module fail if the `_lzma` module is not present. This makes e.g. tarfile correctly recognize if LZMA support is not present. It changes the exception File "/usr/lib/python3.6/tarfile.py", line 1694, in xzopen fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset) AttributeError: module 'lzma' has no attribute 'LZMAFile' to the more correct exception File "/usr/lib/python3.6/tarfile.py", line 1692, in xzopen raise CompressionError("lzma module is not available") tarfile.CompressionError: lzma module is not available Also, it prevents that the error "abort: No module named '_lzma'!" is shown when a development warning is to be shown. The reason why that happened is that for showing the warning, we get information about the stack frames from the inspect module, which accesses the `__file__` attribute of all modules in `sys.modules` to build some cache, causing all modules (including `lzma`) to be imported.

merge with stable