phabricator: avoid passing None to pycompat.fsdecode Differential Revision: https://phab.mercurial-scm.org/D8525

setup: stop asking cargo to spam It prints this kind of stuff by default: Compiling regex v1.3.6 Compiling rand v0.7.3 Compiling crossbeam-deque v0.7.3 which is way better than we ask for before this change, which is screen after screen of compilation commands and irrelevant warnings in crates we use. Differential Revision: https://phab.mercurial-scm.org/D8537

git: implement some changelog methods Differential Revision: https://phab.mercurial-scm.org/D8540

git: avoid looking-up parents for the null commit Differential Revision: https://phab.mercurial-scm.org/D8541

git: fix probable missing return Differential Revision: https://phab.mercurial-scm.org/D8539

rust: fix warning about unnecessary mut If there's a reason to use mut (like compability with older compilers), then we should stick `#[allow(unused_mut)]` on the declaration. Differential Revision: https://phab.mercurial-scm.org/D8538

upgrade: support upgrade and downgrade from persistent nodemap The requirements is now recognised and dealt with and the associated files properly handled. The persistent nodemap should be ready for usage in the field now. Differential Revision: https://phab.mercurial-scm.org/D8431

status: also support for `traversedir` callback in the Rust fast-path Repeating the performance numbers from the `hg-core` change: Running `hg clean/purge` on Netbeans' repo (100k files): ``` | No-op | 30% unknown -------------------------- Rust | 1.0s | 1.67s C | 2.0s | 2.87s ``` Differential Revision: https://phab.mercurial-scm.org/D8520

rust-hg-cpython: update status bridge with the new `traversedir` support Differential Revision: https://phab.mercurial-scm.org/D8519

rust-status: collect traversed directories if required Some commands (`hg purge` notably) register the `traversedir` callback on their matcher to run said callback every time a directory is traversed. This is the first of three patches, further broadening Rust support for status. Unfortunately, there is no way around collecting a full `Vec` (or any other owned datastructure, like a radix tree) and pushing it back up the Python layer since keeping the Python callback in a closure would mean giving up multithreading because of the GIL, which is obviously unacceptable. Performance is still a lot better than the Python+C path. Running `hg clean/purge` on Netbeans' repo (100k files): ``` | No-op | 30% unknown -------------------------- Rust | 1.0s | 1.67s C | 2.0s | 2.87s ``` Differential Revision: https://phab.mercurial-scm.org/D8518

rust-status: don't dispatch unknown file when traversing if not listing unknowns This usually isn't a (functional) problem since we ignore the unknown files anyway, but when specifically using `hg purge`, unknown files were iterated over regardless of the option being true. This is both more correct and more efficient. Differential Revision: https://phab.mercurial-scm.org/D8517

status: update comment to reflect the more recent situation This is a gratuitous cleanup. Differential Revision: https://phab.mercurial-scm.org/D8516

hooks: provide access to transaction changes for internal hooks External hooks are skipped here as the environment often has a size limit in the low MBs and that can easily be reached by larger transactions. Differential Revision: https://phab.mercurial-scm.org/D8490

rust-regex: add test for verbatim regex syntax This makes sure it's not modified. Differential Revision: https://phab.mercurial-scm.org/D8508

rust-regex: prevent nonsensical `.*.*` pattern from happening Differential Revision: https://phab.mercurial-scm.org/D8507

rust-regex: fix issues with regex anchoring and performance It turns out that the way I tried to work around `regex`'s behavior difference with `re2` and Python's `re` was 1) buggy and 2) much more complicated than needed. In a few words: `regex` adds `.*` on either side of patterns when no start or end anchor is present. My previous workaround put `^` or `$` for every pattern, which is wrong even without the other 2 bugs on top of it. Using `^(?:<patterns>)` right at the end of the `regex` path fixes the issue. I've opened an issue to get a build option instead: https://github.com/rust-lang/regex/issues/675 Differential Revision: https://phab.mercurial-scm.org/D8506

diff: avoid going from contexts to nodes and back This will allow us to pass in-memory contexts that may not have a valid node to the diffing logic. Differential Revision: https://phab.mercurial-scm.org/D8503

cleanup: avoid extra node/ctx conversions in logcmdutil.diffordiffstat I'm about to write some code that wants to pass a memctx to diffordiffstat, but this feels like a meritorious cleanup anyway, since the first thing this method does is turn nodes into contexts, and most callers have a context handy. Differential Revision: https://phab.mercurial-scm.org/D8502

pyoxidizer: formatting bazel definitions This meets the Bazel style guide: https://docs.bazel.build/versions/master/skylark/bzl-style.html and was mostly done automatically with buildifier. Differential Revision: https://phab.mercurial-scm.org/D8521

revisions: parse "x123" as "nodeid starting with 123" without prefixhexnode `experimental.revisions.prefixhexnode` makes it so the template function `shortest()` uses an "x" prefix to disambiguate between short nodeids and revnums. That config has so far also been used for enabling parsing of "x123" unambiguously as a nodeid. That makes it a little annoying for people who have prefixhexnode=yes to share such nodeids with people who have prefixhexnode=no ("x123" will be considered invalid for them). There seems to be little harm in allowing that parsing for everyone. We still let e.g. bookmark names like "x123" take precedence over the nodeid, so that's not a concern. The only thing I can think of is that people get used to the "x" prefix being valid, making it impossible for us to change to a different prefix if we wanted to do that when graduating the feature. Differential Revision: https://phab.mercurial-scm.org/D8514

status: use cmdutil.check_at_most_one_arg() for checking --rev/--change There are apparently no tests for this either. Differential Revision: https://phab.mercurial-scm.org/D8511

diff: use cmdutil.check_at_most_one_arg() for checking --rev/--change The same check was done in extdiff as well, so I fixed that too. There are apparently no tests for this. Differential Revision: https://phab.mercurial-scm.org/D8510

copy: give better error message when no source paths found with --at-rev The new error message matches what we show when marking copies in the working copy. Differential Revision: https://phab.mercurial-scm.org/D8496

tests: show poor error message for `hg cp -A --at-rev . non-existent dst` Differential Revision: https://phab.mercurial-scm.org/D8495

copy: to find copy source, walk parent of revision we're marking copies in As shown in the previous patch, `hg cp --after --at-rev . src dst` fails if `src` is not in `.`. It seems obvious that you should always walk the *parent* of the revision you're marking copies in, but that's not how it was done for the working copy, and I didn't think to change it when marking copies in a non-working-copy commit. This patch fixes that by walking the parent commit instead, but only if we're marking copies for a non-working-copy commit. We need to leave the working-copy code unchanged because it depends on the weird behavior of `workingctx.walk()`. With these changes, there's very little overlap between the working-copy version and the non-working-copy version of `walkpats()`, but I've refrained from cleaning that up on the stable branch. Differential Revision: https://phab.mercurial-scm.org/D8494

tests: show that `hg cp -A --at-rev .` doesn't work for renames I clearly forgot to implement (and test) support for marking of renames when I added support for marking of copies :( Differential Revision: https://phab.mercurial-scm.org/D8493

rust-matchers: add TODO about incomplete `Display` for `IncludeMatcher` This is purely for future reference, I don't think this is a problem right now, since the `Display` is *only* used to ease debugging and has no real users. Differential Revision: https://phab.mercurial-scm.org/D8492

rust-filepatterns: match exact `rootglob`s with a `HashSet`, not in the regex This optimization yields some very interesting results in `rootglob`-heavy repositories. I build a test repository of the following structure: ``` root /<uuid>/build/empty_file ... repeat for 4000 entries ``` and a `.hgignore` containing the corresponding 4000 `rootglob` entries pointing to all `build/` folders. Rust+c `hg status` goes from 350ms down to 110ms. Differential Revision: https://phab.mercurial-scm.org/D8491

dirstate: force _checkexec to return a bool posix.checkexec can return True, False, or None. The rust status implementation expects a boolean, so make sure _checkexec returns a boolean. Differential Revision: https://phab.mercurial-scm.org/D8432

locking: wait for locks in `hg cp` and `hg mv` I traced this back to revision 1822 (64df422), and there's no explanation why we would prefer to error out instead of waiting for the locks. Differential Revision: https://phab.mercurial-scm.org/D8464

nodemap: teach `hg debugformat` about the persistent nodemap option We have a new requirement, we should display it. Differential Revision: https://phab.mercurial-scm.org/D8430

upgrade: support the --quiet flag The command is currently very verbose with a various bit of output being time sensitive or randomized. The make invocation bulky and hard to match in the test. We move various message from `ui.write` to `ui.status` in order for the `--quiet` flag to have effect and helps the situation. As a side benefit, we can replace the various redirection to > /dev/null with the --quiet flag. Differential Revision: https://phab.mercurial-scm.org/D8429

upgrade: clearly list optimisations This makes the command operation clearer. And this will be necessary to have a short version of this information with the next changesets, teaching `hg debugupgraderepo` about `--quiet`. Differential Revision: https://phab.mercurial-scm.org/D8428

nodemap: move the mode option to storage.revlog.nodemap.mode The option stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8422

nodemap: move the option for mmap usage to storage.revlog.nodemap.mmap The option is stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8421

nodemap: move and update the commend about persistence being experimental The comment was at the wrong place (on the developer option instead of the activation switch). So we move it at the right location and update it. Differential Revision: https://phab.mercurial-scm.org/D8420

nodemap: move the main switch to the `format` section The config to enable persistent nodemap is now `format.use-persistent-nodemap`. However the option remain marked as experimental because it only improve performance for people using the rust extensions. Differential Revision: https://phab.mercurial-scm.org/D8419

nodemap: drop the 'exp-' prefix for internal opener option The feature is now in a descent shape and we can consider having it "less" experimental. We won't be able to make it "totally" non-experimental, because its benefit rely on rust, which is totally experimental. Differential Revision: https://phab.mercurial-scm.org/D8418

nodemap: gate the feature behind a new requirement Now that the feature is working smoothly, a question was still open, should we gate the feature behind a new requirement or just treat it as a cache to be warmed by those who can and ignored by other. The advantage of using the cache approach is a transparent upgrade/downgrade story, making the feature easier to move to. However having out of date cache can come with a significant performance hit for process who expect an up to date cache but found none. In this case the file needs to be stored under `.hg/cache`. The "requirement" approach guarantee that the persistent nodemap is up to date. However, it comes with a less flexible activation story since an explicite upgrade is required. In this case the file can be stored in `.hg/store`. This wiki page is relevant to this questions: https://www.mercurial-scm.org/wiki/ComputedIndexPlan So which one should we take? Another element came into plan, the persistent nodemap use the `add` method of the transaction, it is used to keep track of a file content before a transaction in case we need to rollback it back. It turns out that the `transaction.add` API does not support file stored anywhere than `.hg/store`. Making it support file stored elsewhere is possible, require a change in on disk transaction format. Updating on disk file requires… introducing a new requirements. As a result, we pick the second option "gating the persistent nodemap behind a new requirements". Differential Revision: https://phab.mercurial-scm.org/D8417

nodemap: move on disk file to version 1 The current format contains the information we need, lets freeze it before the release. Differential Revision: https://phab.mercurial-scm.org/D8416

nodemap: add a new mode value, "strict" When "strict" is set, operation will refuse to use the slow path and abort if they would. This is useful for testing, benchmarking and server operation. Differential Revision: https://phab.mercurial-scm.org/D8415

nodemap: add a new mode option, with an optional "warn" value When "warn" is set, user will get notified when the slow code, used for compatibility is used. This can help people to detect situation were using that feature will give them a slowdown instead of a speedup. Differential Revision: https://phab.mercurial-scm.org/D8414

nodemap: also warm manifest nodemap with other caches The `hg debugupdatecache` command now also warm the persistent nodemap for the manifest (when applicable). Differential Revision: https://phab.mercurial-scm.org/D8411

nodemap: also use persistent nodemap for manifest The manifest as a different usage pattern than the changelog. First, while the lookup in changelog are not garanteed to match, the lookup in the manifest nodemap come from changelog and will exist in the manifest. In addition, looking up a manifest almost always result in unpacking a manifest an operation that rarely come cheap. Nevertheless, using a persistent nodemap provide a significant gain for some operations. For our measurementw, we use `hg cat --rev REV FILE` on the our reference mozilla-try. On this repository the persistent nodemap cache is about 29 MB in side for a total store side of 11,988 MB File with large history (file: b2g/config/gaia.json, revision: 195a1146daa0) no optimisation: 0.358s using mmap for index: 0.297s (-0.061s) persistent nodemap for changelog only: 0.275s (-0.024s) persistent nodemap for manifest too: 0.258s (-0.017s) File with small history (file: .hgignore, revision: 195a1146daa0) no optimisation: 0.377s using mmap for index: 0.296s (-0.061s) persistent nodemap for changelog only: 0.274s (-0.022s) persistent nodemap for manifest too: 0.257s (-0.017s) Same file but using a revision (8ba995b74e18) with a smaller manifest (3944829 bytes vs 10 bytes) no optimisation: 0.192s (-0.185s) using mmap for index: 0.131s (-0.061s) persistent nodemap for changelog only: 0.106s (-0.025s) persistent nodemap for manifest too: 0.087s (-0.019s) Differential Revision: https://phab.mercurial-scm.org/D8410

nodemap: create files in the repository used in the test We need a manifest with more content to test persistent nodemap for manifest. This change the repository content and affect all the hashes. Differential Revision: https://phab.mercurial-scm.org/D8409

rust-matchers: add timing tracing to regex compilation This might be useful to diagnose later performance issues or just to show the difference between engines. Differential Revision: https://phab.mercurial-scm.org/D8498

merge with stable

merge with stable

copies: fix the changeset based algorithm regarding merge In 99ebde4fec99, we changed the list of files stored into the `files` field. This lead to the changeset centric copy algorithm to break in various merge situation involving merge. Older information could reach the merge through `p1`, and while information from `p2` was strictly fresher, it would get overwritten anyway. We update the situation with more details about which revision introduces rename information. This help use making the right decision in case of merge. We are now running a more comprehensive suite of test with include this kind of situation. The behavior differ slightly from the filelog based in a couple of instance. There is mostly two distinct cases: 1) there are conflicting rename information in a merge (different rename history on each side). In this case the filelog based implementation arbitrarily pick a side based on the file-revision-number. So it depends on a local factor. The changeset centric algorithm will use a deterministic approach, by picking the information coming from the first parent of the merge. This is stable across different clone. 2) rename information related to file that exist in both source and destination. The filelog based implementation do not even try to detect these, however the changeset centric one get them for "free" (it is simpler to detect them than not). The new implementation focus on correctness. Performance improvement will come later. Differential Revision: https://phab.mercurial-scm.org/D8244

merge with stable

rust-chg: clean up excessive indents Differential Revision: https://phab.mercurial-scm.org/D8450

rust-chg: do not terminate tokio runtime until pager exits We no longer need to spawn a task just to keep the pager handle. The pager handle can be held by ChgUiHandler since the handler itself is not consumed and recreated across async calls. Differential Revision: https://phab.mercurial-scm.org/D8449

rust-chg: modernize entry function Finally the entire build passes. There's a bug that run() no longer waits for the spawned pager, which will be fixed by the next patch. Differential Revision: https://phab.mercurial-scm.org/D8448

rust-chg: reimplement locator by using async/await and tokio-0.2 connect_spawned() is rewritten from scratch by using std::process. Before, it would select completion of either connection or server process. New code could be implemented as such, but it's much simpler to occasionally run try_wait() to detect server death. Differential Revision: https://phab.mercurial-scm.org/D8447

rust-chg: reimplement ChgClientExt as ChgClient wrapper ChgClient is no longer an extension trait because: a. Client object is not consumed and recreated in future-0.3 world, which unblocks writing a simple wrapper struct. b. async fn isn't allowed in trait. Overall, the API should become simpler. Differential Revision: https://phab.mercurial-scm.org/D8446

rust-chg: reimplement run_command operation as async function The crafted state machine is no longer needed thanks to async/await. The state machine is basically rewritten as follows: - Ready(..) -> return .. - PollAgain(..) -> run .. and await - Err(..) -> return Err(..) Differential Revision: https://phab.mercurial-scm.org/D8445

rust-chg: reimplement uihandler by using async-trait and tokio-0.2 We no longer have to consume self and arguments. Differential Revision: https://phab.mercurial-scm.org/D8444

rust-chg: have attach_io() simply take reference of AsRawFd object We no longer have to deal with the restriction of the Future type. Before, these file objects couldn't be references and that's the only reason why we had to make stderr an Option<T>. This fixes future type deduction issue of stderr = None, where rustc would complain that T of Option<T> couldn't be deduced. Differential Revision: https://phab.mercurial-scm.org/D8443

rust-chg: reimplement attach_io operation as async function In short, MessageLoop<Connection> was redesigned as Protocol<Connection>, and the protocol methods no longer consume self. API changes are briefly documented in the following page: https://docs.rs/tokio-hglib/0.3.0/tokio_hglib/struct.Protocol.html Differential Revision: https://phab.mercurial-scm.org/D8442

rust-chg: upgrade to futures-0.3 based libraries And do some trivial fixes: - BytesMut::put_u32_be() -> put_u32() - tokio_process -> tokio::process, CommandExt -> Command, spawn_async() -> spawn(), stdin() -> stdin - tokio_timer::sleep() -> tokio::time::delay_for() Differential Revision: https://phab.mercurial-scm.org/D8441

rust-chg: exclude futures-dependent modules from build and break things It's impractical to upgrade the codebase incrementally since futures 0.1 and 0.3 APIs are fundamentally different. So this patch temporarily excludes futures-dependent modules from the build. These modules will be upgraded and re-enabled one by one. Differential Revision: https://phab.mercurial-scm.org/D8440

commit: tell user what to do with .hg/last-message.txt I have always assumed that the message will be reused by the next `hg commit`, but it seems it's just silently dropped on the next commit. Let's try to be more helpful by telling the user that they have to manually tell hg to reuse it. The file will still be lost if the user runs some other operation in between (like a non-in-memory rebase). That will be fixed once we've switched all operations to be in-memory :) I didn't include `$(hg root)/` in the path in the message to the user because that would have made the message too long. Hopefully the user will figure that part out themselves. Differential Revision: https://phab.mercurial-scm.org/D8463

test-check-rust-format: specify --edition=2018 rustfmt doesn't read Cargo.toml unless it's executed by cargo. https://github.com/rust-lang/rustfmt#rusts-editions

convert: use bytes for value in extra dict The keys and values of the extra dict are expected to be bytes. Before this fix, there was a crash in mercurial.changelog.encodeextra().

archival: abort if compression method is unavailable `tarfile.CompressionError` is documented to be the "exception for unavailable compression methods". Also, make tests conditional on whether the lzma module is available or not.

demandimport: ignore `lzma` module for demandimport This makes importing the module fail if the `_lzma` module is not present. This makes e.g. tarfile correctly recognize if LZMA support is not present. It changes the exception File "/usr/lib/python3.6/tarfile.py", line 1694, in xzopen fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset) AttributeError: module 'lzma' has no attribute 'LZMAFile' to the more correct exception File "/usr/lib/python3.6/tarfile.py", line 1692, in xzopen raise CompressionError("lzma module is not available") tarfile.CompressionError: lzma module is not available Also, it prevents that the error "abort: No module named '_lzma'!" is shown when a development warning is to be shown. The reason why that happened is that for showing the warning, we get information about the stack frames from the inspect module, which accesses the `__file__` attribute of all modules in `sys.modules` to build some cache, causing all modules (including `lzma`) to be imported.

py3: fix crash when server address is 0.0.0.0 (issue6362) `socket.getfqdn()` assumes that the name is passed as `str` on Python 3 and always returns `str` in this case. Mercurial passed `bytes` (but still expected a `str` result), which worked by chance in many cases, except for e.g. b'0.0.0.0', which was returned unchanged, breaking later code. Instead of calling `socket.getfqdn()`, we can also use `self.server_name` from the base `HTTPServer` class, which already stores the FQDN of the locally-bound socket name (see `BaseHTTPServer.py` in the Python 2 stdlib and `http/server.py` in the Python 3 stdlib).

run-tests: find python binary on Python 3 (issue6361) Return strings from _findprogram as all callers expect unicode strings. Previously the check in _usecorrectpython agains sysexecutable was always false on Python 3. Differential Revision: https://phab.mercurial-scm.org/D8674

tests: ignore possible diagnostics from gpg 2.2 Differential Revision: https://phab.mercurial-scm.org/D8672

Added signature for changeset 0ea9c86fac89

Added tag 5.4.2 for changeset 0ea9c86fac89

convert: handle percent-encoded bytes in file URLs like Subversion 75b59d221aa3 added most of the code that gets removed by this patch. It helped making progress on Python 3, but the reasoning was wrong in many ways. I tried to retract it while it was queued, but it was too late. Back then, I was asssuming that what happened on Python 2 (preserving bytes) is correct and my Python 3 change is a hack. However it turned out that Subversion interprets percent-encoded bytes as UTF-8. Accepting the same format as Subversion is a good idea. Consistency with urlreq.pathname2url() (as described in the removed comment) doesn’t matter because that function is only used for passing paths to urllib. This is not a backwards-incompatible change because before 5c0d5b48e58c, non-ASCII filenames didn’t work at all on Python 2. When the locale encoding is ISO-8859-15, `svn` accepts `file:///tmp/a%E2%82%AC` for `/tmp/a€`. Before this patch, this was the case for this extension on Python 3, but not on Python 2. This patch makes it work like with `svn` on both Python 2 and Python 3.

convert: add docstring on convert.subversion.geturl() The function is unusual for a bytes-handling function in Mercurial because it can’t handle arbitrary bytes. Therefore we should document this fact. Pointed out by Yuya Nishihara while reviewing e3b19004087a.

tests: use path inside test dir This will make the diff for the next patch less noisy.

convert: convert URLs to UTF-8 for Subversion Preamble: for comprehension, note that the `path` of geturl() would better be called `path_or_url` (the argument of the call of getsvn() is called `url`). For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to ASCII. For file URLs, the reasoning is the same as for paths: we have to roundtrip with what Subversion is doing. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `file:///tmp/a€` failed before like this: file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0 Decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

convert: correctly convert paths to UTF-8 for Subversion The previous code using encoding.tolocal() only worked by chance in these situations: * The string is ASCII: The fast path was triggered and the string was returned unmodified. * The local encoding is UTF-8: The source and target encoding is the same. * The string is not valid UTF-8 and the native encoding is ISO-8859-1: If the string doesn’t decode using UTF-8, ISO-8859-1 is tried as a fallback. During `hg convert`, the local encoding is always UTF-8. The irony is that in this case, encoding.tolocal() behaves like what someone would expect the reverse function, encoding.fromlocal(), to do. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `/tmp/a€` failed before like this: file:///tmp/a%C2%A4 does not look like a Subversion repository to libsvn version 1.14.0 The correct URL is `file:///tmp/a%E2%82%AC`. Unlike previously (with the ISO-8859-1 fallback), decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.

py3: pass URL as str Before the patch, HTTP(S) URLs were never recognized as a Subversion repository on Python 3.

convert: bail out in Subversion source if encountering non-ASCII HTTP(S) URL Before this patch, in the tested case, urllib raised `httplib.InvalidURL: URL can't contain control characters. '/\xff/!svn/ver/0/.svn' (found at least '\xff')`, which resulted in that the URL was never recognized as a Subversion repository. This patch adds a check that bails out if the URL contains non-ASCII characters. The warning is not overly user-friendly, but giving the user something to type into a search engine is definitively better than not explaining why the repository was not recognized. We could support non-ASCII chracters by quoting them before passing them to urllib. However, we would want to be compatible with what the `svn` command does, which converts the URL from the locale encoding to UTF-8, percent-encodes it and sends it to the server. If the locale encoding is not UTF-8, the behavior is IMHO not very intuitive, as the `svn` command may send different (percent-encoded) octets than what was passed on the console. Instead of copying this behavior, we better leave it forbidden.

run-tests: fix escapes with conditions Before this fix, escapes with conditions in tests failed like this on Python 3: $ $PYTHON -c 'from mercurial.utils.procutil import stdout; stdout.write(b"\xff")' - \xff (no-eol) (esc) (true !) + \xff (no-eol) (esc) The unicode_escape encoding decodes br'\xff' to u'\xff'. To convert the first 256 code points to bytes with the same ordinal, the latin-1 encoding must be used. Escapes without conditions already worked before on Python 3, but not through `el == l` a few lines below the changed line in run-tests.py. I didn’t investigate further.

convert: set LC_CTYPE around calls to Subversion bindings The Subversion bindings require that LC_CTYPE is set. However, we don’t want to set it all the time, as it changes the behavior of str methods on Python 2. The taken approach is hopefully fine-grained enough to not trigger any locale-specfic behavior of the str methods and coarse-grained enough to not clutter the code. Emulating the with-statement behavior in before() and after() should be safe, as after() is always called when before() is called. hgext.convert.hg takes a similar approach.

curses: do not initialize LC_ALL to user settings (issue6358) 701341f57ceb moved the setlocale() call to right before curses was used. This didn’t fully solve the problem it was supposed to solve (locale-dependent functions, like date formatting/parsing and str methods on Python 2), but only postponed it. Initializing LC_CTYPE seems to be sufficient for curses to work correctly. Therefore LC_CTYPE is set while curses is used and reset afterwards. Some locale-dependent str methods might behave differently on Python 2 while curses is used, but that shouldn’d be a problem.

hgweb: encode WSGI environment like OS environment Previously, the WSGI environment keys and values were encoded using latin-1. This resulted in a crash if a WSGI environment key or value could not be encoded using latin-1. On Unix, the OS environment is byte-based. Therefore we should do the reverse of what Python does for os.environ. On Windows, there’s no native byte-based OS environment. Therefore we should do the same as what mercurial.encoding does with the OS environment.

hgweb: deduplicate code A following patch will change the way keys and values are encoded. To reduce the diff, I’ve split off the uninteresting part.

share: provide a more useful text for hg help Differential Revision: https://phab.mercurial-scm.org/D8651

procutil: make recent fix for zombies compatible with py2 The fix in ed684a82e29b (procutil: always waiting on child processes to prevent zombies with 'hg serve', 2020-05-07) works only on Python 3 because it passes a `daemon` argument to `threading.Thread()`. Python 2 requires you to assign to the `.daemon` property instead. Python 3 also seems to support that, so this patch fixes the code by unconditionally using the old form. Differential Revision: https://phab.mercurial-scm.org/D8657

pycompat: fix crash when default locale is unknown Instead, fall back to the filesystem encoding if the default locale is unknown.

help: document meaning of '%' in graphlog output I added the feature in 14d0e89520a2 (graphlog: use '%' for other context in merge conflict, 2020-01-28), but I didn't think to look for documentation until today (because an internal user asked about it). Differential Revision: https://phab.mercurial-scm.org/D8649

py3: fix broken man page generation, it was generating `(default: NUL*)` `bytes(default)` was producing things like `(default: \x00)` when handed non-bytes values such as `1`, `10`, or `True`. The man page generation would apparently ignore these bytes and produce man pages that had the string `(default: )`. Test Plan: - Ran `cd doc; python3 gendoc.py "hg.1.gendoc"` and grepped for bad output - Ran `make deb`, extracted the deb, manually inspected `hg.1` file. Differential Revision: https://phab.mercurial-scm.org/D8639

py3: fix comparison between int and None If stop is None, the condition was always false on Python 2, as None compares smaller than ints. Therefore we make the condition false if stop is None.

py3: pass regex as bytes

py3: avoid using %r format on bytes Before the patch, the 'b' prefix appeared in the formatted string. Wrapping the bytes as pycompat.bytestr solves this problem. Eventually, I think that we should move away from using %r (like 975e517451a6 and 4d6019c0e0ef did), but that would change output of non-ASCII bytes on Python 2, so we can’t do it on the stable branch. Also, many places continue to use %r, so it would be a good idea to do the change all at once.

py3: use `%d` for int in % formatting On Python 3, `%s` is an alias to `%b`, which requires that the object implements `__bytes__()`, which is not the case for `int`.

py3: pass native string to urlreq.url2pathname() Of course, I’m not happy with the warning, but it’s better than crashing. Solving the problem properly is hard, and non-UTF-8 percent-encoded bytes in file URLs seem rare enough to block solving that all file URLs (even if not SVN-specific) will cause a crash.

py3: suppress DeprecationWarning about deprecated base64 module aliases base64.encodestring() / base64.decodestring() were renamed to base64.encodebytes() / base64.decodebytes() in Python 3. The old names still worked, but raised a DeprecationWarning.

py3: use `pycompat.ziplist()`

py3: use `%d` for int in % formatting On Python 3, `%s` is an alias to `%b`, which requires that the object implements `__bytes__()`, which is not the case for `int`.

py3: fix bytes iteration

py3: unbyteify arguments to warnings.filterwarnings() This fixes a crash when trying to import the convert extension on Python 3.

tests: adjust to the new format in pyflakes output According to the pyflakes' NEWS.rst, the default output format changed recently: 2.2.0 (2020-04-08) - Include column information in error messages So the lines now read: contrib/perf.py:149:15 undefined name 'xrange' mercurial/hgweb/server.py:427:13 undefined name 'reload' mercurial/util.py:2862:24 undefined name 'file' This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8630

tests: consistently use pyflakes as a Python module We check availability of pyflakes as a module, and also running it for real as a module. Only fair to test filterpyflakes.py working correctly when using pyflakes as a module too. This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8629

tests: skip pyflakes for mercurial/thirdparty/ The current version of pyflakes (2.2.0) correctly detects one issue: mercurial/thirdparty/selectors2.py:335:40 '...'.format(...) has unused arguments at position(s): 1 But we're not interested in fixing lint errors in third-party code, so we need to exclude at least selectors2.py. And in the discussion for this patch it was decided to just skip the entire thirdparty directory. This is a graft of a similar fix that ended up on default. Differential Revision: https://phab.mercurial-scm.org/D8628

zeroconf: fix non existant formatting in the vendored zeroconf module On Tue Mar 1st 2016 at 09:33:39 timeless decided to wrap long line in `hgext/zeroconf/Zeroconf.py`. Doing so, he fat fingered a "%w" instead of a "%s" in a string. %w does not exists, 4 year later, pyflakes (rightfully) complains about it. So I am fixing it. Differential Revision: https://phab.mercurial-scm.org/D8627

ignore: note debugignore on ignore man page It took me a long time to find debugignore. I found the ignore man page quickly. This change adds a debugging section to the ignore man page letting people know there is a debug command.

nodemap: fix validity checking when revlog is too short We cannot check the nodeid of a revision that is not even there. We add a simple fix and simple test.

resourceutil: fix location of line comments These comments slipped out of position when the sources where formatted with black in 2372284d9457.

rebase: avoid clobbering wdir() with --dry-run or --confirm (issue6291) See 56d3e0b499df for the source of adding originalwd to the list of things that cause wdir to be updated. That change didn't come with tests, and attempts to recreate the scenario described have thus far failed. Differential Revision: https://phab.mercurial-scm.org/D8489

tests: show that rebase --dry-run and --confirm wipeout uncommitted changes It looks like the carnage is limited to rebasing something that is not an ancestor of wdir(), as both of these abort in a preflight check for that case with a dirty working directory. Differential Revision: https://phab.mercurial-scm.org/D8488

Added signature for changeset 065704cbdbdb

Added tag 5.4.1 for changeset 065704cbdbdb

py3: update comment to account for Python 2 and Python 3 differences

py3: add warning about buffering behavior of pycompat.{stdout,stderr}

tests: fix indentation

graft: fix --base value to be saved in state file 'True' just works because it is treated as an integer revision '1' and only the truthiness of the basectx is important. If multiple source revisions were supported with --base, the resumed graft operation would go wrong.

flags: also test merge with executable bit removed This might catch more bug in the future. Differential Revision: https://phab.mercurial-scm.org/D8536

flags: also test the removal of the exec flag Differential Revision: https://phab.mercurial-scm.org/D8535

flags: read flag from dirstate/disk for workingcopyctx (issue5743) In 491855ea9d62, various piece of code are moved from committablectx to workingctx. The reason given is "These read from the dirstate, so they shouldn't be used in other subclasses." At least for `flags` this change introduce a bug, because the value flags end up being read from `_manifest` disregarding the actual state in the working copy (ie: on disk). When merging exec flag change with renames, this means a new files (the local content, renamed) is properly written on disk, with the right flags, but the flags part is later ignored when actually reading flags during merge. It is not clear to me why the `flags` function was moved, because the code does not actually hit the dirstate (the reason given in the changeset description). So I am moving it back to were it comes from and we use a simpler version of that code (that hit the dirstate everytime) in workingcopyctx. This fix the last know bug with merging rename and executable byte changes. Other similar bug might be lurking in 491855ea9d62, but I have not investigated them. Differential Revision: https://phab.mercurial-scm.org/D8534

flags: actually merge flags in simplemerge Since b86fc43e4b73, the local flag were blindly taken. This resulted in bug when rename are involved. exec flag change are now properly merged (when merged from the rename side). Another bug is affecting this when merging from the side without the rename. Differential Revision: https://phab.mercurial-scm.org/D8533

flags: add a test for merging exec flag change with rename and file change Changing the file activate other code path that also have bugs… There are two distinct bugs depending of which side of the merge you stand on. They both leading to exec flag loss. We add tests for both, the fix are coming in later changesets. Differential Revision: https://phab.mercurial-scm.org/D8532

flags: account for flag change when tracking rename relevant to merge There are some logic filtering rename to the one relevant to the merge. That logic was oblivious of flag change, leading to exec flag being dropped when merged with a renamed. There are two others bugs affecting this scenario. This patch fix the was where there is not modification involved except for the flag change. Fixes for the other bug are coming in later changesets. Differential Revision: https://phab.mercurial-scm.org/D8531

flags: also test merging a rename with and exec flag change This case is currently buggy and was not tested. This is probably a quite old regression. The next changeset fix this case. Move exec+rename related bug will gain a test later. To highlight the expected behavior the currently missing line are marked with (false !) and the bad one with (true !) note: we should probably gain explicit "test bool" for this usecases. Differential Revision: https://phab.mercurial-scm.org/D8530

flags: introduce explicit testing for merging change to exec flag It turns out that we do not seems to test the simple case for merging exec flag changes. More advanced case are test (merging exec flag without a common ancestors, merging with a symlink, etc…) but not the basic. We are about introduce various fixes to merging flag change across renames, having the most basic case tested first seems useful. note: We are only testing "adding" an exec flag here, not removing it. We introduce basic test on stable and will consolidate them on default. Differential Revision: https://phab.mercurial-scm.org/D8529

graft-state: save --base in graft's state, fixing bug with graft --continue Without this change, running graft --continue after grafting a merge commit using --base (and encountering conflicts) will output "skipping ungraftable merge revision" even though we specified a base in the initial graft command. Graft's improve behaviour is reflected in test-graft.t. Differential Revision: https://phab.mercurial-scm.org/D8578

setup: raise minimum Python version to 2.7.4 On older Python versions, Mercurial is not really usable because of https://bugs.python.org/issue10211. Recently someone reported a crash on the mailing list when running Mercurial on Python 2.7.3. There was consensus that fixing compatibility for a Python version more than 7 years old is not worth it. So, instead of making Mercurial crash with an obscure exception, this patch raises the minimum Python version to 2.7.4.

fsmonitor: coerce `clock` variable to byte-string (issue6321) Callers of `fsmonitor.state.setlastclock` pass their arguments wrapped in `pycompat.sysbytes` to ensure the value is a `bytes` on Python 3. However in `fsmonitor.poststatus.__call__`, if the return value of `getlastclock()` is `None`, we use the value of `fsmonitor.poststatus._startclock` instead, which is not converted to a byte string in the same manner. This commit converts the value of `startclock` to a byte string using `pycompat.sysbytes` in the constructor for `poststatus`, to avoid the "`str` + `bytes`" error from issue 6321. Differential Revision: https://phab.mercurial-scm.org/D8573

py3: change default priority and length used for sorting hooks to be compatible with python 3 The call to `sorted(hooks.values())` can on line 213 of hooks.py can raise when using python 3. For instance, when hooks.values is `[(0, 2, b'post-commit.check-status', b''), (None, None, b'changegroup.app-hooks', <object object at 0x7f5279885590>)]`, the error is `TypeError: '<' not supported between instances of 'NoneType' and 'int'` This fix keeps the same order that was used in python 2 without relying on comparison with None. Differential Revision: https://phab.mercurial-scm.org/D8527

relnotes: copy "next" to "5.4" and clear "next" This is the same thing as we've done for the previous few releases. Differential Revision: https://phab.mercurial-scm.org/D8546

dirstate: make sure the dirstate is loaded before the changelog (issue6303) Before this change, it was possible for the changelog to be loaded before the dirstate. If a transaction happens betwen the changelog and dirstate reading, the dirstate can up end poitning toward a revision not existing in the (olded) changelog. This lead to a warning. With this revision, we preload the dirstate parent before reading the changelog. This has a negligible performance impact on performance for all case we are tracking. Differential Revision: https://phab.mercurial-scm.org/D8528

rust-dirstatemap: don't read the dirstate when requesting parents A future patch for issue 6303 reveals a big performance regression in the Rust `DirstateMap` that reads the entire dirstate when requesting parents instead of the first 40 bytes. `perfdiscovery` gets a *significant* speedup (from 0.101 to 0.016) when applied against said patch. I'm assuming it has other performance benefits, but this is already a good enough win. Differential Revision: https://phab.mercurial-scm.org/D8513

py3: fix exception in pull when several things happen to a bookmark Specifically, when `changes` is: [(b'@upstream-committed', None, <function ui.status>, b'updating bookmark @upstream-committed\n'), (b'@upstream-committed', binary-node, <function ui.warn>, b'divergent bookmark @ stored as @upstream-committed\n')] sorting the list raises: TypeError: '<' not supported between instances of 'bytes' and 'NoneType' Differential Revision: https://phab.mercurial-scm.org/D8523

fastexport: adjust output to be more canonical For time zones, git doesn't consider +0 and -0 the same timezone, so use the former canonically. Add a test case to ensure that non-UTC offsets are handled correctly. The real name part of the committer name is normally not quoted, so don't enforce that. Differential Revision: https://phab.mercurial-scm.org/D8522

bash_completion: do not use aliased hg if it sources a script (issue6308) I have an alias that sources a script around hg. Mercurial's bash_completion script tries to use this as its main hg binary. But sourcing a wrapper breaks Bash's completion. So this patch disables using the alias as the hg binary if it starts with "source ". Alias resolution was introduced in rev 191ab08e7099 for users with "alias hg='hg --some_opts'". See https://www.mercurial-scm.org/repo/hg/rev/191ab08e7099

demandimport: fix compatibility with meta path finders w/o find_spec() method Meta path finders got a find_spec() method in Python version 3.4. The sys.meta_path documentation says that the deprecated find_module() method is used as a fallback. Setuptool’s VendorImporter still doesn’t have the find_spec() method, which resulted in a crash e.g. within a virtual environment. For reference, I opened an issue for that: https://github.com/pypa/setuptools/issues/2104. An alternative implementation would have been to implement a wrapper for find_module() itself and raise an AttributeError when accessing find_spec() if the wrapped finder doesn’t have it.

tests: fix timer scaling in wait-on-file When using the default test timeouts, wait-on-file would not wait for $n seconds, but $n/100 seconds. This resulted in easy timeouts on even moderately busy fast machines. Fix the scaling to apply in all cases. Adjust the stepping slightly to be nicer to systems with the historic 100Hz time base to ensure that the scheduler actually switches to a different process and gives them time to work. Differential Revision: https://phab.mercurial-scm.org/D8505

manifest-cache: ignore IOError while writing If the wcache directory is non writable for some reason (eg: belong to root). Trying to write to it currently crash Mercurial. Instead we ignore the error and skip writing that cache. We should probably improve the user experience of multiple users interacting with the same repository. However this is not an adventure for stable. Differential Revision: https://phab.mercurial-scm.org/D8512

copy: give better error message when no source paths found with --at-rev The new error message matches what we show when marking copies in the working copy. Differential Revision: https://phab.mercurial-scm.org/D8496

tests: show poor error message for `hg cp -A --at-rev . non-existent dst` Differential Revision: https://phab.mercurial-scm.org/D8495

copy: to find copy source, walk parent of revision we're marking copies in As shown in the previous patch, `hg cp --after --at-rev . src dst` fails if `src` is not in `.`. It seems obvious that you should always walk the *parent* of the revision you're marking copies in, but that's not how it was done for the working copy, and I didn't think to change it when marking copies in a non-working-copy commit. This patch fixes that by walking the parent commit instead, but only if we're marking copies for a non-working-copy commit. We need to leave the working-copy code unchanged because it depends on the weird behavior of `workingctx.walk()`. With these changes, there's very little overlap between the working-copy version and the non-working-copy version of `walkpats()`, but I've refrained from cleaning that up on the stable branch. Differential Revision: https://phab.mercurial-scm.org/D8494

tests: show that `hg cp -A --at-rev .` doesn't work for renames I clearly forgot to implement (and test) support for marking of renames when I added support for marking of copies :( Differential Revision: https://phab.mercurial-scm.org/D8493

formatting: add missing newline Differential Revision: https://phab.mercurial-scm.org/D8509

fastexport: downgrade message about already exported changesets to debug The old warning level is just too noisy for incremental processing. Differential Revision: https://phab.mercurial-scm.org/D8487

url: fix a bytes vs str crash in processing proxy headers (issue6249) I have no idea how to make a test for this, so if somebody knows, feel free to add one or follow up on this. The bug reporter reported that it worked for them, so there may not be other hidden issues here. Differential Revision: https://phab.mercurial-scm.org/D8485

pullbundles: use unfiltered repo for head/base matching The unfiltered view works even when changeset transistion from draft to hidden phase. The normal visibility is already ensured by discovery as invisible heads would have been filtered out before. Skipping the filtering has a positive impact on performance, too. Differential Revision: https://phab.mercurial-scm.org/D8481

procutil: always waiting on child processes to prevent zombies with 'hg serve' When runbgcommand is invoked by an extension with ensurestart=False, we never called waitpid - which is fine in most cases, except if that's happening on a command server (e.g. chg), in which case the child defunct process will just sit there for as long as the server is running. The actual semantics of SIGCHLD signal handling is a lot more complex than it seems, and the POSIX standard *seems* to read that it's ignored by default and everything would just work without the waitpid if we're not listening for it, but the truth is that it's only ignored if we *explicitly* set it to SIG_IGN. We further cannot set it to SIG_IGN or to a catch-all handler across all of 'hg serve', because Python's suprocess.Popen relies on that signal, and a few specific parts of hg also set custom handlers, so instead we wait for specific PIDs in dedicated threads. I did a poor-man's benchmark of the thread creation and it seems to take about 1ms, which is way better than the 20+ms from ensurestart=True. Differential Revision: https://phab.mercurial-scm.org/D8497

tests: use regular POSIX shell wait-on-file requires one POSIX extension (sleep with non-integral argument), but it doesn't require any bash extensions, so just require a normal POSIX shell. While here, use consistent formatting without redundant ; Differential Revision: https://phab.mercurial-scm.org/D8500

rust-regex: increase the DFA size limit for the `regex` crate `re2`'s DFA limit is already increased in `rust/hg-core/src/re2/rust_re2.cpp`, the same has to be done for the `regex` crate. Big repositories with big `.hgignore`s will sometimes hit this limit and face extreme performance regressions (I've seen one take *minutes* for `hg status`). Differential Revision: https://phab.mercurial-scm.org/D8499

Added signature for changeset cf3e07d7648a

Added tag 5.4 for changeset cf3e07d7648a

tests: clarify a comment describing a phabricator test scenario Per review feedback. Differential Revision: https://phab.mercurial-scm.org/D8455

phabricator: ensure that `phabsend` is given a contiguous, linear commit range Supplying a non-linear range was another orphan factory. While in theory there could be a use case for skipping over garbage commits (like adding debugging) and getting the valuable commits extracted out at the same time as posting a review, it seems far more likely that specifying a non-linear range is a user error. This is another case of issue6045, but predates both 0680b8a1992a and 601ce5392cb0. Neither the `--no-amend` case nor resubmitting a previously submitted commit would cause orphans. But for the sake of simplicity and to keep the parents tracked on Phabricator in the proper state, ban missing commits unconditionally. Differential Revision: https://phab.mercurial-scm.org/D8454

automation: support building Python 3 MSI installers This is very similar to what we just did for Inno. Differential Revision: https://phab.mercurial-scm.org/D8484

automation: support building Python 3 Inno installers The core packaging code now supports building Python 3 installers using PyOxidizer. Let's teach the automation code to invoke it so that we produce both Python 2 and Python 3 based exe installers. When publishing the artifacts, the Python 3 versions are preferred over the Python 2 versions given their higher weight (10 versus 9). This may be a controversial change. But I think making Python 3 the default is warranted, as it is the future. The Python 2 installers are still fully supported and can be installed should issues with Python 3 arise. Differential Revision: https://phab.mercurial-scm.org/D8483

automation: add extra arguments when building Inno These were being fed into the template expansion but not being used. This meant --version was not getting set when it should have been. Differential Revision: https://phab.mercurial-scm.org/D8482

packaging: add -python2 to Windows installer filenames We just taught the Windows installers to produce Python 3 variants built with PyOxidizer. Our plan is to publish both Python 2 and Python 3 versions of the installers for Mercurial 5.4. This commit teaches the Inno and WiX installers to add an optional string suffix to the installer name. On Python 2, that suffix is "-python2." We reserve the existing name for the Python 3 installers, which we want to make the default. Differential Revision: https://phab.mercurial-scm.org/D8479

automation: support building Windows wheels for Python 3.7 and 3.8 The time has come to support Python 3 on Windows. Let's teach our automation code to produce Windows wheels for Python 3.7 and 3.8. We could theoretically support 3.5 and 3.6. But I don't think it is worth it. People on Windows generally use the Mercurial installers, not wheels. And I'd prefer we limit variability and not have to worry about supporting earlier Python versions if it can be helped. As part of this, we change the invocation of pip to `python.exe -m pip`, as this is what is being recommended in Python docs these days. And it seemed to be required to avoid a weird build error. Why, I'm not sure. But it looks like pip was having trouble finding a Visual Studio files when invoked as `pip.exe` but not when using `python.exe -m pip`. Who knows. Differential Revision: https://phab.mercurial-scm.org/D8478

packaging: support building WiX installers with PyOxidizer We initially implemented PyOxidizer support for Inno installers. That did most of the heavy work of integrating PyOxidizer into the packaging system. Implementing WiX installer support was pretty straightforward. Aspects of this patch look very similar to Inno's. The main difference is the handling of the Visual C++ Redistributable Runtime files. The WiX installer was formerly using merge modules to install the VC++ 9.0 runtime because this feature is supported by the WiX installer (it isn't easily available to Inno installers). Our strategy for the runtime files is to install the vcruntime140.dll file next to hg.exe just like any other file. While we could leverage WiX's functionality for invoking a VCRedist installer, I don't want to deal with the complexity at this juncture. So, we let run_pyoxidizer() copy vcruntime140.dll into the staging directory (like it does for Inno) and our dynamic WiX XML generator picks it up as a regular file and installs it. We did, however, have to teach mercurial.wxs how to conditionally use the merge modules. But this was rather straightforward. Comparing the file layout of the WiX installers before and after: * Various lib/*.{pyd, dll} files no longer exist * python27.dll was replaced by python37.dll * vcruntime140.dll was added All these changes are expected due to the transition to Python 3 and to PyOxidizer, which embeded the .pyd and .dll files in hg.exe. Differential Revision: https://phab.mercurial-scm.org/D8477

packaging: move version derivation to run_wix_packaging() With the previous commit moving signing inline, we no longer need to compute the version string in build_installer() and can instead move this logic to run_wix_packaging(). This makes the logic in build_installer() simpler, which makes it easier to implement alternate building mechanisms. Differential Revision: https://phab.mercurial-scm.org/D8476

packaging: integrate signing into run_wix_packaging() Previously, signing was implemented via a separate function which called build_installer() and then called signing functionality. In this model, in order to implement an alternative build mechanism, we would have to invent a new variant to handle signing as well. This commit merges the signing logic into the function invoking wix. If we pass an argument holding metadata about how to sign, we sign hg.exe and the installer. This means all we have to do is pass in signing info and the signing just works. A slight change here is that signing of hg.exe happens in the staging directory as opposed to before the staging directory is populated. I don't think this matters. Differential Revision: https://phab.mercurial-scm.org/D8475

packaging: isolate invocation of WiX to own function Like we did for Inno, we want to split out the building of Mercurial from invoking the packaging tool so that we can introduce an alternate build mechanism. As part of this refactor, there are inconsequential changes to file layouts. Before, some shared files such as the WiX binaries and merge modules would be installed under build/. Now, they are installed under build/wix-*. This is to keep implementation simpler. But it also helps keep build state more isolated. Differential Revision: https://phab.mercurial-scm.org/D8474

packaging: support building Inno installer with PyOxidizer We want to start distributing Mercurial on Python 3 on Windows. PyOxidizer will be our vehicle for achieving that. This commit implements basic support for producing Inno installers using PyOxidizer. While it is an eventual goal of PyOxidizer to produce installers, those features aren't yet implemented. So our strategy for producing Mercurial installers is similar to what we've been doing with py2exe: invoke a build system to produce files then stage those files into a directory so they can be turned into an installer. We had to make significant alterations to the pyoxidizer.bzl config file to get it to produce the files that we desire for a Windows install. This meant differentiating the build targets so we can target Windows specifically. We've added a new module to hgpackaging to deal with interacting with PyOxidizer. It is similar to pyexe: we invoke a build process then copy files to a staging directory. Ideally these extra files would be defined in pyoxidizer.bzl. But I don't think it is worth doing at this time, as PyOxidizer's config files are lacking some features to make this turnkey. The rest of the change is introducing a variant of the Inno installer code that invokes PyOxidizer instead of py2exe. Comparing the Python 2.7 based Inno installers with this one, the following changes were observed: * No lib/*.{pyd, dll} files * No Microsoft.VC90.CRT.manifest * No msvc{m,p,r}90.dll files * python27.dll replaced with python37.dll * Add vcruntime140.dll file The disappearance of the .pyd and .dll files is acceptable, as PyOxidizer has embedded these in hg.exe and loads them from memory. The disappearance of the *90* files is acceptable because those provide the Visual C++ 9 runtime, as required by Python 2.7. Similarly, the appearance of vcruntime140.dll is a requirement of Python 3.7. Differential Revision: https://phab.mercurial-scm.org/D8473

packaging: split Inno installer building from Mercurial building We want to make the logic for producing the installer agnostic about how Mercurial is built to allow for alternate build methods (like PyOxidizer). Differential Revision: https://phab.mercurial-scm.org/D8472

packaging: remove pyoxidizer.bzl from packaging directory We have another version in rust/hgcli that is more modern and is already associated with our Rust CLI project. Differential Revision: https://phab.mercurial-scm.org/D8471

contrib: install PyOxidizer in Linux and Windows environments For Linux, this was trivial. For Windows, we need to teach the powershell script to install Rust as well. This was also pretty straightforward. Differential Revision: https://phab.mercurial-scm.org/D8468

diff: re-establish linear runtime performance The previous method with sum() and list() creates a new list object for every hunk. Then sum() is used to flatten out this sequence of lists. The sum() function is not "lazy", but creates a new list object for every "+" operation and so this code had quadratic runtime behaviour.

rust-status: check for '.hg' regardless of file type (issue6300) '.hg' would show up in `hg status` if were a symlink. Differential Revision: https://phab.mercurial-scm.org/D8461

rust: remove extra empty line This is a gratuitous cleanup. Differential Revision: https://phab.mercurial-scm.org/D8460

upgrade: properly filter action depending on planned work The `determineactions` function filters out deficiency that are not scheduled to be fixed by the target repository configuration. However it only did so for requirement we currently support, letting other actions for unsupported requirement through even if the target repo configuration did not request it. As a result the output of the command was easily polluted by experimental feature with no upgrade support. We rework the code to still filter out requirement based action without the faulty filtering. Differential Revision: https://phab.mercurial-scm.org/D8427

nodemap: skip persistent nodemap warming for revlog not using it Before this patch, the usual checking (especially, inline-ess) were not performed when warming the cache through `hg debugupdatecache`. This is now fixed. Differential Revision: https://phab.mercurial-scm.org/D8408

wait-on-file: adjust the timer counter The wait performed in increment of 0.01 second, but the timer was expressed in second. So we did not wait 20s, we waited 0.2 second. Internally we adjust the counter to be expressed in centisecond. This prevent some flackyness in the test. Differential Revision: https://phab.mercurial-scm.org/D8453

packaging: add docutils as dependency The previous commit revealed that attempting to run `python setup.py build_doc` from the packaging virtualenv failed due to missing docutils package. We didn't notice this before because py2exe Windows packaging appears to use a Python from another virtualenv (which does include docutils) to invoke setup.py. I discovered this as part of implementing packaging outside of that virtualenv environment. Differential Revision: https://phab.mercurial-scm.org/D8470

setup: use sysstr() on process output Otherwise we get a str-bytes mismatch on Python 3 if an error occurs. Differential Revision: https://phab.mercurial-scm.org/D8469

automation: install latest Python versions in Linux Staying up to date. Keeping parity with the Windows environment. Differential Revision: https://phab.mercurial-scm.org/D8467

contrib: update to latest Python 2.7, 3.7, and 3.8 We would ideally update the 3.5 and 3.6 versions as well. But Python didn't generate exe installers for newer versions for some reason. Differential Revision: https://phab.mercurial-scm.org/D8466

automation: always use latest Windows AMI The old AMI isn't available any more. We seem to run into this problem every few months. Amazon (or Microsoft) appears to be removing the old AMIs when they are superseded or something. Let's give up on tracking known images and switch the image selection logic to use the latest published image. Differential Revision: https://phab.mercurial-scm.org/D8465

tests: stabilize test-log.t on Windows I think this is because Windows doesn't recognize single quote. Other ssh based clones use this clunky style too. Differential Revision: https://phab.mercurial-scm.org/D8459

tests: stabilize test-convert-hg-source.t on Windows This was a missing update in 1b8fd4af3318. Differential Revision: https://phab.mercurial-scm.org/D8458

Added signature for changeset 26ce8e751503

Added tag 5.4rc0 for changeset 26ce8e751503

merge default into stable for 5.4 release

templatekw: fix shownames() to check if namespace exists in repo (issue6301) Namespace registration is dynamic, but the corresponding template keyword is registered statically.

wait-on-file: use proper variable in math This seems better and safer to be explicit. Differential Revision: https://phab.mercurial-scm.org/D8426

wait-on-file: don't quote arithmetic argument This is unnecessary and Mac OS X choke on them. Differential Revision: https://phab.mercurial-scm.org/D8425

graft: exit 1 on conflicts, like merge It's more consistent, and makes it nicer to script around hg if you don't have to ignore exit code 255, which is the error code for basically everything in hg. Differential Revision: https://phab.mercurial-scm.org/D8423

tests: deal with "ls" vs "ls -A" difference on 2BSD derived systems BSD-derived systems will use "ls -A" when running as root. Adjust the test cases to always use the flag and include .hg and related output as appropiately. Differential Revision: https://phab.mercurial-scm.org/D8397

tests: skip non-readable check for root Trying to check for errors on non-readable hgrc requires UNIX permissions, but still won't work for root. So adjust the check. Differential Revision: https://phab.mercurial-scm.org/D8396

tests: skip CVS tests for root It is not uncommon for cvs to check for root and to refuse work in that case. Differential Revision: https://phab.mercurial-scm.org/D8395

make: drop the `-c` arg to `install` in the documentation makefile This arg caused `gmake install` on OpenIndiana 2019.10 (illumos) fail with: install: The -c, -f, -n options each require a directory following! install: The -c, -f, -n options each require a directory following! install: The -c, -f, -n options each require a directory following! gmake[1]: *** [Makefile:41: install] Error 2 gmake[1]: Leaving directory '/usr/local/share/mercurial/doc' The workaround is to run `gmake install-bin`. The man page for 10.14 says this is to copy the file and is only for compatability, as it is the default. The CentOS 7 man page says it is ignored. The top level makefile doesn't use this argument at all, so I'm not sure why it's here. Differential Revision: https://phab.mercurial-scm.org/D8439

phabricator: restack any new orphans created by phabsend (issue6045) Previously, posting a new review for a non head commit would orphan the head. The general case is any descendant of the selected revisions got orphaned if this was the first time the selected revisions were submitted. It doesn't happen when resubmitting. I've already had coworkers hit this a few times and get confused. Since posting a review isn't generally thought of as an editing operation, it would probably be easier for new users if we just restacked. This avoids restacking existing orphans around the submission because that may involve merge conflict resolution. Users who already have orphans should know how to stabilize them anyway. Differential Revision: https://phab.mercurial-scm.org/D8438

phabricator: prevent posting obsolete commits I don't see why this would be useful in the first place. But I had a coworker submit a single commit that was not a branch head, and the result was to orphan its child and keep the original commit visible. He then did up arrow + Enter, and it happily created a new review (since the URL isn't amended into the original commit specified on the command line) and a new successor, resulting in a local divergence. I'd like to fix the issue with creating orphans, but this is simple enough to prevent on its own. Differential Revision: https://phab.mercurial-scm.org/D8437

phabricator: avoid creating unstable children within the review stack The instability occurred when rebasing something that has already been submitted onto something that hasn't, and then resubmitting the stack. Or as the test shows, just resubmitting and including something earlier that wasn't previously submitted. There's a general case here where any children (not just the ones in the range of commits posted for review) should be re-stabilized. But handling the selected commits here will cause the `local:commit` node values that are tracked on Phabricator to be properly kept in sync. Differential Revision: https://phab.mercurial-scm.org/D8436

phabricator: add an option to fold several commits into one review (issue6244) Now that all of the pieces are in place, alter the user facing command to allow it. This is the default behavior when using `arc`, but I much prefer the 1:1 approach, and I'm tempted to mark this advanced to limit its abuse. I started out calling this `--no-stack` like the feature request suggested, but I found it less obvious (especially when writing the code), so I went with the `hg fold` analogue. This will populate the `Commits` tab in the web UI with the hash of each commit folded into the review. From experimentation, it seems to list them in the order they are received from the extension instead of the actual parent/child relationship. The extension sends them in sorted order, thanks to `templatefilters.json()`. Since there's enough info there for them to put things in the right order, JSON is unordered aside from lists (IIUC), and there doesn't seem to be any harmful side effects, I guess we write this off as their bug. It is simple enough to workaround by putting a check for `util.sortdict` into `templatefilters.json()`, and don't resort in that case. There are a handful of restrictions that are documented in the code, which somebody could probably fix if they're interested. Notably, this requires the (default) `--amend` option, because there's not an easy way to apply a local tag across several commits. This also doesn't do preflight checking to ensure that all previous commits that were part of a single review are selected when updating. That seems expensive. What happens is the excluded commit is dropped from the review, but it keeps the Differential Revision line in the commit message. Not everything can be edited, so it doesn't seem worth making the code even more complicated to handle this edge case. There are a couple of "obsolete feature not enabled but X markers found!" messages that appeared on Windows but not macOS. I have no idea what's going on here, but that's an unrelated issue, so I conditionalized those lines. Differential Revision: https://phab.mercurial-scm.org/D8314

tests: move the phabricator auth token to the global config file The next commit introduces a new repo to simplify its development. This value needs to be modified to record tests, so it doesn't make sense to have to do that twice. The callsign and URL are *not* moved because there are tests that fallback to the .arcconfig file when those aren't present. Differential Revision: https://phab.mercurial-scm.org/D8390

phabricator: add debug logging to show previous node values in `phabsend` This isn't real useful here, but was very useful showing how `phabsend --fold` handles commits created by `hg fold` and `hg split`. It introduces a new debug function and flag instead of using `ui.debug()`, because `--debug` prints out all of the API chatter. Differential Revision: https://phab.mercurial-scm.org/D8389