status: update comment to reflect the more recent situation This is a gratuitous cleanup. Differential Revision: https://phab.mercurial-scm.org/D8516

hooks: provide access to transaction changes for internal hooks External hooks are skipped here as the environment often has a size limit in the low MBs and that can easily be reached by larger transactions. Differential Revision: https://phab.mercurial-scm.org/D8490

rust-regex: add test for verbatim regex syntax This makes sure it's not modified. Differential Revision: https://phab.mercurial-scm.org/D8508

rust-regex: prevent nonsensical `.*.*` pattern from happening Differential Revision: https://phab.mercurial-scm.org/D8507

rust-regex: fix issues with regex anchoring and performance It turns out that the way I tried to work around `regex`'s behavior difference with `re2` and Python's `re` was 1) buggy and 2) much more complicated than needed. In a few words: `regex` adds `.*` on either side of patterns when no start or end anchor is present. My previous workaround put `^` or `$` for every pattern, which is wrong even without the other 2 bugs on top of it. Using `^(?:<patterns>)` right at the end of the `regex` path fixes the issue. I've opened an issue to get a build option instead: https://github.com/rust-lang/regex/issues/675 Differential Revision: https://phab.mercurial-scm.org/D8506

diff: avoid going from contexts to nodes and back This will allow us to pass in-memory contexts that may not have a valid node to the diffing logic. Differential Revision: https://phab.mercurial-scm.org/D8503

cleanup: avoid extra node/ctx conversions in logcmdutil.diffordiffstat I'm about to write some code that wants to pass a memctx to diffordiffstat, but this feels like a meritorious cleanup anyway, since the first thing this method does is turn nodes into contexts, and most callers have a context handy. Differential Revision: https://phab.mercurial-scm.org/D8502

pyoxidizer: formatting bazel definitions This meets the Bazel style guide: https://docs.bazel.build/versions/master/skylark/bzl-style.html and was mostly done automatically with buildifier. Differential Revision: https://phab.mercurial-scm.org/D8521

revisions: parse "x123" as "nodeid starting with 123" without prefixhexnode `experimental.revisions.prefixhexnode` makes it so the template function `shortest()` uses an "x" prefix to disambiguate between short nodeids and revnums. That config has so far also been used for enabling parsing of "x123" unambiguously as a nodeid. That makes it a little annoying for people who have prefixhexnode=yes to share such nodeids with people who have prefixhexnode=no ("x123" will be considered invalid for them). There seems to be little harm in allowing that parsing for everyone. We still let e.g. bookmark names like "x123" take precedence over the nodeid, so that's not a concern. The only thing I can think of is that people get used to the "x" prefix being valid, making it impossible for us to change to a different prefix if we wanted to do that when graduating the feature. Differential Revision: https://phab.mercurial-scm.org/D8514

status: use cmdutil.check_at_most_one_arg() for checking --rev/--change There are apparently no tests for this either. Differential Revision: https://phab.mercurial-scm.org/D8511

diff: use cmdutil.check_at_most_one_arg() for checking --rev/--change The same check was done in extdiff as well, so I fixed that too. There are apparently no tests for this. Differential Revision: https://phab.mercurial-scm.org/D8510

copy: give better error message when no source paths found with --at-rev The new error message matches what we show when marking copies in the working copy. Differential Revision: https://phab.mercurial-scm.org/D8496

tests: show poor error message for `hg cp -A --at-rev . non-existent dst` Differential Revision: https://phab.mercurial-scm.org/D8495

copy: to find copy source, walk parent of revision we're marking copies in As shown in the previous patch, `hg cp --after --at-rev . src dst` fails if `src` is not in `.`. It seems obvious that you should always walk the *parent* of the revision you're marking copies in, but that's not how it was done for the working copy, and I didn't think to change it when marking copies in a non-working-copy commit. This patch fixes that by walking the parent commit instead, but only if we're marking copies for a non-working-copy commit. We need to leave the working-copy code unchanged because it depends on the weird behavior of `workingctx.walk()`. With these changes, there's very little overlap between the working-copy version and the non-working-copy version of `walkpats()`, but I've refrained from cleaning that up on the stable branch. Differential Revision: https://phab.mercurial-scm.org/D8494

tests: show that `hg cp -A --at-rev .` doesn't work for renames I clearly forgot to implement (and test) support for marking of renames when I added support for marking of copies :( Differential Revision: https://phab.mercurial-scm.org/D8493

rust-matchers: add TODO about incomplete `Display` for `IncludeMatcher` This is purely for future reference, I don't think this is a problem right now, since the `Display` is *only* used to ease debugging and has no real users. Differential Revision: https://phab.mercurial-scm.org/D8492

rust-filepatterns: match exact `rootglob`s with a `HashSet`, not in the regex This optimization yields some very interesting results in `rootglob`-heavy repositories. I build a test repository of the following structure: ``` root /<uuid>/build/empty_file ... repeat for 4000 entries ``` and a `.hgignore` containing the corresponding 4000 `rootglob` entries pointing to all `build/` folders. Rust+c `hg status` goes from 350ms down to 110ms. Differential Revision: https://phab.mercurial-scm.org/D8491

dirstate: force _checkexec to return a bool posix.checkexec can return True, False, or None. The rust status implementation expects a boolean, so make sure _checkexec returns a boolean. Differential Revision: https://phab.mercurial-scm.org/D8432

locking: wait for locks in `hg cp` and `hg mv` I traced this back to revision 1822 (64df422), and there's no explanation why we would prefer to error out instead of waiting for the locks. Differential Revision: https://phab.mercurial-scm.org/D8464

nodemap: teach `hg debugformat` about the persistent nodemap option We have a new requirement, we should display it. Differential Revision: https://phab.mercurial-scm.org/D8430

upgrade: support the --quiet flag The command is currently very verbose with a various bit of output being time sensitive or randomized. The make invocation bulky and hard to match in the test. We move various message from `ui.write` to `ui.status` in order for the `--quiet` flag to have effect and helps the situation. As a side benefit, we can replace the various redirection to > /dev/null with the --quiet flag. Differential Revision: https://phab.mercurial-scm.org/D8429

upgrade: clearly list optimisations This makes the command operation clearer. And this will be necessary to have a short version of this information with the next changesets, teaching `hg debugupgraderepo` about `--quiet`. Differential Revision: https://phab.mercurial-scm.org/D8428

nodemap: move the mode option to storage.revlog.nodemap.mode The option stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8422

nodemap: move the option for mmap usage to storage.revlog.nodemap.mmap The option is stay experimental as long as the main feature is. Differential Revision: https://phab.mercurial-scm.org/D8421

nodemap: move and update the commend about persistence being experimental The comment was at the wrong place (on the developer option instead of the activation switch). So we move it at the right location and update it. Differential Revision: https://phab.mercurial-scm.org/D8420

nodemap: move the main switch to the `format` section The config to enable persistent nodemap is now `format.use-persistent-nodemap`. However the option remain marked as experimental because it only improve performance for people using the rust extensions. Differential Revision: https://phab.mercurial-scm.org/D8419

nodemap: drop the 'exp-' prefix for internal opener option The feature is now in a descent shape and we can consider having it "less" experimental. We won't be able to make it "totally" non-experimental, because its benefit rely on rust, which is totally experimental. Differential Revision: https://phab.mercurial-scm.org/D8418

nodemap: gate the feature behind a new requirement Now that the feature is working smoothly, a question was still open, should we gate the feature behind a new requirement or just treat it as a cache to be warmed by those who can and ignored by other. The advantage of using the cache approach is a transparent upgrade/downgrade story, making the feature easier to move to. However having out of date cache can come with a significant performance hit for process who expect an up to date cache but found none. In this case the file needs to be stored under `.hg/cache`. The "requirement" approach guarantee that the persistent nodemap is up to date. However, it comes with a less flexible activation story since an explicite upgrade is required. In this case the file can be stored in `.hg/store`. This wiki page is relevant to this questions: https://www.mercurial-scm.org/wiki/ComputedIndexPlan So which one should we take? Another element came into plan, the persistent nodemap use the `add` method of the transaction, it is used to keep track of a file content before a transaction in case we need to rollback it back. It turns out that the `transaction.add` API does not support file stored anywhere than `.hg/store`. Making it support file stored elsewhere is possible, require a change in on disk transaction format. Updating on disk file requires… introducing a new requirements. As a result, we pick the second option "gating the persistent nodemap behind a new requirements". Differential Revision: https://phab.mercurial-scm.org/D8417

nodemap: move on disk file to version 1 The current format contains the information we need, lets freeze it before the release. Differential Revision: https://phab.mercurial-scm.org/D8416

nodemap: add a new mode value, "strict" When "strict" is set, operation will refuse to use the slow path and abort if they would. This is useful for testing, benchmarking and server operation. Differential Revision: https://phab.mercurial-scm.org/D8415

nodemap: add a new mode option, with an optional "warn" value When "warn" is set, user will get notified when the slow code, used for compatibility is used. This can help people to detect situation were using that feature will give them a slowdown instead of a speedup. Differential Revision: https://phab.mercurial-scm.org/D8414

nodemap: also warm manifest nodemap with other caches The `hg debugupdatecache` command now also warm the persistent nodemap for the manifest (when applicable). Differential Revision: https://phab.mercurial-scm.org/D8411

nodemap: also use persistent nodemap for manifest The manifest as a different usage pattern than the changelog. First, while the lookup in changelog are not garanteed to match, the lookup in the manifest nodemap come from changelog and will exist in the manifest. In addition, looking up a manifest almost always result in unpacking a manifest an operation that rarely come cheap. Nevertheless, using a persistent nodemap provide a significant gain for some operations. For our measurementw, we use `hg cat --rev REV FILE` on the our reference mozilla-try. On this repository the persistent nodemap cache is about 29 MB in side for a total store side of 11,988 MB File with large history (file: b2g/config/gaia.json, revision: 195a1146daa0) no optimisation: 0.358s using mmap for index: 0.297s (-0.061s) persistent nodemap for changelog only: 0.275s (-0.024s) persistent nodemap for manifest too: 0.258s (-0.017s) File with small history (file: .hgignore, revision: 195a1146daa0) no optimisation: 0.377s using mmap for index: 0.296s (-0.061s) persistent nodemap for changelog only: 0.274s (-0.022s) persistent nodemap for manifest too: 0.257s (-0.017s) Same file but using a revision (8ba995b74e18) with a smaller manifest (3944829 bytes vs 10 bytes) no optimisation: 0.192s (-0.185s) using mmap for index: 0.131s (-0.061s) persistent nodemap for changelog only: 0.106s (-0.025s) persistent nodemap for manifest too: 0.087s (-0.019s) Differential Revision: https://phab.mercurial-scm.org/D8410

nodemap: create files in the repository used in the test We need a manifest with more content to test persistent nodemap for manifest. This change the repository content and affect all the hashes. Differential Revision: https://phab.mercurial-scm.org/D8409

rust-matchers: add timing tracing to regex compilation This might be useful to diagnose later performance issues or just to show the difference between engines. Differential Revision: https://phab.mercurial-scm.org/D8498

merge with stable

merge with stable

copies: fix the changeset based algorithm regarding merge In 99ebde4fec99, we changed the list of files stored into the `files` field. This lead to the changeset centric copy algorithm to break in various merge situation involving merge. Older information could reach the merge through `p1`, and while information from `p2` was strictly fresher, it would get overwritten anyway. We update the situation with more details about which revision introduces rename information. This help use making the right decision in case of merge. We are now running a more comprehensive suite of test with include this kind of situation. The behavior differ slightly from the filelog based in a couple of instance. There is mostly two distinct cases: 1) there are conflicting rename information in a merge (different rename history on each side). In this case the filelog based implementation arbitrarily pick a side based on the file-revision-number. So it depends on a local factor. The changeset centric algorithm will use a deterministic approach, by picking the information coming from the first parent of the merge. This is stable across different clone. 2) rename information related to file that exist in both source and destination. The filelog based implementation do not even try to detect these, however the changeset centric one get them for "free" (it is simpler to detect them than not). The new implementation focus on correctness. Performance improvement will come later. Differential Revision: https://phab.mercurial-scm.org/D8244

merge with stable

rust-chg: clean up excessive indents Differential Revision: https://phab.mercurial-scm.org/D8450

rust-chg: do not terminate tokio runtime until pager exits We no longer need to spawn a task just to keep the pager handle. The pager handle can be held by ChgUiHandler since the handler itself is not consumed and recreated across async calls. Differential Revision: https://phab.mercurial-scm.org/D8449

rust-chg: modernize entry function Finally the entire build passes. There's a bug that run() no longer waits for the spawned pager, which will be fixed by the next patch. Differential Revision: https://phab.mercurial-scm.org/D8448

rust-chg: reimplement locator by using async/await and tokio-0.2 connect_spawned() is rewritten from scratch by using std::process. Before, it would select completion of either connection or server process. New code could be implemented as such, but it's much simpler to occasionally run try_wait() to detect server death. Differential Revision: https://phab.mercurial-scm.org/D8447

rust-chg: reimplement ChgClientExt as ChgClient wrapper ChgClient is no longer an extension trait because: a. Client object is not consumed and recreated in future-0.3 world, which unblocks writing a simple wrapper struct. b. async fn isn't allowed in trait. Overall, the API should become simpler. Differential Revision: https://phab.mercurial-scm.org/D8446

rust-chg: reimplement run_command operation as async function The crafted state machine is no longer needed thanks to async/await. The state machine is basically rewritten as follows: - Ready(..) -> return .. - PollAgain(..) -> run .. and await - Err(..) -> return Err(..) Differential Revision: https://phab.mercurial-scm.org/D8445

rust-chg: reimplement uihandler by using async-trait and tokio-0.2 We no longer have to consume self and arguments. Differential Revision: https://phab.mercurial-scm.org/D8444

rust-chg: have attach_io() simply take reference of AsRawFd object We no longer have to deal with the restriction of the Future type. Before, these file objects couldn't be references and that's the only reason why we had to make stderr an Option<T>. This fixes future type deduction issue of stderr = None, where rustc would complain that T of Option<T> couldn't be deduced. Differential Revision: https://phab.mercurial-scm.org/D8443

rust-chg: reimplement attach_io operation as async function In short, MessageLoop<Connection> was redesigned as Protocol<Connection>, and the protocol methods no longer consume self. API changes are briefly documented in the following page: https://docs.rs/tokio-hglib/0.3.0/tokio_hglib/struct.Protocol.html Differential Revision: https://phab.mercurial-scm.org/D8442

rust-chg: upgrade to futures-0.3 based libraries And do some trivial fixes: - BytesMut::put_u32_be() -> put_u32() - tokio_process -> tokio::process, CommandExt -> Command, spawn_async() -> spawn(), stdin() -> stdin - tokio_timer::sleep() -> tokio::time::delay_for() Differential Revision: https://phab.mercurial-scm.org/D8441

rust-chg: exclude futures-dependent modules from build and break things It's impractical to upgrade the codebase incrementally since futures 0.1 and 0.3 APIs are fundamentally different. So this patch temporarily excludes futures-dependent modules from the build. These modules will be upgraded and re-enabled one by one. Differential Revision: https://phab.mercurial-scm.org/D8440

commit: tell user what to do with .hg/last-message.txt I have always assumed that the message will be reused by the next `hg commit`, but it seems it's just silently dropped on the next commit. Let's try to be more helpful by telling the user that they have to manually tell hg to reuse it. The file will still be lost if the user runs some other operation in between (like a non-in-memory rebase). That will be fixed once we've switched all operations to be in-memory :) I didn't include `$(hg root)/` in the path in the message to the user because that would have made the message too long. Hopefully the user will figure that part out themselves. Differential Revision: https://phab.mercurial-scm.org/D8463

test-check-rust-format: specify --edition=2018 rustfmt doesn't read Cargo.toml unless it's executed by cargo. https://github.com/rust-lang/rustfmt#rusts-editions

convert: use bytes for value in extra dict The keys and values of the extra dict are expected to be bytes. Before this fix, there was a crash in mercurial.changelog.encodeextra().

archival: abort if compression method is unavailable `tarfile.CompressionError` is documented to be the "exception for unavailable compression methods". Also, make tests conditional on whether the lzma module is available or not.

demandimport: ignore `lzma` module for demandimport This makes importing the module fail if the `_lzma` module is not present. This makes e.g. tarfile correctly recognize if LZMA support is not present. It changes the exception File "/usr/lib/python3.6/tarfile.py", line 1694, in xzopen fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset) AttributeError: module 'lzma' has no attribute 'LZMAFile' to the more correct exception File "/usr/lib/python3.6/tarfile.py", line 1692, in xzopen raise CompressionError("lzma module is not available") tarfile.CompressionError: lzma module is not available Also, it prevents that the error "abort: No module named '_lzma'!" is shown when a development warning is to be shown. The reason why that happened is that for showing the warning, we get information about the stack frames from the inspect module, which accesses the `__file__` attribute of all modules in `sys.modules` to build some cache, causing all modules (including `lzma`) to be imported.

py3: fix crash when server address is 0.0.0.0 (issue6362) `socket.getfqdn()` assumes that the name is passed as `str` on Python 3 and always returns `str` in this case. Mercurial passed `bytes` (but still expected a `str` result), which worked by chance in many cases, except for e.g. b'0.0.0.0', which was returned unchanged, breaking later code. Instead of calling `socket.getfqdn()`, we can also use `self.server_name` from the base `HTTPServer` class, which already stores the FQDN of the locally-bound socket name (see `BaseHTTPServer.py` in the Python 2 stdlib and `http/server.py` in the Python 3 stdlib).

run-tests: find python binary on Python 3 (issue6361) Return strings from _findprogram as all callers expect unicode strings. Previously the check in _usecorrectpython agains sysexecutable was always false on Python 3. Differential Revision: https://phab.mercurial-scm.org/D8674

tests: ignore possible diagnostics from gpg 2.2 Differential Revision: https://phab.mercurial-scm.org/D8672

Added signature for changeset 0ea9c86fac89

Added tag 5.4.2 for changeset 0ea9c86fac89

convert: handle percent-encoded bytes in file URLs like Subversion 75b59d221aa3 added most of the code that gets removed by this patch. It helped making progress on Python 3, but the reasoning was wrong in many ways. I tried to retract it while it was queued, but it was too late. Back then, I was asssuming that what happened on Python 2 (preserving bytes) is correct and my Python 3 change is a hack. However it turned out that Subversion interprets percent-encoded bytes as UTF-8. Accepting the same format as Subversion is a good idea. Consistency with urlreq.pathname2url() (as described in the removed comment) doesn’t matter because that function is only used for passing paths to urllib. This is not a backwards-incompatible change because before 5c0d5b48e58c, non-ASCII filenames didn’t work at all on Python 2. When the locale encoding is ISO-8859-15, `svn` accepts `file:///tmp/a%E2%82%AC` for `/tmp/a€`. Before this patch, this was the case for this extension on Python 3, but not on Python 2. This patch makes it work like with `svn` on both Python 2 and Python 3.

convert: add docstring on convert.subversion.geturl() The function is unusual for a bytes-handling function in Mercurial because it can’t handle arbitrary bytes. Therefore we should document this fact. Pointed out by Yuya Nishihara while reviewing e3b19004087a.

tests: use path inside test dir This will make the diff for the next patch less noisy.

convert: convert URLs to UTF-8 for Subversion Preamble: for comprehension, note that the `path` of geturl() would better be called `path_or_url` (the argument of the call of getsvn() is called `url`). For HTTP(S) URLs, the changes don’t make a difference, as they are restricted to ASCII. For file URLs, the reasoning is the same as for paths: we have to roundtrip with what Subversion is doing. When the locale encoding is ISO-8859-15, trying to convert a SVN repo `file:///tmp/a€` failed before like this: file:///tmp/a%A4 does not look like a Subversion repository to libsvn version 1.14.0 Decoding the path using the locale encoding can fail. In this case, we have to bail out, as Subversion won’t be able to do anything useful with the path.