patch: make __repr__() return str Caught by pytype: line 969, in __repr__: Function bytes.join was called with the wrong arguments [wrong-arg-types] Expected: (self, iterable: Iterable[bytes]) Actually passed: (self, iterable: Iterator[str]) Differential Revision: https://phab.mercurial-scm.org/D7682

pytype: suppress warnings about no 'open_binary' on importlib.resources Fixes these pytype warnings: line 43, in <module>: No attribute 'open_binary' on module 'importlib.resources' [module-attr] line 47, in open_resource: No attribute 'open_binary' on module 'importlib.resources' [module-attr] For some reason, I can't upgrade from 3.6.8 in my WSL environment. Differential Revision: https://phab.mercurial-scm.org/D7681

windows: if username(uid=None) is loaded, just use getpass This is at least consistent with what we do on other platforms in the base case. I don't know enough about Windows to fill in other cases that might exist here, but this at least should be a start. Differential Revision: https://phab.mercurial-scm.org/D7679

transplant: use check_incompatible_arguments() Differential Revision: https://phab.mercurial-scm.org/D7663

bookmarks: use check_incompatible_arguments() for inactive+action Differential Revision: https://phab.mercurial-scm.org/D7662

bookmarks: use cmdutil.check_incompatible_arguments() for action+rev Differential Revision: https://phab.mercurial-scm.org/D7648

bookmarks: use cmdutil.check_at_most_one_arg() for action Differential Revision: https://phab.mercurial-scm.org/D7647

rebase: use cmdutil.check_at_most_one_arg() for action Here we also needed to know what the action was (if any), so I've updated the helper to return any specified option. Differential Revision: https://phab.mercurial-scm.org/D7640

releasenotes: extract helper for checking for incompatible arguments This patch extracts a new check_incompatible_arguments() function similar to check_at_most_one_arg(). The difference is that the new function is for checking for arguments that are disallowed together with some other argument but not mutually exclusive among themselves. Differential Revision: https://phab.mercurial-scm.org/D7639

fix: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7638

patchbomb: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7637

export: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7636

amend: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7635

commit: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7634

clone: extract helper for checking mutually exclusive args We have some duplicated code for aborting if the user provided mutually exclusive arguments. Extensions surely have more such code. We also have duplicated translations and inconsistent output in this area. This patch introduces a simpler helper for checking if more than one option among a given set was given on the command line. I've made the clone code call the function to show that it works. The function has no good way of checking arguments with hyphens in them. I'll add that later if necessary. The function still won't be applicable in all cases, but I think it's still better than nothing. Differential Revision: https://phab.mercurial-scm.org/D7633

dirstate: when calling rebuild(), avoid some N^2 codepaths I had a user repo with 200k files in it. Calling `hg debugrebuilddirstate` took tens of minutes (I didn't wait for it). In that situation, changedfiles==allfiles, and both are lists. This meant that we had to run an average of 100k comparisons, for each of 200k files, just to check whether a file needed to have normallookup called (it always did), or drop. While it's probably not a huge issue, in my very awkward synthetic benchmark I wrote (not using a benchmark library or anything), I was seeing some slowdowns for small-changedfiles and very-large-allfiles invocations, with an inflection somewhere around 10 items in changedfiles (regardless of the size of allfiles); above 10 items in changedfiles, the new code appears to always be faster. For the case of 50k files in changedfiles and the same items in allfiles, I'm seeing differences of 15s of just running comparisons vs. 0.003793s. I haven't bothered to run a comparison of 200k items in changedfiles and allfiles. :) Differential Revision: https://phab.mercurial-scm.org/D7665

rust-warnings: fix warnings in tests It turns out that I also missed those warnings inside tests. This should be the last of them. One day we will get rid of this interface anyway. Differential Revision: https://phab.mercurial-scm.org/D7678

relnotes: mention the merging of index and nodemap

crecord: remove toggleamend Previous commit removed its only calling site.

crecord: repurpose "a" key to toggle all selections (BC) I really don't like "a". I keep accidentally hitting it when I actually want "A", and then I'm suddenly in a state I don't want to be in. There's a big wall of text telling me that I've turned amend mode on or off (which one was I orginally in?), and this seems very useless. If I wanted to amend or not, I would have chosen that from the command-line, not change my mind after I've already started picking hunks apart. Furthermore, for most uses of the hunk selector (revert, uncommit, shelve/unshelve), this amend toggle doesn't make sense. It seems much better to repurpose this key to be a "weaker" version of "A". It toggles all selections. This is pretty harmless if hit accidentally, (can just hit "a" again to toggle everything and undo it), and has immediate visual feedback that something happened: all the x's and blank spaces get switched around. And unlike with amend, the current flipped state is also immediately visible without having to read a wall of text. I'm calling this a BC, however, because somewhere, someone out there has probably really fallen in love with the old use of "a" and will get angry that we took it away.

hgweb: fix error in docstring Despite the subtle semantic difference, this sentence really meant to say "overridden", not "overwritten".

crecord: rewrite help string to avoid mentioning "crecord" Despite its heritage, "crecord" is now mostly Mercurial-internal jargon. I find it better to call it "the curses hunk selector". Also slightly rewrote the part about which commands can use it. While I do believe that commit, shelve, and revert are the only commands in core that can use it, Evolve also adds at least amend and uncommit to the list.

update: add some tests for the status quo of morestatus on update conflicts Differential Revision: https://phab.mercurial-scm.org/D7666

typing: suppress a warning that mercurial.windows.checkosfilename is missing This function is used to check filename portability everwhere, so it isn't tucked into the windows.py module. I supposed the alternative is to move it and then alias it in `util`. I'm guessing it was done like this to save an import. Differential Revision: https://phab.mercurial-scm.org/D7677

typing: add an assertion to util.versiontuple Fixes the following warning: line 1177, in versiontuple: No attribute 'split' on None [attribute-error] In Optional[bytes] Differential Revision: https://phab.mercurial-scm.org/D7676

typing: add a couple of assertions to lrucachedict to help pytype Fixes the following warnings: line 1488, in pop: No attribute 'markempty' on None [attribute-error] In Optional[Union[Any, _lrucachenode, nothing]] line 1578, in popoldest: No attribute 'markempty' on None [attribute-error] In Optional[Union[Any, _lrucachenode, nothing]] Differential Revision: https://phab.mercurial-scm.org/D7674

util: rename a variable to avoid confusing pytype Fixes the following warning: line 1205, in f: No attribute 'append' on Dict[nothing, nothing] [attribute-error] In Union[Dict[nothing, nothing], list] Differential Revision: https://phab.mercurial-scm.org/D7673

pytype: suppress the import-error in util.py when importing re2 Differential Revision: https://phab.mercurial-scm.org/D7672

debugcommands: finish moving `extendeddateformats` from util to dateutil Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7671

ui: convert exception data to bytes when printing chained exception info Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7670

ui: use a context manager to handle file streams in edit() Differential Revision: https://phab.mercurial-scm.org/D7669

graft: reuse cmdutl.resolvecommitoptions() Differential Revision: https://phab.mercurial-scm.org/D7632

rebase: make rebase not crash if p1 == p2 != nullid While this shouldn't happen normally, some historical bugs can have caused this kind of commit to exist. Instead of crashing and having it be unobvious how to recover, let's try to continue on. Without this, we get an error like "ValueError: min() arg is an empty sequence" Differential Revision: https://phab.mercurial-scm.org/D7664

merge: add commands.merge.require-rev to require an argument to hg merge This is related to commands.rebase.requiredest, commands.update.requiredest, and commands.push.require-revs. Since it isn't really a "destination", I went with require-rev to be similar to push's require-revs. Differential Revision: https://phab.mercurial-scm.org/D7620

tests: make test-config-env.py a little less hacky Overriding os.path.isdir to always returns False is very confusing. Let's make it more explicit by specifically not returning any defaultrc/ configs. Differential Revision: https://phab.mercurial-scm.org/D7623

help: get helptext/ data from `resources` module if available For PyOxidizer, we need to read configs using the `resources` module. This patch makes it so we use that module if available (i.e. Python >= 3.7). It does that by adding a new `open_resource()` function to our `resourceutil` module. Tested by running `$PYTHON ./hg help pager` for each $PYTHON in {python2, python3.6, python3.7}. Differential Revision: https://phab.mercurial-scm.org/D7622

pycompat: allow pycompat.sysbytes() even if input already is bytes pycompat.sysstr() on py3 accepts an input that's already str (i.e. unicode). This patch makes it so pycompat.sysbytes() on py3 accepts an input that's already bytes. Allowing that makes it possible to do pycompat.sysbytes(fp.name) where fp.name is either bytes or unicode, as we'll get when fp can come from either open() or resources.open_binary(). Differential Revision: https://phab.mercurial-scm.org/D7621

merge with stable

rust-dirs: handle forgotten `Result`s In 1fe2e574616e I introduced a temporary bugfix to align Rust code with a new behavior from C/Python and forgot about a few `Result`s (cargo's compiler cache does not re-emit warnings on cached modules). This fixes it. For the record, I am still unsure that this behavior change is a good idea. Note: I was already quite unhappy with the setters and getters for the `DirstateMap` and, indirectly, `Dirs`, and this only further reinforces my feelings. I hope we can one day fix that situation at the type level; Georges Racinet and I were just talking about devising a POC for using the builder pattern in the context of FFI with Python, we'll see what comes out of it. Differential Revision: https://phab.mercurial-scm.org/D7609

merge with stable

status: outputting structured unfinished-operation information This adds a new item in the json/template output for morestatus and added item types to all entries. Differential Revision: https://phab.mercurial-scm.org/D7595

chg: fix chg to work with py3.7+ "coercing" the locale When the environment is empty (specifically: it doesn't contain LC_ALL, LC_CTYPE, or LANG), Python will "coerce" the locale environment variables to be a UTF-8 capable one. It sets LC_CTYPE in the environment, and this breaks chg, since chg operates by: - start hg, using whatever environment the user has when chg starts - hg stores a hash of this "original" environment, but python has already set LC_CTYPE even though the user doesn't have it in their environment - chg calls setenv over the commandserver. This clears the environment inside of hg and sets it to be exactly what the environment in chg is (without LC_CTYPE). - chg calls validate to ensure that the environment hg is using (after the setenv call) is the one that the chg process has - if not, it is assumed the user changed their environment and we should use a different server. This will *never* be true in this situation because LC_CTYPE was removed. Differential Revision: https://phab.mercurial-scm.org/D7550

fuzz: add support for fuzzing under either Python 2 or 3 This was more of a hairball than I hoped, but it appears to work. The hg-py3 branch of my oss-fuzz fork on github has the remaining changes to switch us to Python 3, but we may as well retain Python 2 fuzzing support for at least a little while. Differential Revision: https://phab.mercurial-scm.org/D7592

phabricator: color the status in the "phabstatus" view I couldn't figure out strikethrough for "abandoned" like I've see with word diff. Differential Revision: https://phab.mercurial-scm.org/D7608

phabricator: add the "Changes Planned" status name I noticed this when coloring the status output. But it looks like status names are also used in the pseudo revset language as well, and abort when parsing a name not in this list. Differential Revision: https://phab.mercurial-scm.org/D7607

fuzz: fix mpatch_corpus to not have an overridden __repr__ on py3 Differential Revision: https://phab.mercurial-scm.org/D7606

formatting: fix some recent formatting regressions Differential Revision: https://phab.mercurial-scm.org/D7605

cext-revlog: fixed __delitem__ for uninitialized nodetree This is a bug in a code path that's seldom used, because in practice (at least in the whole test suite), calls to `del index[i:j]` currently just don't happen before the nodetree has been initialized. However, in our current work to replace the nodetree by a Rust implementation, this is of course systematic. In `index_slice_del()`, if the slice start is smaller than `self->length`, the whole of `self->added` has to be cleared. Before this change, the clearing was done only by the call to `index_invalidate_added(self, 0)`, that happens only for initialized nodetrees. Hence the removal was effective only from `start` to `self->length`. The consequence is index corruption, with bogus results in subsequent calls, and in particular errors such as `ValueError("parent out of range")`, due to the fact that parents of entries in `self->added` are now just invalid. This is detected by the rebase tests, under conditions that the nodetree of revlog.c is never initialized. The provided specific test is more direct. Differential Revision: https://phab.mercurial-scm.org/D7603

filemerge: fix a missing attribute usage Flagged by both pytype and VSCode. Differential Revision: https://phab.mercurial-scm.org/D7465

filemerge: drop a default argument to appease pytype The function slices and takes the length of this argument without internally setting it if not provided. There was no bug here because both callers passed the argument. Differential Revision: https://phab.mercurial-scm.org/D7464

fuzz: add a seed corpus for the dirs fuzzer I was hoping to trigger an asan violation under Python 3 that some internal tests at Google found, but for some reason that's beyond me I can't seem to manage. Differential Revision: https://phab.mercurial-scm.org/D7600

fuzz: clean up production of seed corpora This was getting out of hand. Differential Revision: https://phab.mercurial-scm.org/D7599

status: add template/json data about whether a file has unresolved conflicts Differential Revision: https://phab.mercurial-scm.org/D7594

status: split morestatus data loading from display This is a small refactoring in preparation for adding more morestatus functionality (notably for templated/JSON output) - the goal is to use the data inside the status display loop, as well as output the overall state in a templatable/structured way. Differential Revision: https://phab.mercurial-scm.org/D7593

phabricator: add a "phabstatus" template keyword We add a "phabstatus" template keyword, returning an object with "url" and "status" keys. This is quite similar to "phabreview" template keyword, but it queries phabricator for each specified revision so it's going to be slow (as compared to the "phabstatus" show view from previous changeset). Differential Revision: https://phab.mercurial-scm.org/D7507

phabricator: add a "phabstatus" show view We add a "phabstatus" show view (called as "hg show phabstatus") which renders a dag with underway revisions associated with a differential revision and displays their status. The revisions shown is a subset of that shown by "work" view, only including revisions with known by Phabricator. Differential Revision: https://phab.mercurial-scm.org/D7506

dirs: fix out-of-bounds access in Py3 The hack for mutating Python's variable-length integers that was ported to py3 in cb3048746dae (dirs: port PyInt code to work on Python 3, 2016-10-08) was reading from ob_digit[1] instead of ob_digit[0] for some reason. Space for ob_digit[1] would only be allocated for integers larger than 30 bits, so we ended up writing to unallocated memory. Also, we would write an integer that's 2^30 times too large, so we would never free these integers. Found by AddressSanitizer. Differential Revision: https://phab.mercurial-scm.org/D7597

manifestcache: add some test involving shares Some issue around shares have been reported. I am adding tests to better cover the share case. The test show that the code behave as expected so far. Differential Revision: https://phab.mercurial-scm.org/D7602

fuzz: fix test-fuzz-targets.t to run with python3 Differential Revision: https://phab.mercurial-scm.org/D7601

py3: fix a bytes vs str issue in remotefilelog extension Differential Revision: https://phab.mercurial-scm.org/D7598

rust-matchers: add doctests for `AlwaysMatcher` Differential Revision: https://phab.mercurial-scm.org/D7527

rust-hg-path: add method to get part of a path relative to a prefix This will be used in the next patch in this series. Differential Revision: https://phab.mercurial-scm.org/D7526

rust-matchers: improve `Matcher` trait ergonomics `VisitChildrenSet` has no need to own the set, this will save allocations. The `file_set` return type change is motivated by both ergonomics and... being able to compile code. The `AlwaysMatcher` does not store a `file_set`, which requires it to return an owned `HashSet`, which in turn would change our return type to `Cow<&HgPath>` (lifetimes omitted). This is both un-ergonomic and troublesome for more complex lifetime issues (especially with the upcoming `FileMatcher` in the following patch). Differential Revision: https://phab.mercurial-scm.org/D7525

rust-dirs-multiset: use `AsRef` instead of concrete types when possible I also renamed `vec` to `dirstate`, because it was not a great name. Differential Revision: https://phab.mercurial-scm.org/D7524

logcmdutil: call _exthook() in changesettemplater Class changesetprinter has an _exthook() method that is called in _show() before the patch is displayed. Call the method as well in changesettemplater._show(). Differential Revision: https://phab.mercurial-scm.org/D7505

phabricator: fix processing of tags/desc in getoldnodedrevmap() It seems that the previous logic was wrong (it essentially comes from changeset 3ab0d5767b54 where the result got accumulated instead of early returned). First of all, the "continue" in first "if m:" is useless because we're at the end of the loop. Then, the algorithm seems weird because we will process all predecessors of a node and possibly override `toconfirm[node]` for each of these having a tag (maybe this doesn't happen, but still). Finally, we would also override `toconfirm[node]` when the "Differential Revision: " is found in changeset description. Maybe this is not a big deal when there is no mix of local tag and changeset description update? The logic is changed so that the loop on predecessors stops upon first match of a tag and so that the changeset description is only checked if no tag was found. Therefore, `toconfirm[node]` is only set once. Differential Revision: https://phab.mercurial-scm.org/D7513

tests: replace [[]] bashism with portable [] invocation In this case nothing fancy is required. Differential Revision: https://phab.mercurial-scm.org/D7596

rust-hg-path: implement `Display` for `HgPath` and `HgPathBuf` This is useful when debugging, to get a human readable output instead of an array of `u8`. Differential Revision: https://phab.mercurial-scm.org/D7523

rust-performance: introduce FastHashMap type alias for HashMap Rust's default hashing is slow, because it is meant for preventing collision attacks. For all of the current Rust code, we don't care about those attacks, because if an person with bad intentions has write access to your repo, you have other issues. I've chosen to use the TwoXHash crate because it was made by a reputable member of the Rust community and has very good benchmarks. For now it does not seem to improve performance by much for the current code, but it's something else to not worry about when benchmarking code: in a previous experiment with copytracing in Rust, it accounted for more than 10% of the time of the entire script. Differential Revision: https://phab.mercurial-scm.org/D7116

mail: use procutil.shellsplit instead of bytes.split to parse command Differential Revision: https://phab.mercurial-scm.org/D7541

annotate: describe --skip as taking a revset It's obvious to me, but probably wouldn't be obvious to a novice user. Differential Revision: https://phab.mercurial-scm.org/D7557

automation: use latest Windows AMI as base It looks like the previous base image no longer exists. I guess Amazon expires them or something. Let's switch to the newest version of the equivalent image. Differential Revision: https://phab.mercurial-scm.org/D7571

tests: stabilize test-extdiff.t on Windows This goes with 765a9c299c44. I'm not sure if we care about the missing file being spelled `nul`, so I removed it from the glob to be explicit about it. The line needed to be special cased anyway because of the quoting on Windows. Differential Revision: https://phab.mercurial-scm.org/D7572

exchange: fix an attempt to format a list into bytes This goes with 27c6d6f53d46. Differential Revision: https://phab.mercurial-scm.org/D7583

status: remove pointless filtering by alwaysmatcher in morestatus The code has been like this since it was imported from FB's hg-experimental repo. It has been like that even since it was added in that repo. So I don't know why it looks that way. Perhaps the idea was to one day filter the unresolved paths by any patterns provided by the user. We can add a matcher back if we ever decide to do that. Differential Revision: https://phab.mercurial-scm.org/D7591

tests: add test for Rust formatting We enforce formatting for Python and C. It makes sense to do it for Rust as well. Since our rustfmt.toml relies on unstable rustfmt features, we need to use a Nightly rustfmt with --unstable-features in order for it to work. This is a bit hacky and I would prefer we remove this requirement. But for now, this commit assumes this is the way things must be and we go out of our way to detect and use the rustfmt from the "nightly" toolchain, as installed via rustup. We had to add some environment variables to the tests to make the Rust binaries happy. Otherwise when running rustfmt we get an error about no default toolchain being installed. Differential Revision: https://phab.mercurial-scm.org/D7579

rust: run rustfmt # skip-blame automated reformatting Differential Revision: https://phab.mercurial-scm.org/D7578

fuzz: clean up some repetition on building parsers.so fuzzers There was a lot of repetition here that I realized could be cleared out. I think there's some more work we could do here, but this is enough for now. Differential Revision: https://phab.mercurial-scm.org/D7568

fuzz: remove legacy setup for running fuzzers outside oss-fuzz We don't need this anymore. Differential Revision: https://phab.mercurial-scm.org/D7567

tests: finally fix up test-fuzz-targets.t It's been failing on my workstation for a while, since I have a new enough LLVM that I had the fuzzer goo, but not so new that I actually had FuzzedDataProvider. This is a better solution all around in my opinion. I _believe_ this should let us run these tests on most systems, even those using GCC instead of clang. That said, my one attempt to test this on my macOS laptop failed miserably, and I don't feel like doing more work on this right now. Differential Revision: https://phab.mercurial-scm.org/D7566

fuzz: make standalone_fuzz_target_runner call LLVMFuzzerInitialize Otherwise some of our fuzzers crash when they try and use Python. Differential Revision: https://phab.mercurial-scm.org/D7565

fuzz: use a more standard approach to allow local builds of fuzzers This is taken from the (improved since we started fuzzing) guide on ideal integrations. Rather than have our own wonky targets for building outside the fuzzer universe, we have a driver program we carry along and use when we're not using LibFuzzer. This will let us jettison a fair amount of goo. contrib/fuzz/standalone_fuzz_target_runner.cc is https://github.com/google/oss-fuzz/ file projects/example/my-api-repo/standalone from git revision c4579d9358a73ea5dbcc99cb985de1f2bf76dcf7, reformatted with out clang-format settings and a no-check-code comment added. It allows running a single test input through a fuzzer, rather than performing ongoing fuzzing as libfuzzer would. contrib/fuzz/FuzzedDataProvider.h is https://github.com/llvm/llvm-project/ file /compiler-rt/include/fuzzer/FuzzedDataProvider.h from git revision a44ef027ebca1598892ea9b104d6189aeb3bc2f0, reformatted with our clang-format settings and a no-check-code comment added. We can discard this if we instead want to add an hghave check for a new enough llvm that includes FuzzedDataProvder.h in the fuzzer headers. Differential Revision: https://phab.mercurial-scm.org/D7564

fuzz: use a variable to allow specifying python-config to use Eventually we should probably default this to just `python-config` and have the oss-fuzz build.sh script specify the sanpy python-config, but for now this lets us make progress. Differential Revision: https://phab.mercurial-scm.org/D7563

fuzz: suppress deprecated-register warnings in our compile These come from the Python.h headers still using the `register` keyword and our use of C++17. I think this will go away when we're using Python 3 for our fuzzing, but that can come later. Differential Revision: https://phab.mercurial-scm.org/D7562

fuzz: follow modern guidelines and use LIB_FUZZING_ENGINE This will make our lives easier in an upcoming change, but it's also how we're supposed to set things up anyway. Differential Revision: https://phab.mercurial-scm.org/D7561

fuzz: always define LLVMFuzzerInitialize() even if we don't need it This will make it easier to test our fuzzers outside oss-fuzz. Differential Revision: https://phab.mercurial-scm.org/D7560

fuzz: remove debug prints from revlog_corpus.py Differential Revision: https://phab.mercurial-scm.org/D7559

fuzz: fix an unused result on getcwd() in pyutil clang was rightly complaining about this, so let's just fix it. Differential Revision: https://phab.mercurial-scm.org/D7558

upgraderepo: add a config option for parallel computation The option is put to use to compute new copy tracing side data in parallel. It use the multiprocessing module as it had the appropriate primitive for what we needed. Gregory Szorc had concerned on windows so we disabled it there. See inline comment for details on the parallel implementation.

upgrade-repo: colorize some of the output Having clear color for requirement added and removed is useful.

tests: remove hardcoded errno values Fixes test failures on hurd-i386 Ref: https://bugs.debian.org/946178 Differential Revision: https://phab.mercurial-scm.org/D7556

tests: expect return status 255 on exception for test-blackbox.t with chg Differential Revision: https://phab.mercurial-scm.org/D7554

tests: fix command name in test-blackbox.t to be bytes This command is only used when tests are run with --with-chg, so this was missed before. Differential Revision: https://phab.mercurial-scm.org/D7553

tests: fix deprecation warning about regex flags not at beginning of expr This may only show up when running the tests under python3.6+. Currently the only test that does this is test-patchbomb-tls.t, and it only uses (?i), so that's all that's handled at the moment. Differential Revision: https://phab.mercurial-scm.org/D7552

tests: fix test-chg.t to work with py3 (no setprocname) Differential Revision: https://phab.mercurial-scm.org/D7551

amend: check for file modifications when updating dirstate (issue6233) Previously, we called dirstate.normal(f), which would put information into the dirstate claiming that the file on disk is what it "should be" for the current checkout, and it would have the size and timestamp of the most recent modification to the file (which is not necessarily the one we just committed). If the file was modified while the commit message editor was open, we would put incorrect information into the dirstate. Differential Revision: https://phab.mercurial-scm.org/D7521

graft: never set both parents equal in the dirstate (issue6098) merge.graft() can set both parents equal in the dirstate when keepparent=True. We don't seem to set that in core, but the evolve extension does use it. So I couldn't figure out a way to add a test for this patch in core. Differential Revision: https://phab.mercurial-scm.org/D7549

graft: extract `repo['.']` to local variable It's used in two places and I'm about to use it more. Differential Revision: https://phab.mercurial-scm.org/D7548

graft: rename `pctx` argument to `base` since that's what it is The new name better matches the docstring. It also frees up `pctx` to be used for something else (next patch). Differential Revision: https://phab.mercurial-scm.org/D7547

exchange: replace a "not x in ys" by more Pythonic "x not in ys" Found by one of our (Google-)internal tools. Differential Revision: https://phab.mercurial-scm.org/D7546

packaging: include defaultrc/*.rc instead of default.d/*.rc Differential Revision: https://phab.mercurial-scm.org/D7555

merge with stable

py3: make a pycompat.osdevnull, use it in extdiff Differential Revision: https://phab.mercurial-scm.org/D7545

subrepo: use pycompat.open directly instead of importing open from pycompat Differential Revision: https://phab.mercurial-scm.org/D7544

rust-dirs: address failing tests for `dirs` impl with a temporary fix https://phab.mercurial-scm.org/D7252 (5d40317d42b7083b49467502549e25f144888cb3) introduced a regression in Rust tests. This is a temporary fix that replicates the behavior of the C and Python impl, pending the resolution of the discussion (in the phabricator link) about how we actually want to solve this problem. Differential Revision: https://phab.mercurial-scm.org/D7503

cleanup: fix docstring formatting This is just removing the b'' prefix (except demandimportpy2), and making sure it is triple quoted. I skipped the mapping.py module in zope because that's 3rd party code. Differential Revision: https://phab.mercurial-scm.org/D7539

copies: split the combination of the copies mapping in its own function In some case, this part take up to 95% of the copy tracing that take about a hundred second. This poor performance comes from the fact we keep duplciating and merging dictionary that are mostly similar. I want to experiment with smarter native code to do this, so I need to isolate the function first.

copies: do not initialize the dictionary with root in changeset copies We don't strictly need the dict to be prefiled with `{}`. Not doing so will make the next changeset simpler. This part of a preparation to use native code for this part of copy tracing.

copies: expand `_chain` variable name to make the function easier to read This came up while explaining what the function is about. I find the function easier to follow that way.

copies: show we can read data from the "first" mapping In practice this make no difference since `t` is a copy of `a`. Having this change before hand will make the next cleanup changeset clearer. This is work on the road to using some native code for some performance critical part of copy tracing.

copies: return consistent type from revinfo The return is expected to be a list, so we should return a list. This make the use of this function from type-checked language (eg: rust) much simpler.

dateutil: correct default for Ymd in parsedate The code uses `0` for the default value of Ymd (year, month, and day), which seems suboptimal. For example, these will fail to parse: dateutil.parsedate('2000', formats=dateutil.extendeddateformats) dateutil.parsedate('Jan 2000', formats=dateutil.extendeddateformats) Fix it by providing sane defaults (1 instead of 0) for year, month, and day. The suboptimal behavior was introduced by 91bc001a592 (2010-12-29, "date: fix matching of underspecified date ranges"), which does not seem to justify the current behavior. Note end-users should not notice the subtle issue, because there are no formats in `defaultdateformats` that allow an explicit year with omitted month, or an explicit month with omitted day. Differential Revision: https://phab.mercurial-scm.org/D7520

test-doctest: include dateutil `mercurial.utils.dateutil` has dostrings that contain doctests. Include them. Differential Revision: https://phab.mercurial-scm.org/D7519

revlog: fix revset in reachableroots docstring `reachableroots` will only return a subset of `roots` when `includepath` is False. For example, given the following linear DAG: 2 | 1 | 0 Using roots=0+2, heads=1, the definition in the docstring does not match what `reachableroots` actually does: ipdb> repo.changelog.reachableroots(0, roots=[0,2],heads=[1]) [0] ipdb> repo.revs('heads(::(0+2) & (0+2)::1)') <baseset+ [1]> The fix is to do `heads & ::roots` (or `heads & heads::roots`) first, then select their ancestors: ipdb> repo.revs('heads(::((0+2) & (0+2)::1))') <baseset+ [0]> The docstring was introduced by fd92bfbbe02d9 (2015-06-19 "revset: rename revsbetween to reachableroots and add an argument"), which introduced the `includepath=False` behavior for graphlog grandparents use-case. I believe the docstring instead of the code should be changed because changing the code to match the docstring can result in suboptimal graphlog like: o :\ : o : : :/ o As opposite to the current "linearized" graphlog: o | o : o Differential Revision: https://phab.mercurial-scm.org/D7518

lock: pass "success" boolean to _afterlock callbacks This lets the callback decide if it should actually run or not. I suspect that most callbacks (and hooks) *should not* run in this scenario, but I'm trying to not break any existing behavior. `persistmanifestcache`, however, seems actively dangerous to run: we just encountered an exception and the repo is in an unknown state (hopefully a consistent one due to transactions, but this is not 100% guaranteed), and the data we cache may be based on this unknown state. This was observed by our users since we wrap some of the functions that persistmanifestcache calls and it expects that the repo object is in a certain state that we'd set up earlier. If the user hits ctrl-c before we establish that state, we end up crashing there. I'm going to make that extension resilient to this issue, but figured it might be a common issue and should be handled here as well instead of just working around the issue. Differential Revision: https://phab.mercurial-scm.org/D7459

relnotes: add note about changes to match.{explicit,reverse}dir Differential Revision: https://phab.mercurial-scm.org/D7508

graphlog: change state dict to attr struct This should help static analysis.

status: fix default value of status struct The default argument isn't overloaded. Before, the default constructor would create a struct having 7 list type objects.

typing: fix return type of logcmdutil.getrevs() Fixes the following errors: Invalid type annotation "'Tuple[smartset.BaseSet, changesetdiffer]'" [invalid-annotation] No attribute 'BaseSet' on module 'mercurial.smartset' getrevs: bad option in return type [bad-return-type] Expected: Tuple[mercurial.smartset.abstractsmartset, changesetdiffer] Actually returned: Tuple[mercurial.smartset.baseset, None]

typing: consolidate "if not globals():" trick Removes redundant inline comments. I think pycompat is good place to host this kind of constants.

rust-cpython: do not convert warning pattern to utf-8 bytes On Unix, both Rust Path and Mercurial expect a locale-dependent bytes, and we don't support Windows yet.

rust-cpython: import utils::files::* function at module level IIRC, it's common in Rust to call functions with the module prefix.

exchange: guard against method invocation on `b2caps=None` args I couldn't figure out how these are called, but the value is pretty obviously set at least for the cases that have tests. Differential Revision: https://phab.mercurial-scm.org/D7512

exchange: eliminate some bytes.format() calls Differential Revision: https://phab.mercurial-scm.org/D7511

windows: suppress pytype warnings for Windows imports and functions This should allow the modules to not be excluded, and not generate complaints on non Windows platforms. Differential Revision: https://phab.mercurial-scm.org/D7510

changectx: mark the parents of the working copy as non filtered If we successfully accessed the working copy, its parents are not filtered. Differential Revision: https://phab.mercurial-scm.org/D7491

repoview: add an explicit set of all filter that show the wc parents The `visible` set will always show the working copy parents. We record this in a specific set. This will allow to fast path some access. Differential Revision: https://phab.mercurial-scm.org/D7490

localrepo: introduce a `_quick_access_changeid` property Having faster access to `null` is cute… but limited. We want to speedup access to more useful revision, like `.`. We start with turning the fast path for `null` into something more generic. Differential Revision: https://phab.mercurial-scm.org/D7488

changectx: use unfiltered changelog to access parents of unfiltered revs If a revision is not filtered, we know that its parents are not either. So we can take a shortcut. This shortcut avoid the computation of all filtered revs in some cases. Differential Revision: https://phab.mercurial-scm.org/D7487

locarepo: also fastpath `nullid` lookup in __getitem__ We already use that fastpath for `"null"` and `nullrev`, using it for `nullid` is similar. Differential Revision: https://phab.mercurial-scm.org/D7486

repoview: add more tests to track operation not supposed to trigger filtering This test is useful to confirm we removed filtering trigger and to prevent it to come back without us noticing. The commands tested in the test were initially introduced one by one. However, on Martin von Zweigbergk request, we are adding them all at once. Differential Revision: https://phab.mercurial-scm.org/D7514

localrepo: mark nullrev has never filtered All repository have a null, and it cannot be filtered. Differential Revision: https://phab.mercurial-scm.org/D7484

changectx: add a "maybe filtered" filtered attribute There are changeset that we know not to be filtered (eg: `null`). In this case, we could access some information without triggering changelog filtering. We add an attribute to changectx that track this property. Differential Revision: https://phab.mercurial-scm.org/D7483

repoview: add a test that access actual changeset data of `null` Ideally, we would not trigger filtering here. However some work needs to happens first. Differential Revision: https://phab.mercurial-scm.org/D7482

localrepo: recognize trivial "null" queries in `anyrev` Bypassing the revset logic for trivial "null" queries means we can avoid to trigger the filtering logic in some cases. Differential Revision: https://phab.mercurial-scm.org/D7481

localrepo: also fastpath `nullrev` in __getitem__ As explained earlier, nullrev will exist in all repository, we do not need any special checking. Differential Revision: https://phab.mercurial-scm.org/D7480

repoview: add a test to track operation not supposed to trigger filtering This test will be useful to confirm we removed filtering trigger and to prevent it to come back without us noticing. Differential Revision: https://phab.mercurial-scm.org/D7479

repoview: display stack trace along side the debug message When a filter computation is triggered, If --traceback is provided, we will display a traceback in addition to the debug message. Differential Revision: https://phab.mercurial-scm.org/D7478

util: add an optional `prefix` argument to debugstacktrace This is useful when using it in a specific context. Differential Revision: https://phab.mercurial-scm.org/D7477

repoview: add a 'devel.debug.repo-filter' option If set, there will be debug message when a filter computation is triggered. This is going to be useful to remove various filtering trigger and to test they do not get reintroduced. Differential Revision: https://phab.mercurial-scm.org/D7476

localrepo: extract handling of some special value in __getitem__ The value "tip" should always be accessible, otherwise there is some problematic bug in the lower level. So we can access this outside of the general try/catch. If it fails some wider and actual big is in play. Differential Revision: https://phab.mercurial-scm.org/D7475

localrepo: extract handling of some special value in __getitem__ The value "null" will always be present in a repository. So this lookup should always succeed and do not need to be in the general try/catch. Differential Revision: https://phab.mercurial-scm.org/D7474

localrepo: add some basic comment for block in __getitem__ There are different early processing before getting to the core of the function. We highlight that fact. Differential Revision: https://phab.mercurial-scm.org/D7473

windows: further build fixes for the WiX installer With these fixes in place, the .msi actually builds for me again. Differential Revision: https://phab.mercurial-scm.org/D7509

cleanup: update references to /help/ that should now be /helptext/ I noticed this only because the WiX build was broken after 2e017696181f. I haven't tested this change with WiX, but it seems pretty obvious by inspection this should fix it. Differential Revision: https://phab.mercurial-scm.org/D7472

merge with stable

tests: byteify a few more things in simplestorerepo.py Differential Revision: https://phab.mercurial-scm.org/D7471

profiling: add a missing argument to the ProgrammingError constructor Differential Revision: https://phab.mercurial-scm.org/D7470

profiling: disable the import-error warning for the flamegraph module The module being missing is properly handled, but pytype was complaining. Differential Revision: https://phab.mercurial-scm.org/D7469

phases: make `allphases` a list on py3 also Pytype complained that `range` doesn't support slicing, so make this a list like it was on py2. Differential Revision: https://phab.mercurial-scm.org/D7468

wireprotov1server: capture Abort type before accessing the `hint` attribute Although the previous code worked, pytypes complained because `exc` is caught above as `BundleValueError, Abort, PushRaced`, and the other two don't have this attribute. Differential Revision: https://phab.mercurial-scm.org/D7463

tests: add more tests for "hg shelve --delete" It appears that the only tests for "hg shelve --delete" concern command errors (e.g. incompatible command options). Adding some more to check that non-existent names are handled and a success case. Differential Revision: https://phab.mercurial-scm.org/D7460

scmutil: add assertions to help pytype Differential Revision: https://phab.mercurial-scm.org/D7456

revset: add an assertion to help pytype Differential Revision: https://phab.mercurial-scm.org/D7455

import: add a --secret option Similarly to "hg commit", we add a --secret option to "hg import" for committing with the secret phase. The option has no short form since there already is a "-s" for "--similarity". .. feature:: ``hg import`` has a new --secret option for committing with the secret phase.

hgweb: add diffs to the json changeset template While it's possible to get the diff for each file individually via an obscure url that I'm not even sure is documented, (e.g. diff/{rev}/{filename}?style=json ), it is more convenient to provide the full diff from the changeset view. This is already normally computed for other non-JSON templates, so it seems like an oversight to omit it for the JSON one. The output format is a bit unwieldy but it's consistent with the existing format output by the aforementioned obscure url.

hgweb: add files to the json changeset template As far as I can tell, it's currently impossible via JSON to figure out what files are touched by a particular revision. Reusing the filenodelink and filenolink templates from the other hgweb templates should provide this information.

hgweb: add a status property to file list context The web templates merely indicate if files touched by this revision are in the commit or not, i.e. if they are removed. It would be helpful to have more context and also indicate whether the files are added, modified, or removed.

match: remove explicitdir attribute No one sets it or reads it anymore. Differential Revision: https://phab.mercurial-scm.org/D7441

dirstate: stop caring about match.explicitdir No one passes it in anymore. Differential Revision: https://phab.mercurial-scm.org/D7440

dirstate: include explicit matches in match.traversedir calls The caller should not have to register to get both explicit and non-explicit matches. Differential Revision: https://phab.mercurial-scm.org/D7439

commit: drop unused "vdirs" argument from repo.checkcommitpatterns() Differential Revision: https://phab.mercurial-scm.org/D7438

commit: rewrite check for `hg ci <path>` being a directory The matcher API is complicated by match.bad, match.explicitdir, and match.traversedir. We already have very few users of match.explicitdir. By rewriting this check we get close to being able to remove match.explicitdir. This may make the check slower, but I think that will be very marginal. Disclosure: I actually wrote this patch to solve a bug we've seen with an internal extension. The internal extension overrides the dirstate walk to only walk the files that our FUSE tells us are modified. That led to "vdirs" not getting populated as this code expected. I have wanted to get rid of match.explicitdir for a very long time, though. Differential Revision: https://phab.mercurial-scm.org/D7437

lock: fix race in lock-breaking code With low frequency, I see hg pulls fail with output like: abort: no such file or directory: .hg/store/lock I think what happens is, in lock.py, in: def _testlock(self, locker): if not self._lockshouldbebroken(locker): return locker # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: with lock(self.vfs, self.f + b'.break', timeout=0): self.vfs.unlink(self.f) except error.LockError: return locker if a lock is breakable on disk, and two hg processes concurrently get to the "if locker dead" comment, a possible interleaving is: process1 finishes executing the function and then process2 finishes executing the function. If that happens, process2 will either get ENOENT in self.vfs.unlink (resulting in the spurious failure above), or break a valid lock and potentially cause repository corruption. The fix is simple enough: make sure the lock is breakable _inside_ the critical section, because only then can we know that no other process can invalidate our knowledge on the lock on disk. I don't think there are tests for this. I've tested this manually with: diff --git a/mercurial/lock.py b/mercurial/lock.py --- a/mercurial/lock.py +++ b/mercurial/lock.py @@ -351,6 +351,8 @@ class lock(object): if not self._lockshouldbebroken(locker): return locker + import random + time.sleep(1. + random.random()) # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: @@ -358,6 +360,7 @@ class lock(object): self.vfs.unlink(self.f) except error.LockError: return locker + time.sleep(1) def testlock(self): """return id of locker if lock is valid, else None. and I see this change of behavior before/after this commit: $ $hg init repo $ cd repo $ ln -s $HOSTNAME/effffffc:987654321 .hg/wlock $ touch a $ $hg commit -Am_ & $hg commit -Am _; wait -abort: No such file or directory: '/tmp/repo/.hg/wlock' adding a +warning: ignoring unknown working parent 679a8959a8ca! +nothing changed Differential Revision: https://phab.mercurial-scm.org/D7199

lock: refactor in preparation for next commit Differential Revision: https://phab.mercurial-scm.org/D7198

extensions: suppress a pytype failure due to a typeshed bug Bug filed upstream, suppress the failure here so we can move on. Differential Revision: https://phab.mercurial-scm.org/D7410

dispatch: add some assertions to give pytype a helping hand Differential Revision: https://phab.mercurial-scm.org/D7409

extensions: hide two confusing import statements from pytype Differential Revision: https://phab.mercurial-scm.org/D7408

debugcommands: add assertions to convince pytype peer is not None This function is moderately annoyingly defined, and peer is set up iff we're not in raw-proto mode. That's fine, but it confuses pytype. Adding these assertions is a low-overhead way to convince pytype we're doing reasonable things. Differential Revision: https://phab.mercurial-scm.org/D7407

debugcommands: suppress import errors for pytype Differential Revision: https://phab.mercurial-scm.org/D7385

state: add a pytype annotation Differential Revision: https://phab.mercurial-scm.org/D7383

logcmdutil: add a type annotation This wasn't actually required in the end, as there was a real bug found by pytype, but the annotation helped me figure that out. We can drop this patch if that's the preference. Differential Revision: https://phab.mercurial-scm.org/D7382

cmdutil: add a pytype annotation to help out some callsites I think we could constrain the values here more than Any, but this lets us move forward with typechecking commands.py. Differential Revision: https://phab.mercurial-scm.org/D7381

hghave: add a check for the `xz` compression utility This isn't install by default on Mac, which causes a test failure. The logic for avoiding the command is a little goofy, but nested `#if` isn't supported, and it still seems worth running the hg command to see if anything explodes. With this, the py3 tests run (almost) cleanly on 10.14.6: # Ran 835 tests, 58 skipped, 1 failed. Alas, the mac-packaging test is skipped because it's slow. The failure here is in test-releasenotes-merging.t, complaining about not being able to import the `fuzzywuzzy` module. I have it installed on py3 (thus the test isn't skipped), but not on py2. So there must be some unintended cross pollination here when running `hg` commands. Differential Revision: https://phab.mercurial-scm.org/D7374

py3: use '%d' for formatting a revnum in `hg co --date` code Differential Revision: https://phab.mercurial-scm.org/D7429

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7454

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7453

templates: make {indent("", " ")} be empty indent() is documented to indent all non-empty lines, but it made an exception for the first line, which always got indented. I also made indent() not indent the first line even if an indent override was given for the first line. I think that is what one would usually want. Differential Revision: https://phab.mercurial-scm.org/D7432

tests: document behavior of indent() with empty first line indent() is documented to indent all non-empty lines, but it still indents the first line even if it's empty. Let's have tests for that. Differential Revision: https://phab.mercurial-scm.org/D7431

tests: check for Windows line endings We shouldn't generally be using Windows line endings in files under version control. I've accidentally committed a few files with Windows line endings recently. So let's add a test for this. Differential Revision: https://phab.mercurial-scm.org/D7448

tests: verify that Python scripts are blackened Scripts are Python too. They should be blackened. (As suggested by Pierre-Yves in D7420.) Differential Revision: https://phab.mercurial-scm.org/D7447

black: blacken scripts These scripts weren't blackened. I found these as part of adding script checking to test-check-format.t. # skip-blame black Differential Revision: https://phab.mercurial-scm.org/D7446

doc: don't use mutable default arguments It appears our mutable default arguments checker doesn't find arguments not on the first line of a function definition :( The arguments aren't used, so the default value is irrelevant. I found this when blackening this script in a future commit. Differential Revision: https://phab.mercurial-scm.org/D7445

tags: don't check for clean files when checking for dirty working copy We didn't request status for clean files so there should never be any entries in the list of clean files (at least not since dd773340a085), but it's misleading to check the list. Differential Revision: https://phab.mercurial-scm.org/D7428

branchmap: make "closed" a set from beginning instead of converting from list Differential Revision: https://phab.mercurial-scm.org/D7427

debugextensions: gracefully handle missing __file__ attributes This was crashing PyOxidizer. Differential Revision: https://phab.mercurial-scm.org/D7452

debuginstall: gracefully handle missing __file__ attributes This was crashing PyOxidizer. While here, point "Python lib" and "installed modules" to the oxidized binary when read from memory instead of pretending their location is unknown. Differential Revision: https://phab.mercurial-scm.org/D7451

typing: add pseudo localstr.__init__() to help pytype Apparently, pytype failed to parse localstr.__new__()? This fixes the following errors: line 126, in __hash__: No attribute '_utf8' on localstr [attribute-error] line 188, in tolocal: Function localstr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, string: str, ...) Actually passed: (self, string: bytes, ...)

typing: fix argument type of encoding.localstr() It takes both utf-8 and local strings in bytes type.

typing: suppress error of py2 encoding.strtolocal() and .strfromlocal() I don't know how to conditionally get rid of these py2/py3 overloads from .pyi file. Instead, this patch makes pytype ignore the false-positives: line 271, in strtolocal: bad option in return type [bad-return-type] Expected: bytes Actually returned: str

typing: constrain argument/return types of encoding.toutf8b()

typing: fix argument type of encoding.tolocal() and .fromutf8b() Fixes various pytype errors.

typing: fix forward reference in _Tlocalstr type bound AFAIK, a quoted string in type position is translated to its ForwardRef. https://github.com/python/typing/issues/34 This fixes the following error: File "mercurial/encoding.py", line 38, in <module>: Name 'localstr' is not defined [name-error]

encoding: make getcols() raise exception explicitly This helps pytype. File "mercurial/encoding.py", line 329, in getcols: bad option in return type [bad-return-type] Expected: bytes Actually returned: None

setup: conditionalize access to `sys.dllhandle` when building extensions This code is only run on Windows, and was crashing PyOxidizer when running in `setup-py-install` mode. Now an oxidized binary can be built by simply pointing to setup.py. Something is slightly different now that it's not being built from a virtualenv. Previously, `hg version` could print to the screen, but now it aborts saying "Incorrect function". But I can see the output if redirected to a file, and it's not complaining about missing C extensions, so I think those are loading now (unlike from the virtualenv). The interesting this about this incorrect function output is that it failed when initially built. I then went back and did a `make clean` and `make local` with py3 and then py2 to ensure I didn't break the existing code. At that point I ran the oxidized executable again and it was able to print to the screen normally! So I ran `pyoxidizer build` again, it only output the following, and then running the executable failed to output again: (pyO2_venv) C:\Users\Matt\hg3\hg_pyO2>pyoxidizer build Finished dev [unoptimized + debuginfo] target(s) in 0.12s packaging application into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug purging C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug copying C:/Users/Matt/hg3/hg_pyO2\build\target\x86_64-pc-windows-msvc\debug\hg_pyO2.exe to C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe resolving packaging state... writing license for [...] hg_pyO2 packaged into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug executable path: C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe Differential Revision: https://phab.mercurial-scm.org/D7444

setup: use bytes for assumed python version Caught by PyOxidizer when hacking on Windows. # skip-blame for b'' prefixing Differential Revision: https://phab.mercurial-scm.org/D7443

setup: combine two contiguous string literals Differential Revision: https://phab.mercurial-scm.org/D7442

util: remove datapath and swith users over to resourceutil The util module no longer needs the datapath for injecting it into the i18n module, so we can remove it from there and update other users. Differential Revision: https://phab.mercurial-scm.org/D7436

i18n: get datapath directly from resourceutil The new resourceutil module is lower in the dependency graph, so we can depend directly on it and won't have to depend on the util module to inject the datapath. Differential Revision: https://phab.mercurial-scm.org/D7435

util: move definition of datapath to resourceutil Since this means moving the function into a subdirectory, we have to compensate by adding another layer of os.path.dirname(). Differential Revision: https://phab.mercurial-scm.org/D7434

procutil: move mainfrozen() to new resourceutil.py The i18n module practically depends on procutil for mainfrozen() but since procutil depends on i18n, it would be a circular dependency if i18n depended directly on procutil. The cycle is currently resolved by having the higher-level util module calculate the "datapath" and inject it into i18n. Extracting mainfrozen() to a new module lets us clean up the dependencies. Differential Revision: https://phab.mercurial-scm.org/D7433

defaults: rename default.d to defaultrc to make it a valid package name "default.d" is not a valid Python package name, and it needs to be a valid package name if we want to be able to load the files from here as resources. Differential Revision: https://phab.mercurial-scm.org/D7426

help: remove references to default.d/ from user-facing docs The user doesn't need to know what we call the internal directory and I'm about to rename it. Differential Revision: https://phab.mercurial-scm.org/D7425

bisect: replace try:/finally: by a "restore_state" context manager This should help pytype to not consider "bgood" variable as NameError. See https://phab.mercurial-scm.org/D7384 for context. Differential Revision: https://phab.mercurial-scm.org/D7430

histedit: render a rolled up description using the proper roll colours Users have rightfully complained that the old behaviour of completely removing the description of a rolled commit makes it difficult to remember what was in that commit. Instead, we now render the removed description in red. I couldn't think of a simpler way to do this. You can't just combine existing curses colours into new effects; only secondary effects like bold or underline can be logically OR'ed to generate a combined text effect. It seems easier to just redundantly keep track of what the roll colour should be.

histedit: define new colour pairs for roll action Red seems like a good colour to indicate removal, since that's the colour we use in diffs. We need three varieties with three different backgrounds corresponding to the possible ways we can display a rule.

histeditrule: split __bytes__ property into prefix and desc In order to be able to colourise the description of the rule, we need to have it as a separate bytestring. Curses doesn't make it easy to take existing text on the screen and give it different properties; we can only add new text with new properties.

merge with stable

packaging: convert to UNIX line endings I must have my editor on Windows configured incorrectly because I submitted patches with Windows line endings :( # skip-blame whitespace only line ending changes Differential Revision: https://phab.mercurial-scm.org/D7421

black: blacken scripts test-check-format.t doesn't appear to detect Python scripts with shebangs. But my editor which is configured to auto run black on Python files does and it appears some files are not blackened. This commit blackens scripts that need it. # skip-blame reformatting Differential Revision: https://phab.mercurial-scm.org/D7420

dirs: resolve fuzzer OOM situation by disallowing deep directory hierarchies It seems like 2048 directories ought to be enough for any reasonable use of Mercurial? A previous version of this patch scanned for slashes before any allocations occurred. That approach is slower than this in the happy path, but much faster than this in the case that too many slashes are encountered. We may want to revisit it in the future using memchr() so it'll be well-optimized by the libc we're using. .. bc: Mercurial will now defend against OOMs by refusing to operate on paths with 2048 or more components. This means that _extremely_ deep path hierarchies will be rejected, but we anticipate nobody is using hierarchies this deep. Differential Revision: https://phab.mercurial-scm.org/D7411

py3: use native string for 'macosx_app' I couldn't find any definitive information on this, but all examples (except for ours) seem to use a native string. Differential Revision: https://phab.mercurial-scm.org/D7414

py3: drop an unnecessary fsencode() before comparing with constant Differential Revision: https://phab.mercurial-scm.org/D7413

py3: use native string as fallback value for __file__ for consistency This is not a bugfix (pycommpat.fsencode(b'') is a no-op on py3), but the b'' value was inconsistent and confusing. Differential Revision: https://phab.mercurial-scm.org/D7412

scmutil: convert status data object from a tuple to an attrs (API) We've been pushing towards the property names for a while, and the subclassing of the tuple confuses pytype. Rather than bend over backwards to try and annotate the tuple subclass, let's just use attrs here. Differential Revision: https://phab.mercurial-scm.org/D7406

perf: bool() elements of dirstate.status return instead of len() I'm about to make scmutil.status no longer have a len(), so we need to do something else to "use" the results in this perf method. Differential Revision: https://phab.mercurial-scm.org/D7405

tags: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7404

subrepo: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7403

revset: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7402

patch: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7401

logcmdutil: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7400

context: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7399

commands: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7398

cmdutil: convert terse status result back to an scmutil.status This is also part of detupling scmutil.status. Differential Revision: https://phab.mercurial-scm.org/D7397

cmdutil: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7396

uncommit: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7395

transplant: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7394

split: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7393

mq: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7392

hgk: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7391

extdiff: use field names instead of field numbers on scmutil.status As part of my pytype adventures I want to make scmutil.status no longer a subclass of tuple. This is part of that process. Differential Revision: https://phab.mercurial-scm.org/D7390

encoding: fix bad type annotation This function returns utf-8 in a bytes, not a unicode. Differential Revision: https://phab.mercurial-scm.org/D7379

branchmap: annotate constructor type for branchcache This type signature is...big. But it's correct as far as I can tell, and it detected a bug. Differential Revision: https://phab.mercurial-scm.org/D7389

debugcommands: don't shadow the error module Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7378

branchmap: correctly set()-ify list argument Caught with pytype. I'm more than a little curious how this never caused problems. Differential Revision: https://phab.mercurial-scm.org/D7289

utils: move finddirs() to pathutil This is a follow-up to c21aca51b392 (utils: move the `dirs` definition in pathutil (API), 2019-11-06). finddirs() is closely related to dirs and used by it. Differential Revision: https://phab.mercurial-scm.org/D7388

help: create packages for the help text These files need to be loaded as resources with PyOxidizer, instead of using filesystem representations. AFAICT, the resource loading mechanisms only work for the named package given to it, and can't reach into a subdirectory. While here, the `help` directory is renamed to `helptext`. Without this, trying to load external help text crashed in mercurial/help.py when importing `.i18n`, saying there's no `mercurial.help.i18n` module. Differential Revision: https://phab.mercurial-scm.org/D7376

tests: test "hg log" with --line-range and --copies This now works (does not crash), due to previous changeset. Since --line-range implies --follow, --copies option is redundant.

logcmdutil: let getlinerangerevs() return "revs" as a smartset This makes it consistent in "hg log" command where 'revs' can come from logcmdutil.getrevs() as a smartset or from getlinerangerevs(), previously as a list. This will help type hinting as noticed in https://phab.mercurial-scm.org/D7377.

patchbomb: fix wrong argument type when calling mail generator.flatten()

mail: move strtolocal call in _addressencode()

mail: use a native string for "subtype" value This is somehow similar to previous changeset and avoids one str conversion.

mail: let all charset values be native strings Charset values will typically be used to build email.header.Header instances, which takes str (though it tolerates bytes) or passed to decode()/encode() methods of string values (which want str). It seems that using native str involves less conversions than before and this also helps type hinting (as illustrates removal of pytype disabling instructions).

mail: add type hints for pytype We essentially annotate functions in which handling of bytes/str is not obvious in order to hopefully clear things out. See also changeset 2ade00f3b03b introducing typing hints in Mercurial. Most types are straightforward but a few is wrong, and we need to either disable pytype on respective instructions or use wrong annotations. These will be fixed in next changesets. Notice the type Union[bytes, str] of "s" parameter of headencode(), this reflects how email.header.Header.append() behaves.

mail: fix a bad return type in _encode() This particular instruction returned only a string and omitted the charset value.

packaging: stage files and dynamically generate WiX installer Like we did for Inno, we want to make the WiX installer "dumb" and simply consume source files from a directory tree rather than have to define every single file in installer files. This will greatly decrease the amount of effort required to maintain the WiX installer since we don't have to think that much about keeping files in sync. This commit changes the WiX packager to populate a staging directory as part of packaging. After it does so, it scans that directory and dynamically generates WiX XML defining the content within. The IDs and GUIDs being generated are deterministic. So, upgrades should work as expected in Windows Installer land. (WiX has a "heat" tool that can generate XML by walking the filesystem but it doesn't have this deterministic property, sadly.) As part of this change, GUIDs are now effectively reset. So the next upgrade should be a complete wipe and replace. This could potentially cause issues. But in my local testing, I was able to upgrade an existing 5.1.2 install without issue. Compared to the previous commit, the installed files differ in the following: * A ReleaseNotes.txt file is now included * A hgrc.d/editor.rc file is now generated (mercurial.rc has been updated to reflect this logical change to the content source) * All files are marked as read-only. Previously, only a subset of files were. This should help prevent unwanted tampering. Although we may want to consider use cases like modifying template files... This change also means that Inno and WiX are now using very similar code for managing the install layout. This means that on disk both packages are nearly identical. The differences in install layout are as follows: * Inno has a Copying.txt vs a COPYING.rtf for WiX. (The WiX installer wants to use RTF.) * Inno has a Mercurial.url file that is an internet shortcut to www.mercurial-scm.org. (This could potentially be removed.) * Inno includes msvc[mpr]90.dll files and WiX does not. (WiX installs the MSVC runtime via merge modules.) * Inno includes unins000.{dat,exe} files. (WiX's state is managed by Windows Installer, which places things elsewhere.) Because file lists are dynamically generated now, the test ensuring things remain in sync has been deleted. Good riddance. While this is a huge step towards unifying the Windows installers, there's still some improvements that can be made. But I think it is worth celebrating the milestone of getting both Inno and WiX to essentially share core packaging code and workflows. That should make it much easier to change the installers going forward. This will aid support of Python 3. Differential Revision: https://phab.mercurial-scm.org/D7173

tests: add optional output when wheel is missing If the `wheel` package isn't installed, pip will emit a warning on stderr complaining about that. Alternatively, we could redirect stderr to stdout and swallow the warning. But I like tests having visibility of warnings, as it helps keep us honest regarding edge cases. Differential Revision: https://phab.mercurial-scm.org/D7371

py3: replace "%r" by"'%s'% for py3-compatible (and clearer) quoting in chg Differential Revision: https://phab.mercurial-scm.org/D7375

dirstate: respect request to not list unknown/ignored/clean files (API) Unknown files that are explicitly mentioned by the matcher are returned even if the caller said unknown=False (and it seems the same is done for ignored files). That seems pretty surprising. Let's make the interface less surprising by respecting the caller's request. Differential Revision: https://phab.mercurial-scm.org/D7150

py3: pass a bytes value for "msg" to nouideprecwarn() That function formats "msg" with the "version" value. On Python 3, this leads to "TypeError: can only concatenate str (not "bytes") to str". Also eliminate spurious strings concatenation in single-line declarations. Differential Revision: https://phab.mercurial-scm.org/D7373

remotefilelog: handle **kwargs correctly when overriding changelog.add() Differential Revision: https://phab.mercurial-scm.org/D7372

packaging: ship all help .txt files on WiX These are technically not needed. But it is easier to ship all files than to cherry-pick. A `make install` will copy these files, so the new behavior is consistent with that. This also makes WiX consistent with Inno, which is my main reason for doing this. If we don't want to ship the files (which is a valid argument), I think we can do that in a follow up. Differential Revision: https://phab.mercurial-scm.org/D7166

packaging: install contrib/mq.el on WiX This makes things consistent with the Inno installer. Differential Revision: https://phab.mercurial-scm.org/D7167

fuzz: clean out most of fuzzutil It's now a header-only setup that just selects absl:: or std:: versions of things as needed, and a logging helper. There's some room for future cleanups here: we could move to just requiring a C++17 compiler and get rid of the absl stuff. Also, the mpatch parser has a fair amount of parsing the input string into char* blocks that we can and probably should fix that up to use FuzzedDataProvider as well. Differential Revision: https://phab.mercurial-scm.org/D7370

fuzz: migrate xdiff fuzzer to use FuzzedDataProvider Differential Revision: https://phab.mercurial-scm.org/D7369

fuzz: migrate bdiff fuzzer to use FuzzedDataProvider Differential Revision: https://phab.mercurial-scm.org/D7368

dirstate: re-blacken file This got overlooked in landing a recent patch, but it's easy to fix up. Differential Revision: https://phab.mercurial-scm.org/D7367

rust-matchers: remove default implementations for `Matcher` trait We don't expect a whole lot of matchers to be defined, and this makes it more obvious what a matcher does by reading its `impl Matcher for FooMatcher`. This patch has the added benefit of fixing the `AlwaysMatcher`, its `matches` function differs from the former default. Differential Revision: https://phab.mercurial-scm.org/D7255

packaging: use lower case for RC files The file case doesn't matter. I'm not sure why it is capitalized. This is a source of inconsistency between Inno and WiX. So let's standardize on the lower case names. The MSI should remove the old files before installing the new one, so there shouldn't be a problem with a case collision. And, users should not have modified these files, so it should be safe to rename them. Differential Revision: https://phab.mercurial-scm.org/D7172

packaging: write out editor.rc in Python Let's dynamically generate the file from Python instead of relying on Inno functionality, as that will give more visibility into what's going on. I spotted this as part of converging the install layout of Inno and WiX: the former doesn't install an editor.rc and instead defines the editor via its mercurial.rc file. We will change the behavior of WiX later once it is consuming the staged install directory for packaging. Differential Revision: https://phab.mercurial-scm.org/D7171

packaging: stop installing Contributors.txt This file isn't up to date. If we are going to call out contributors, we should auto generate a file from `hg log` or something. I don't think we need to include it in the Windows install. This change makes Inno more consistent with WiX, which doesn't ship this file. Differential Revision: https://phab.mercurial-scm.org/D7170

packaging: remove hg-ssh.8.html from Inno installer We don't ship hg-ssh because it requires a python.exe to run, which we don't ship. So it doesn't make sense to ship the HTML documentation for this tool. This change makes the Inno install layout more consistent with WiX, which doesn't ship this file. Functionality for removing files has been made generic, in anticipation of future expansion. Differential Revision: https://phab.mercurial-scm.org/D7169

packaging: install .rc files to hgrc.d Currently, Inno installs these .rc files to a default.d directory and WiX to a hgrc.d directory. Mercurial automatically loads files in an hgrc.d directory next to the executable and in an "internal" default.d directory. This "internal" directory is util.datadir, which on Windows installs made with py2exe is the directory containing hg.exe. So on Windows both hgrc.d and default.d directories relative to hg.exe are loaded. For the install layout, I think it makes sense for hgrc.d to be used, as that path is documented as non-internal. So this commit changes the Inno install layout to use hgrc.d instead of default.d. .. bc:: Changed default config file location in Windows .exe installers Mercurial's installer installs some default .rc config files. Previously, these were installed in a default.d directory. These are now installed in an hgrc.d directory. Custom default.d/*.rc files should still be loaded. But it is recommended to move these to hgrc.d/*.rc. This change only affects the .exe installers (not the .msi installers). Differential Revision: https://phab.mercurial-scm.org/D7168

rust: introduce SIZE_FROM_OTHER_PARENT constant This will clarify what `-2` meant in the codebase. This change also merged imports in affected files for cleanup. Differential Revision: https://phab.mercurial-scm.org/D7301

rust-status: refactor dispatch case for normal files This should make the code easier to read and more idiomatic. Differential Revision: https://phab.mercurial-scm.org/D7300

rust-status: return a ParallelIterator instead of a Vec from stat_dmap_entries This allows the caller function to choose when and how the iteration and/or collection happens. This change also cleans up the now unused `filter_map`. Differential Revision: https://phab.mercurial-scm.org/D7299

rust-status: improve status performance This change does more things in the parallel loop, refactors the file-level logic into two functions for added clarity. This bit of Rust code takes 55ms to execute on a repo where the stat'ing part of Valentin's fast path takes 40ms. While the code differs a bit and it's hard to get an exact measurement of how much of a performance impact it has, I can be fairly certain that this implementation is *at worse* twice as slow. Differential Revision: https://phab.mercurial-scm.org/D7254

bookmarks: accept explicit -r 'wdir()' when adding new bookmarks (issue6218) Even though the bookmark semantics can't be fully encoded to the virtual working changeset idea, the active bookmark can be considered a bookmark of the working revision. Before, 'tgt' was None, and changes=[(bm, None)] means deleting a bookmark named 'bm'.

bookmarks: use changectx instead of remembering hex of hidden revision It should be better to not depend on the ctx variable which was assigned conditionally.

bookmarks: resolve target revision out of the bookmarks loop The target revision doesn't depend on the bookmark to be added.

bookmarks: fix handling of multiple bookmarks with one to be deactivated Before, "hg bookmark --inactive Z Y" would ignore "Y" if "Z" were currently active. I'm pretty sure it is a bug.

import-checker: allow 'from typing import ...' Suppresses the following error in test-check-module-imports.t: mercurial/encoding.py:24: relative import of stdlib module

match: drop support for passing '.' for root dir to visit*() methods We said we'd drop support for it after 5.1, so it's time to clean it up now. Differential Revision: https://phab.mercurial-scm.org/D7249

tests: use time.time() for relative start and stop times os.times() does not work on Windows. This was resulting in the test start, stop, and duration times being reported as 0. This commit swaps in time.time() for wall clock measurements. This isn't ideal, as time.time() is not monotonic. But Python 2.7 does not have a monotonic timer that works on Windows. So it is the best we have which is trivially usable. And test times aren't terribly important, so variances due to clock skew are arguably acceptable. Differential Revision: https://phab.mercurial-scm.org/D7126

tests: rename stopped and started variables to reflect times In preparation for introducing more variables that will have similar names. Differential Revision: https://phab.mercurial-scm.org/D7125

revset: simplify checkstatus() by using any() Differential Revision: https://phab.mercurial-scm.org/D7179

rebase: check for unfinished ops even when inmemory (issue6214) When using rebase.experimental.inmemory, we should be able to work well with a dirty working directory, but we can not reliably work if we're in the middle of another operation (such as another rebase), as we'll potentially stomp on some state that the other operation needs. Differential Revision: https://phab.mercurial-scm.org/D7298

status: move initialization closer together The default initialization happened long before it needed to happen. Differential Revision: https://phab.mercurial-scm.org/D7145

largefiles: avoid dynamically subclassing context instances E.g. copies.pathcopies() compares context objects for equality and basectx.__eq__ compares the type. But since largefiles was dynamically creating classes, they would all be unequal. That led pathcopies(), after some changes I made, to not short-circuit to get copy info only from the dirstate. This patch fixes that short-circuiting (and other places where context are being compared for equality). Differential Revision: https://phab.mercurial-scm.org/D7143

largefiles: delete unused repo.status_nolfiles() Differential Revision: https://phab.mercurial-scm.org/D7142

largefiles: use context manager for wlock in repo.status() override Differential Revision: https://phab.mercurial-scm.org/D7141

largefiles: allow "lfstatus" context manager to set value to False Differential Revision: https://phab.mercurial-scm.org/D7140

largefiles: move lfstatus context manager to lfutil Differential Revision: https://phab.mercurial-scm.org/D7139

revlog: delete references to deleted nullid sentinel value We stopped keeping the nullid in the index in 781b2720d2ac (index: don't include nullid in len(), 2018-07-20). I forgot to update some comments about it. Differential Revision: https://phab.mercurial-scm.org/D7366

revlog: remove the, now unused, `revlog._nodepos` attribute Spotted by Yuya Nishihara.

nodemap: use bytes for the error message Spotted by Yuya Nishihara.

py3: use native strings when forming email headers in patchbomb Per previous changesets, encoded header's values are native str. We complete the change in patchbomb extension to have literal header values native str as well. Then we can also change headers' keys to be str. In _msgid(), we still need to use encoding.strfromlocal() because usage of os.environ is not allowed by check-code. This finally removes the "if pycompat.ispy3:" TODO.

mail: convert addr to str early in addrlistencode()

mail: let addressencode() / addrlistencode() return native strings Avoids conversion to "str" on py3.

mail: let headencode() return a native string This is to avoid conversion to/from str on py3.

revlog: deprecate direct `nodemap` access Now that all in-core user have been updated to the new API, we can deprecated direct access to nodemap. Differential Revision: https://phab.mercurial-scm.org/D7362

revlog: drop special hack in the `nodemap` property This code does not seems necessary anymore. Differential Revision: https://phab.mercurial-scm.org/D7361

perf: make `perfrevlogindex` use the new `index.rev` api if available Differential Revision: https://phab.mercurial-scm.org/D7360

perf: make `perfnodemap` use the new `index.get_rev` api if available Differential Revision: https://phab.mercurial-scm.org/D7358

index: use `index.get_rev` in `remotefilelog` Differential Revision: https://phab.mercurial-scm.org/D7357

index: use `index.get_rev` in `histedit.processreplacement` Differential Revision: https://phab.mercurial-scm.org/D7354

index: use `index.get_rev` in `histedit.adjustreplacementsfrommarkers` Differential Revision: https://phab.mercurial-scm.org/D7353

index: use `index.get_rev` in `rebase._computeobsoletenotrebased` Differential Revision: https://phab.mercurial-scm.org/D7350

index: use `index.get_rev` in `rebase.successorrevs` The code is slightly updated in the process for simplicity. Differential Revision: https://phab.mercurial-scm.org/D7349

index: use `index.get_rev` in `unionrepo.unionrevlog` Differential Revision: https://phab.mercurial-scm.org/D7338

index: use `index.rev` in `bundlerepo.bundlerevlog` Differential Revision: https://phab.mercurial-scm.org/D7340

index: use `index.get_rev` in `exchange._pullapplyphases` Differential Revision: https://phab.mercurial-scm.org/D7347

index: use `index.rev` in `exchange.fallbackheads` Differential Revision: https://phab.mercurial-scm.org/D7345

index: use `index.get_rev` in `revset._mapbynodefunc` We slightly update the code in the process. Differential Revision: https://phab.mercurial-scm.org/D7343

index: use `index.get_rev` in `discovery._postprocessobsolete` Differential Revision: https://phab.mercurial-scm.org/D7341

index: use `index.get_rev` in `repoview.pinnedrevs` We simplify the code in the process Differential Revision: https://phab.mercurial-scm.org/D7337

index: use `index.get_rev` in `obsolete._computephasedivergentset` Differential Revision: https://phab.mercurial-scm.org/D7336

index: use `index.get_rev` in `obsutil.getobsoleted` Differential Revision: https://phab.mercurial-scm.org/D7335

index: use `index.get_rev` in `phases.newheads` Differential Revision: https://phab.mercurial-scm.org/D7332

index: use `index.get_rev` in `localrepo.known` Differential Revision: https://phab.mercurial-scm.org/D7327

index: add a `get_rev` method (API) The new `index.getrev(node)` is to be preferred over using: `index.nodemap.get(node)`. This get us closer to be able to remove the `nodemap` attribute of the index. Differential Revision: https://phab.mercurial-scm.org/D7326

index: use `index.rev` in `revlog.rev` Differential Revision: https://phab.mercurial-scm.org/D7325

index: add a `rev` method (API) The new `index.rev(node)` is to be preferred over using `node index.nodemap[node]`. This get us closer to be able to remove the `nodemap` attribute of the index. Differential Revision: https://phab.mercurial-scm.org/D7324

perf: make `perfphasesremote` use the new `index.has_node` api (If available) Differential Revision: https://phab.mercurial-scm.org/D7359

index: use `index.has_node` in `infinitypush` Differential Revision: https://phab.mercurial-scm.org/D7356

index: use `index.has_node` in `phabricator.getoldnodedrevmap` Differential Revision: https://phab.mercurial-scm.org/D7355

index: use `index.has_node` in `histedit.cleanupnode` Differential Revision: https://phab.mercurial-scm.org/D7352

index: use `index.has_node` in `histedit._finishhistedit Differential Revision: https://phab.mercurial-scm.org/D7351

index: use `index.has_node` in `exchangev2._pullchangesetdiscovery` Differential Revision: https://phab.mercurial-scm.org/D7348

index: use `index.has_node` in `exchange._pulldiscoverychangegroup` Differential Revision: https://phab.mercurial-scm.org/D7346

index: use `index.has_node` in `bookmarks.bmstore` Differential Revision: https://phab.mercurial-scm.org/D7344

index: use `index.has_node` in `tags.findglobaltags` Differential Revision: https://phab.mercurial-scm.org/D7342

index: use `index.has_node` in `bundlerepo.bundlerevlog` Differential Revision: https://phab.mercurial-scm.org/D7339

index: use `index.has_node` in `obsutil.foreground` Differential Revision: https://phab.mercurial-scm.org/D7334

index: use `index.has_node` in `obsutil.exclusivemarkers` Differential Revision: https://phab.mercurial-scm.org/D7333

index: use `index.has_node` in `phases.filterunknown` Differential Revision: https://phab.mercurial-scm.org/D7331

index: use `index.has_node` in `analyzeremotephases` Differential Revision: https://phab.mercurial-scm.org/D7330

index: use `index.has_node` in `repo._rollback` Differential Revision: https://phab.mercurial-scm.org/D7329

index: use `index.has_node` in `revlog.addgroup` Differential Revision: https://phab.mercurial-scm.org/D7328

index: use `index.has_node` in `revlog.addrevision` Differential Revision: https://phab.mercurial-scm.org/D7323

index: add a `has_node` method (API) The new `index.has_node(node)` is to be preferred over: `node in index.nodemap`. This get us closer to be able to remove the `nodemap` attribute of the index. Differential Revision: https://phab.mercurial-scm.org/D7322

revlog: deal with nodemap deletion within the index Since the nodemap data now live in the index, it should be the index responsibility to ensure the data are up to date. The C version of the index is already dealing with such deletion. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7321

revlog: clean up the node of all revision stripped in the C code For some obscure reason, the loop cleaning up node was skipping the first element… I cannot see a reason for it. The overall code is running fine nevertheless because the node are also explicitly deleted from python. We want to delete this explicit deletion, so we need to fix that code first. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7320

revlog: move nodemap update within the index code Since the nodemap data now live in the index, it should be the index responsibility to ensure the data are up to date. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7319

revlog: deprecate the _nodecache attribute (API) This attribute have been the same object for a long time. Code should directly access the nodemap in case of need. (Or the final API when this will be done). This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7318

revlog: access the nodemap through the index The `revlog.rev` method now access the nodemap through `self.index.nodemap`, instead of using the `_nodecache` attribute that we are trying to remove. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7317

revlog: return the nodemap as the nodecache Ultimately, the nodecache is the nodemap. To stop having this disctinction, we start using the nodemap where the nodecache was used. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7316

revlog: clarify which version use the older API in perf This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7315

revlog: no longer return the nodemap after parsing Now that both pure and cext version have a `nodemap` attribute, we do not need to return the `nodemap` object after parsing. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7314

revlog: move the nodemap into the index object (for pure) This make the pure code closer to the C extension one. The ultimate goal is to merge the two into a single object and offer a unified API. This changeset focus on gathering the data on the same object. For now the code for `revlogoldindex` and `BaseIndexObject` index object are quite similar. However, there will be larger divergence later on, so I don't think is worth doing a base case. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7313

revlog: introduce an explicit NodeMap class for pure code This class make the "pure" nodemap raise the same exception than the C-extension one. This is a step toward unifying nodemap and index, the class is not meant to survive on the long run. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. There is a new `isinstance` call, it will be cleaned up in coming changesets. Differential Revision: https://phab.mercurial-scm.org/D7312

utils: move the `dirs` definition in pathutil (API) Before this change, the `dirs` class was accessible through the `mercurial.util` module. That module is expected to stay free of scm specific content. The `pathutil` destination has been selection by Martin von Zweigbergk. This work is part of a refactoring to unify the revlog index and the nodemap. This unification prepare the use of a persistent nodemap. Differential Revision: https://phab.mercurial-scm.org/D7311

packaging: stop installing i18n files The WiX installer has been shipping the content of the i18n/ directory since it was introduced in 2010 in 1e022c88a0a5. And the installer was subsequently refactored to only ship the .po files and hggettext. The .po files and the hggettext script are only used at build time to produce .mo files, which Mercurial does use at run-time. It doesn't make sense to install these files on Windows. So this commit stops doing that. This change further converges the file layout of the Inno and WiX installers. Differential Revision: https://phab.mercurial-scm.org/D7165

packaging: install hgk as hgk.tcl Giving it a file extension will give Windows a better chance at being able to execute it via standard means. This also makes WiX consistent with the Inno installer. Differential Revision: https://phab.mercurial-scm.org/D7164

packaging: add logo-droplets.svg The WiX installer adds this file. Let's add it to the Inno installer for consistency. Differential Revision: https://phab.mercurial-scm.org/D7163

packaging: install documentation to doc/ directory The source directory is "doc" and "doc" is used by the WiX installer. I think it makes sense to converge on the canonical path name. While we're here, we also lower the case because we can. (I think lower case names make more sense plus it is more consistent.) This only affects the Inno installer at the moment because it is the only installer using the staging code. Differential Revision: https://phab.mercurial-scm.org/D7162

packaging: always pass VERSION into Inno invocation The code in the Inno file was a holdover from before we had Python driving execution. With Python in the driver's seat, we can now have it resolve the version string and pass it into Inno, making the code easier to understand for people who aren't packaging gurus. Differential Revision: https://phab.mercurial-scm.org/D7161

packaging: remove hg.exe.local file <exe>.local files are used by Windows to set up DLL redirection. But these files are ignored if you embed an application manifest in your binary, which we do. So the existence of this file serves no purpose. So we remove it. Differential Revision: https://phab.mercurial-scm.org/D7160

packaging: stage installed files for Inno Previously, the Inno installer maintained its own mapping of source files to install location. (We have to maintain a similar mapping in the WiX installer.) Managing the explicit file layout for Windows packages is cumbersome and redundant. Every time you want to change the layout you need to change N locations. We frequently forget to do this and we only find out when people install Mercurial from our packages at release time. This commit starts the process of consolidating and simplifying the logic for managing the install layout on Windows. We introduce a list of install layout rules. These are simply source filenames (which can contain wildcards) and destination paths. The Inno packaging code has been updated to assemble all files into a staging directory that mirrors the final install layout. The list of files to add to the installer is derived by walking this staging directory and dynamically emitting the proper entries for the Inno Setup script. I diffed the file layout before and after this commit and there is no difference. Another benefit of this change is that it facilitates easier testing of the Windows install layout. Before, in order to test the final install layout, you needed to build an installer and run it. Now, you can stage files into the final layout and test from there, without running the installer. This should cut down on overhead when changing Windows code. Differential Revision: https://phab.mercurial-scm.org/D7159

packaging: process Inno Setup files with Jinja2 I want to make the Inno Setup files dynamically generated. This will enable us to do things like automatically derive the set of files to be packaged instead of having to manually keep lists of files in sync across installers. As the first step towards this, we process the Inno Setup files with Jinja2. As part of this conversion, we had to escape syntax in mercurial.iss that conflicts with Jinja2. I also took the opportunity to include modpath.iss via Jinja2 instead of using Inno's preprocessor. This keeps the Python code a bit simpler. Differential Revision: https://phab.mercurial-scm.org/D7158

packaging: install and run Inno files in a build directory Upcoming commits will refactor the Windows installers so their content is dynamically derived. In preparation for this, we copy the Inno files into a new build directory and run them from there. This required changing some relative paths to work from a new directory level but is otherwise a pretty straightforward change. Differential Revision: https://phab.mercurial-scm.org/D7157

packaging: consolidate CLI functionality into packaging.py Consolidating functionality for invoking code in the hgpackaging package through a single CLI entry point will make things simpler when we add more complexity to that package. For example, it will allow us to run things out of a virtualenv with third party packages. This commit consolidates functionality from the Inno and WiX build.py scripts into a new packaging.py script. That script simply creates a virtualenv and runs the CLI functionality in it. The new virtualenv is populated with jinja2 because I felt it easier to incorporate requirements file processing in this commit and we will soon use jinja2 in an upcoming commit. The unified CLI functionality will also make it easier to script other packaging workflows going forward. e.g. RPM, Debian, and macOS packaging. Differential Revision: https://phab.mercurial-scm.org/D7156

packaging: clean up excess packages for Inno This partially reverts e97ffa5511ba. A bunch of package dependencies were getting picked up because I ran pip-compile on Linux and the keyring package has some Linux-only dependencies. I have run pip-compile again - this time from Windows - to undo the addition of these dependencies. I also normalized whitespace to LF. Differential Revision: https://phab.mercurial-scm.org/D7155

unshelve: correct help string; unshelve does not accept file list The non-option arguments to `hg unshelve` are interpreted as the name of the shelve to unshelve, not the list of files to unshelve. While that functionality would probably be nice to have, that's well beyond the scope of this documentation fix. Differential Revision: https://phab.mercurial-scm.org/D7365

repoview: use class literal for creating filteredchangelog The type name is constant, so we don't need to create it dynamically using type(). As suggested by Yuya. Differential Revision: https://phab.mercurial-scm.org/D7364

vagrant: update vagrant image to buster from jessie I honestly have no idea if this works since I don't use vagrant, but looking around online suggests it should work fine. Differential Revision: https://phab.mercurial-scm.org/D7309

packaging: remove references to debian jessie Jessie is on life support at this point, so let's update the list of releases in the Makefiles. Differential Revision: https://phab.mercurial-scm.org/D7308

sqlitestore: remove superfluous r-prefixes on strings I did this file separately because it's a _ton_ of hits and it had some manual reformatting fallout as a result of black squeezing some expressions onto a single line. Differential Revision: https://phab.mercurial-scm.org/D7307

cleanup: remove pointless r-prefixes on single-quoted strings This is the promised second step on single-quoted strings. These had existed because our source transformer didn't turn r'' into b'', so we had tagged some strings as r-strings to get "native" strings on both Pythons. Now that the transformer is gone, we can dispense with this nonsense. Methodology: I ran hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$' | xargs egrep --color=never -n -- \[\^b\]\[\^a-z\]r\'\[\^\'\\\\\]\*\'\[\^\'\ in an emacs grep-mode buffer, and then used a keyboard macro to iterate over the results and remove the r prefix as needed. # skip-blame removing unneeded r prefixes left over from Python 3 migration. Differential Revision: https://phab.mercurial-scm.org/D7306

rust-threads: force Rayon to respect the worker count in config As per the inline comment, this is a workaround because Rust code does not yet know how to read config files. Differential Revision: https://phab.mercurial-scm.org/D7310

phabricator: use context manager form of progress in uploadchunks Follow-up to 453079605242 / D7046. Differential Revision: https://phab.mercurial-scm.org/D7134

cleanup: remove pointless r-prefixes on double-quoted strings This is only double-quoted strings. I'll do single-quoted strings as a second step. These had existed because our source transformer didn't turn r"" into b"", so we had tagged some strings as r-strings to get "native" strings on both Pythons. Now that the transformer is gone, we can dispense with this nonsense. Methodology: I ran hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$' | xargs egrep --color=never -n -- \[\^a-z\]r\"\[\^\"\\\\\]\*\"\[\^\"\] in an emacs grep-mode buffer, and then used a keyboard macro to iterate over the results and remove the r prefix as needed. # skip-blame removing unneeded r prefixes left over from Python 3 migration. Differential Revision: https://phab.mercurial-scm.org/D7305

formatting: remove another data-ogre from the config example The fix extension will pass the contents on stdin and read the output from stdout. By invoking `rustfmt` as `rustfmt {rootpath}`, we tell rustfmt to format the given path and update it on disk. That also makes rustftm not produce the formatted output to stdout. So the fix extension ends up writing out empty files. Differential Revision: https://phab.mercurial-scm.org/D7304

fix: replace str() by b'%d' for formatting integer Differential Revision: https://phab.mercurial-scm.org/D7303

zeroconf: improve the extension's documentation Differential Revision: https://phab.mercurial-scm.org/D7302

branchmap: pytype is confused about bytestr Differential Revision: https://phab.mercurial-scm.org/D7290

mail: suppress a pytype error that's just experimentally wrong It's probably more correct to pass a str here, but I'm not going to worry about it for now. Differential Revision: https://phab.mercurial-scm.org/D7283

dagparser: suppress some pytype errors around pycompat.bytestring I can't justify why we're getting these errors, but nothing I do fixes these handful of calls, so let's just move on with suppressions. Differential Revision: https://phab.mercurial-scm.org/D7277

encoding: add comment-based type hints for pytype Differential Revision: https://phab.mercurial-scm.org/D7275

merge with stable

dirs: reject consecutive slashes in paths We shouldn't ever see those, and the fuzzer go really excited that if it gives us a 65k string with 55k slashes in it we use a lot of RAM. This is a better fix than what I tried in D7105. It was suggested by Yuya, and I verified it does in fact cause the fuzzer to not OOM. This is a revision of D7234, but with the missing set of an error added. I added a unit test of the dirs behavior because I needed to reason more carefully about the failure modes around consecutive slashes. Differential Revision: https://phab.mercurial-scm.org/D7252

tests: quote $PYTHON in test-byteify-strings.t This is needed because the default install location is "Program Files" on Windows. Differential Revision: https://phab.mercurial-scm.org/D7251

repoview: define filteredchangelog as a top-level (non-local) class As suggested by Greg. This makes it easier for extensions to override the filtering. Differential Revision: https://phab.mercurial-scm.org/D7256

cmdutil: suppress bogus pytype errors pytype seems to handle our subclass of tuple (scmutil.status) especially poorly. We should probably file a bug. Differential Revision: https://phab.mercurial-scm.org/D7294

changegroup: suppress pytype error that's wrong Differential Revision: https://phab.mercurial-scm.org/D7293

changegroup: avoid shadowing a set with an int This confuses the heck out of pytype, and I basically agree with it that this shadowing is confusing. There's a chance this causes some memory to be freed later, but I think it's probably not worth worrying about for now. Differential Revision: https://phab.mercurial-scm.org/D7292

bundle2: more pytype suppressions around bytestring We clearly need to work around this somehow. Differential Revision: https://phab.mercurial-scm.org/D7291

archival: suppress some incorrect pytype failures Again, I suspect some missing coverage in type stubs for the stdlib. I'll revisit these later. Differential Revision: https://phab.mercurial-scm.org/D7288

simplemerge: disable a pytype error where it's just confused Differential Revision: https://phab.mercurial-scm.org/D7287

match: suppress error about subscripting an exception I get the sense I should file a bug about subscripting Exception subclasses. Differential Revision: https://phab.mercurial-scm.org/D7286

vfs: more attribute suppressions Differential Revision: https://phab.mercurial-scm.org/D7285

mdiff: mark diffopts as having dynamic attributes This class looks like we could move it to being an attrs and life would be better, but let's do that later. Differential Revision: https://phab.mercurial-scm.org/D7284

vfs: suppress some pytype errors around us using a private attribute Looking at threading._MainThread seems like we're probably a little unsupported, but since this code appears to work on both Python 2 and 3 I'm not going to sweat this for now. Differential Revision: https://phab.mercurial-scm.org/D7282

wireprototypes: disable pytype where it's just confused By inspection validnames is always a Set[bytes] here, but for some reason pytype is convinced it's a bytes. Let's disable the error for now. Differential Revision: https://phab.mercurial-scm.org/D7281

scmposix: another suppression on IOError subscripting As before, this will clear up when we move to Python 3-only. Differential Revision: https://phab.mercurial-scm.org/D7280

pvec: add an explicit type hint to help pytype Differential Revision: https://phab.mercurial-scm.org/D7279

procutil: suppress pytype warnings around windows-only attributes Differential Revision: https://phab.mercurial-scm.org/D7278

posix: add a pytype suppression This one seems a little sketchier than the others, but this will clean up when we're Python 3-only. Differential Revision: https://phab.mercurial-scm.org/D7276

color: suppress pytype warning on a windows-only module Differential Revision: https://phab.mercurial-scm.org/D7274

compression: tell pytype to not sweat a missing `zstd` module Differential Revision: https://phab.mercurial-scm.org/D7273

pytype: don't warn us about ignored-on-py3 metaclasses We can remove this when we're Python 3-only, but for now it's just too awkward to deal with and it's harmless. Differential Revision: https://phab.mercurial-scm.org/D7272

linelog: be more careful about types before looking for _target attribute Without this, pytype (correctly) complains that fakejge might not have a _cmprev attribute. We're operating on some constraints not expressible in the type system, so we detect the invalid case and raise a nicer exception now. Sadly, we also need an `assert` to give PyType a clue that we know what we're doing, but I can absolutely live with that. Differential Revision: https://phab.mercurial-scm.org/D7271

linelog: suppress annoying pytype warning about an ignored metaclass This isn't actively hurting us right now, and it's not trivial to set up this metaclass correctly on both Python 2 and Python 3 without dragging in all of `six`, but we still get lots of typechecking goodness with this error suppressed. Differential Revision: https://phab.mercurial-scm.org/D7270

hgignore: ignore generated pytype stubs Differential Revision: https://phab.mercurial-scm.org/D7269

hghave: add a check for pytype, Google's Python type checker I'm not unbiased, but I strongly prefer pytype to mypy: it does more inference, so we don't need as many annontations. In theory we can use both, but since I'm doing some tinkering with pytype I figured I'd just try setting it up. Differential Revision: https://phab.mercurial-scm.org/D7268

encoding: define local identify functions with explicit type comments This removes some obfuscation as far as pytype is concerned and corrects many bogus type errors. Differential Revision: https://phab.mercurial-scm.org/D7267

vfs: add NotImplementedError version of join Again, this helps out pytype. Differential Revision: https://phab.mercurial-scm.org/D7266

vfs: add a NotImplementedError implementation of __call__ This helps pytype considerably on this file. Differential Revision: https://phab.mercurial-scm.org/D7265

vfs: fix erroneous bytes constants Detected with pytype. Differential Revision: https://phab.mercurial-scm.org/D7264

pvec: migrate to modern integer division Detected with pytype. Differential Revision: https://phab.mercurial-scm.org/D7263

templateutil: fix a missing ABCMeta assignment Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7262

pvec: fix overlooked chr() call Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7261

vfs: another bytes-str confusion on thread name Differential Revision: https://phab.mercurial-scm.org/D7260

statprof: correctly always pass a str as the thread name Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7259

rust-status: remove dead code The `walk_explicit` function is only called when using a prefix matcher, which the Rust code does not yet support. This function will return in a future patch, probably with a different signature for performance reasons. With it, the `files` argument and its interface code can be removed for now. Differential Revision: https://phab.mercurial-scm.org/D7253

hghave: disallow symlinks on Windows Symlinks on Windows require either a special priviledge, or enabling Developer Mode. It's probably the latter that is enabled on the new CI machine. But since Mercurial itself is saying no to symlinks on Windows, the tests for symlinks shouldn't be attempted. This should fix a lot of the noise in the py3 tests. Differential Revision: https://phab.mercurial-scm.org/D7233

repoview: avoid wrapping changelog if there's nothing to filter This simplifies the code a bit by moving the optimizaton for no filtered revisions to one place. I assume it also makes working with repos without obsmarkers a little faster, but it doesn't seem significant. Differential Revision: https://phab.mercurial-scm.org/D7248

repoview: move changelog.flags() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7247

repoview: move changelog.parentrevs() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7246

repoview: move changelog.linkrev() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7245

repoview: move changelog.node() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7244

repoview: move changelog.rev() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7243

repoview: move changelog.strip() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7242

repoview: move changelog.headrevs() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7241

repoview: move changelog.revs() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7240

repoview: move changelog.__iter__() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7239

repoview: move changelog.__contains__() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7238

repoview: move changelog.tiprev() override to filteredchangelog Differential Revision: https://phab.mercurial-scm.org/D7237

repoview: wrap changelog class when filtering The class doesn't yet do anything. I'll move the filter-aware overrides from the changelog class over one by one to this class in coming patches. That will leave the changelog class simpler and will centralize more of the filtering logic to repoview. I could not measure any performance difference. Differential Revision: https://phab.mercurial-scm.org/D7236

repoview: extract a function for wrapping changelog I would like to clean up the changelog class by moving out knowledge of filtering. The filtering will instead be done in a class that wraps the changelog. This patch prepares for that. Differential Revision: https://phab.mercurial-scm.org/D7235

revlog: move tiprev() from changelog up to revlog This makes the changelog override simpler and it seems more consistent. Differential Revision: https://phab.mercurial-scm.org/D7250

tests: fix a few `(no-windows !)` conditionals that are really `(symlink !)` The CI for py3 is assuming that symlinks are possible (they are when running as Administrator or when Developer Mode is enabled), and these popped up. Differential Revision: https://phab.mercurial-scm.org/D7232

rust-matchers: add `Matcher` trait and implement `AlwaysMatcher` In our quest of a faster Mercurial, we have arrived at the point where we need to implement the matchers in Rust. This RFC mainly for the `Matcher` trait to see if the changes proposed feel fine to people with more experience on the matter. While the `AlwaysMatcher` implementation is here as a trivial example, it should be the first step towards matchers use in Rust as it is currently the only supported one. Notable changes: - `exact` is renamed to `exact_match` - enums for `visit*` methods with `Recursive` instead of `'all'`, etc. - a new `roots`, separate from `file_set` - no `bad`, `explicitdir` or `traversedir` functions as they can be passed to the high functions instead of the matchers Thanks to Martin for suggesting the last two (most important) changes and for reaching out to help a few weeks ago. Differential Revision: https://phab.mercurial-scm.org/D7178

merge with stable

perf: add a way to benchmark `dirstate.status` Getting more details about time spend in this specific internal bit is meaningful.

largefiles: delete obsolete and unused repo.push() The function was removed from localrepo in 4d52e6eb98ea (locarepo: remove the `push` method (API), 2014-09-25). Differential Revision: https://phab.mercurial-scm.org/D7174

censor: document that some commands simply ignore censored data I can't see a benefit for hg grep to ever error out early when it encounters censored data.

grep: warn on censored revisions instead of erroring out We need most of the grep logic to go through in case we encounter a censored revision, so we just return a None body for a censored node, and we stop just short of trying to record matches with the contents of that censored body. The other parts such as recording that the censored file has been considered at this revision needs to go into the proper dicts. I have also gotten weary of all the abbreviations, so while I did a small refactor to move the file-data-getting operation into a common function, I also expanded the abbreviations of the relevant variables within this little function. Hopefully some day this helps someone figure out what all the abbreviations mean. Although the censoring docs currently state that some commands error out or are ignored depending on the `censor.policy` config, I cannot see a benefit for grep to ever stop dead in its tracks when a censored revision is encountered. I will also amend the docs to indicate that some commands, such as grep, unconditionally ignore censored revisions.

py3: enable legacy fs encoding to fix filename compatibility on Windows This patch is untested. I just followed the instruction: https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8

rust-cpython: run cargo fmt

rust-cpython: remove useless PyResult<> from leak_immutable() The caller should know if the shared data is mutably borrowed or not.

rust-cpython: remove useless PyRefMut wrapper

rust-cpython: drop manual management of mutably_borrowed RefCell::borrow() should guarantee there's no mutable borrow. As a follow up, maybe PySharedState can be a pure data structure + function. Most ref-sharing business has already been moved to PySharedRef* and PyLeaked*.

rust-cpython: leverage RefCell::borrow() to guarantee there's no mutable ref Since the underlying value can't be mutably borrowed by PyLeaked, we don't have to manage yet another mutably-borrowed state. We can just rely on the RefCell implementation. Maybe we can add try_leak_immutable(), but this patch doesn't in order to keep the patch series not too long.

rust-cpython: remove useless Option<$leaked> from py_shared_iterator We no longer need to carefully drop the iterator when it's consumed. Mutation is allowed even if the iterator exists. There's a minor behavior change: next(iter) may return/raise something other than StopIteration if it's called after the iterator has been fully consumed, and if the Rust object isn't a FusedIterator.

rust-cpython: allow mutation unless leaked reference is borrowed In other words, mutation is allowed while a Python iterator holding PyLeaked exists. The iterator will be invalidated instead. We still need a borrow_count to prevent mutation while leaked data is dereferenced in Rust world, but most leak_count business is superseded by the generation counter. decrease_leak_count(py, true) will be removed soon.

rust-cpython: add generation counter to leaked reference This counter increments on borrow_mut() to invalidate existing leaked references. This is modeled after the iterator invalidation in Python. The other checks will be adjusted by the subsequent patches.

rust-cpython: add stub wrapper that'll prevent leaked data from being mutated In order to allow mutation of PySharedRefCell value while PyLeaked reference exists, we need yet another "borrow" scope where mutation is prohibited. try_borrow<'a> and try_borrow_mut<'a> defines the "borrow" scope <'a>. The subsequent patches will implement leak counter based on this scope. PyLeakedRef<T> and PyLeakedRefMut<T> could be unified to PyLeakedRef<&T> and PyLeakedRef<&mut T> respectively, but I didn't do that since it seemed a bit weird that deref_mut() would return a mutable reference to an immutable reference.

rust-cpython: rename PyLeakedRef to PyLeaked This series will make PyLeaked* behave more like a Python iterator, which means mutation of the owner object will be allowed and the leaked reference (i.e. the iterator) will be invalidated instead. I'll add PyLeakedRef/PyLeakedRefMut structs which will represent a "borrowed" state, and prevent the underlying value from being mutably borrowed while the leaked reference is in use: let shared = self.inner_shared(py); let leaked = shared.leak_immutable(); { let leaked_ref: PyLeakedRef<_> = leaked.borrow(py); shared.borrow_mut(); // panics since the underlying value is borrowed } shared.borrow_mut(); // allowed The relation between PyLeaked* structs is quite similar to RefCell/Ref/RefMut, but the implementation can't be reused because the borrowing state will have to be shared across objects having no lifetime relation. PyLeaked isn't named as PyLeakedCell since it isn't actually a cell in that leaked.borrow_mut() will require &mut self.

fix: fix grammar/typos in hg help -e fix

progress: flush stderr after clearing On python3, ui.stderr is buffered, it seems, so we need to flush it to actually get the progress bar off the screen. This is important since ui.write() will call into progbar.clear() if it thinks there's a progress bar on the screen, with the intent that the next thing it outputs is at the beginning of the line (instead of at the end of the progress bar line). Without the flush, we buffer up the clearing of the screen, and we get some really weird/corrupt output. Differential Revision: https://phab.mercurial-scm.org/D7784

subrepo: fix a crash when archiving an svn or git subrepo Only hgsubrepos have a repository attribute. This is pretty hacky, but probably the best we can do on stable. Pushing the lfstatus check down into the wrapper for hgsubrepo (and dropping the check for lfstatus at the top of `hgsubrepoarchive()`) resulted in various test failures because: 1) hgsubrepoarchive isn't returning the number of files archived at the bottom, resulting in an error about += NoneType 2) These copypasta archive wrappers don't use progress bars 3) Largefiles are *not* currently archived when using extdiff (68822b7cdd01), but pushing this context manager down into the subrepo resulted in it apparently doing so (as evidenced by progress bars being dropped) The other uses of `lfstatus()` are not in the substate processing loop, so they shouldn't be an issue. I initially put testcases in this test for largefiles-{on,off}, and it flagged a bunch of exit code differences for `cat` and `diff`, so I backed that off. Differential Revision: https://phab.mercurial-scm.org/D7714

Added signature for changeset b4c82b704180

Added tag 5.2.2 for changeset b4c82b704180

py3: force bytestr conversion of "reason" in scmutil.callcatch() For instance, reason may be an InvalidURL as shown in added test.

rust-cpython: do not convert warning pattern to utf-8 bytes On Unix, both Rust Path and Mercurial expect a locale-dependent bytes, and we don't support Windows yet. Differential Revision: https://phab.mercurial-scm.org/D7614

rust-cpython: import utils::files::* function at module level IIRC, it's common in Rust to call functions with the module prefix. Differential Revision: https://phab.mercurial-scm.org/D7613

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7612

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7611

rust-threads: force Rayon to respect the worker count in config As per the inline comment, this is a workaround because Rust code does not yet know how to read config files. Differential Revision: https://phab.mercurial-scm.org/D7610

remotefilelog: remove the `ensurestart` usage The option does not seems to do what we want since it does not wait on the background process actually starting (eg: taking the lock). And the race have been dealt with another way anyway. This effectively back out changeset 2c74337e6483. Differential Revision: https://phab.mercurial-scm.org/D7590

remotefilelog: remove most of the debugwait* command in the tests Since we use a dedicated developer config, we do not need them anymore. Differential Revision: https://phab.mercurial-scm.org/D7589

remotefilelog: use the correct redirection in tests The test were using '2>%1' which just redirect stderr in the "%1" file. The intend was '2>&1' which redirect stderr on stdin. This impact tests because some commit were picking up this '%1' files. Differential Revision: https://phab.mercurial-scm.org/D7588

remotefilelog: remove most sleep in tests Now that we removed most of the races, we can remove the infamous sleeps. Differential Revision: https://phab.mercurial-scm.org/D7587

remotefilelog: have command wait for background task in the tests We leverage the feature introduced in the two previous commit to have the main command wait for the background task to be finished before returning. All test still pass and instability seems to have gone away. Most of the sleep and explicit wait are now useless and will be cleaned up in later changesets. Differential Revision: https://phab.mercurial-scm.org/D7586

remotefilelog: add a developer option to wait for background processes In order to block the main command on the subprocess exiting, we ensure the repo's ui object will call the subprocess.wait() method to ensure the top-level hg process doesn't exit until all background processes have also done so. Currently, in the tests, most operation spawning background process as followed by commands waiting for these operations to complete. However this waiting is racy. First because it seems like we can start waiting before the background operation actually start, in which case it is prematurely detected as "done". Second, because some commands may spawn multiple background operation for the same operation (eg: rebase can apparently trigger multiple prefetch). The current approach could be updated to maybe handle the first issue, but the second one will never be properly handled. In most case, we do not care that the bg process keep running after the command end. (Since we explicitly wait for them to end before doing anything else). So we add an option to wait on the background process before exiting the command. We'll put it in use in the next changeset. Differential Revision: https://phab.mercurial-scm.org/D7585

procutil: add a option to not fully detach background process When writing test, it is useful to make code pretending to spawn a fully detached process while still actually waiting for it to finish. We add the option to `procutil.runbgcommand`, to do so. We will use this in remote-filelog tests. Differential Revision: https://phab.mercurial-scm.org/D7584

Added signature for changeset a50fecefa691

Added tag 5.2.1 for changeset a50fecefa691

windows: fix an issue causing registry config paths to be ignored on py3 `util.lookupreg()` returns bytes. Differential Revision: https://phab.mercurial-scm.org/D7532

tests: cover revision conversion logic in githelp tests There was no test involving actual conversion of option values when they contain a git revision name (to be converted as a hg one by hgext.githelp.convert()). Adding one. This test would fail on Python 3 without https://phab.mercurial-scm.org/D7537. Differential Revision: https://phab.mercurial-scm.org/D7540

pvec: fix a `str` type conditional for py3 Differential Revision: https://phab.mercurial-scm.org/D7538

githelp: fix a `str` type conditional for py3 Differential Revision: https://phab.mercurial-scm.org/D7537

histedit: fix an `isinstance(nodelist, str)` check for py3 Differential Revision: https://phab.mercurial-scm.org/D7536

win32mbcs: fix a `str` type conditional for py3 Differential Revision: https://phab.mercurial-scm.org/D7535

hgweb: fix a few `str` type conditional for py3 Differential Revision: https://phab.mercurial-scm.org/D7534

repair: fix an `isinstance(nodelist, str)` check for py3 All of the callers appear to pass a list, so this doesn't fix anything in core hg. But maybe out of tree extensions use this shortcut. Differential Revision: https://phab.mercurial-scm.org/D7533

log: map None rev to wdirrev when filtering revisions with --line-range When 'hg log -f --line-range <file>,<range>' is invoked with <range> containing uncommitted changes, the command crashes on Python 3 as follows: [...] File "/usr/lib/python3/dist-packages/mercurial/commands.py", line 4725, in log revs, differ = logcmdutil.getlinerangerevs(repo, revs, opts) File "/usr/lib/python3/dist-packages/mercurial/logcmdutil.py", line 933, in getlinerangerevs if rev not in userrevs: File "/usr/lib/python3/dist-packages/mercurial/smartset.py", line 969, in __contains__ if l < x: TypeError: '<' not supported between instances of 'int' and 'NoneType' The None value is because requested line range has uncommitted changes, so 'rev' is the working directory revision. This only occurs in Python 3 as Python 2 allows comparing None with int. As suggested by Yuya Nishihara, mapping None to node.wdirrev resolves the issue and also make the '--line-range' option properly work with -r 'wdir()'. We add extra tests for non-regression and to illustrate handling of 'wdir()'.

tests: check that 'log --line-range' follows uncommitted changes The reason we start walking revisions from the working directory (None revision) in logcmdutil.getlinerangerevs() is because we can follow uncommitted changes. Adding a test to illustrate this based on an uncommitted rename as there was none before. This helps understand the fix in next changeset.

test: don't put $BINDIR in $PATH for test-merge-tools.t We call $BINDIR/hg explicitly anyway, so don't need it in $PATH. This fixes failures when running the test --with-hg=/usr/bin/hg, where we pick up /usr/bin/false as merge tool when we expected not to find it.

webutil: add missing argument to join() Differential Revision: https://phab.mercurial-scm.org/D7516

singlehead: making config item a bool again with the use of `configsuboptions`, the main config item has become a string (unless it's just the default value). This makes it in particular hard to override in a cascade of HGRC files, as we do in Heptapod to re-allow multiple heads on specific repositories while the default behaviour is to forbid them. The added test case reflects that use-case

util: convert an exception to bytes when passing to Abort() I happened to notice this searching for how to convert an exception to bytes in the previous patch. I'm pretty sure I've got a bunch of other instances that use `pycompat.bytestr()` suppressed locally. Differential Revision: https://phab.mercurial-scm.org/D7467

patch: fix a str + bytes issue in an exception handler Flagged by pytype. Differential Revision: https://phab.mercurial-scm.org/D7466

py3: wrap a __func__ in sysbytes() before logging as bytes Differential Revision: https://phab.mercurial-scm.org/D7461

py3: make doc strings containing deprecated '\.' escape sequence raw strings Differential Revision: https://phab.mercurial-scm.org/D7462

shelve: add the missing `create` parameter to the bundlerepo constructor Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7458

shelve: fix a missing variable in the exception handler for delete Caught by pytype. I haven't paid much attention to the progress of this extension, but I *think* this was the intent. Differential Revision: https://phab.mercurial-scm.org/D7457

py3: use pycompat.bytestr() instead of pycompat.sysstr() pycompat.sysstr() doesn’t work because it doesn’t accept arguments of type `type` and returns a unicode object on Python3, while the format string wants a bytes-like object.

zeroconf: fix traceback under py3 hg serve under py3 caused struct.error: char format requires a bytes object of length 1 <pulkit25> ah, I think that should be `pycompat.bytechr` instead of chr

cffi: fix build on Python 3 CFFI expects the arguments to be of type str, which means that the string literals should not have the `b` prefix.

pure: use string for another exception in the pure version of base85 That message does not seems tested, but I am assuming that the same reasoning as for the previous changeset applies.

pure: use string for exception in the pure version of base85 Without this change, running the test with python3 and --pure gives the following error:: --- /home/marmoute/src/mercurial-dev/tests/test-import-git.t +++ /home/marmoute/src/mercurial-dev/tests/test-import-git.t.err @@ -518,7 +518,7 @@ > > EOF applying patch from stdin - abort: could not decode "binary2" binary patch: bad base85 character at position 6 + abort: could not decode "binary2" binary patch: b'bad base85 character at position 6' [255] $ hg revert -aq To make the cext implementation, we use a "native" string for the exception. This fix the test failure.

py3: avoid iterating over a literal bytes in highlight In Python 3, iterating over a bytes literal yields integers. Since we use the value in `text.replace()`, this fails on Python 3 with the following trackback: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mercurial/hgweb/hgwebdir_mod.py", line 378, in run_wsgi for r in self._runwsgi(req, res): File "/usr/lib/python3/dist-packages/mercurial/hgweb/hgweb_mod.py", line 326, in run_wsgi for r in self._runwsgi(req, res, repo): File "/usr/lib/python3/dist-packages/mercurial/hgweb/hgweb_mod.py", line 449, in _runwsgi return getattr(webcommands, cmd)(rctx) File "/usr/lib/python3/dist-packages/mercurial/hgweb/webcommands.py", line 211, in file return _filerevision(web, webutil.filectx(web.repo, web.req)) File "/usr/lib/python3/dist-packages/hgext/highlight/__init__.py", line 72, in filerevision_highlight pygmentize(web, b'fileline', fctx, web.tmpl) File "/usr/lib/python3/dist-packages/hgext/highlight/__init__.py", line 58, in pygmentize field, fctx, style, tmpl, guessfilenameonly=filenameonly File "/usr/lib/python3/dist-packages/hgext/highlight/highlight.py", line 62, in pygmentize text = text.replace(c, b'') TypeError: a bytes-like object is required, not 'int'

relnotes: copy "next" to "5.2" and clear "next" This is the same thing as we did for 5.1 in cba59b338976 (relnotes: copy "next" to "5.1" and clear "next", 2019-08-01). Differential Revision: https://phab.mercurial-scm.org/D7231

py3: avoid `b'%s' % type(...)` in a ProgrammingError Differential Revision: https://phab.mercurial-scm.org/D7363

py3: fix sorting of obsolete markers in obsutil (issue6217) This is similar to 01e8eefd9434 and others. We move the sortedmarkers() function from exchange module to obsutil.

py3: fix handling of ctrl keys in crecord (issue6213) The "keypressed" value in handlekeypressed() is a key name obtained by curses's getkey(); this can be a multibyte string for special keys like CTRL keys. Calling curses.unctrl() with such a value fails on Python 3 with a TypeError as described in issue6213. (On Python 2, this does not crash, but I'm not sure the result is correct, though it does no matter here.) So instead of calling unctrl(), we compare "keypressed" with the expected "^L" obtained by curses.ascii.ctrl("L").

py3: keep "keypressed" a native str in crecord This will help in the next changeset by avoiding a decode step. Also, the actual bytes conversion seems superfluous since values coming from curses's getkey() will be a native string. As a consequence, we open the "testcommands" file (used in test-interactive-curses.t) in text mode.

py3: compare response of crecord's confirmationwindow with str confirmationwindow() returns a native string, as a result of calling chr() on getch(). On Python 3, response.lower().startswith(b"y") leads to a TypeError. This fixes a crash when typing "r" in the curses interface of interactive commit.

py3: compare http server's command with a native string The "command" attribute is an str, so comparing with a bytes would not work on Python 3. This might solve issues in test-lfs-serve-access.t that happens sometimes (especially in CI): --- /hgwork/src/tests/test-lfs-serve-access.t +++ /hgwork/src/tests/test-lfs-serve-access.t.err @@ -163,11 +163,13 @@ $ cat $TESTTMP/access.log $TESTTMP/errors.log $LOCALIP - - [$LOGDATE$] "POST /missing/objects/batch HTTP/1.1" 404 - (glob) + $LOCALIP - - [05/Nov/2019 16:32:34] "{"objects": [{"oid": "f03217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e", "size": 20}], "operation": "download"}" HTTPStatus.BAD_REQUEST - $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=capabilities HTTP/1.1" 200 - (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=batch HTTP/1.1" 200 - x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point?cmd=getbundle HTTP/1.1" 200 - x-hgarg-1:bookmarks=1&bundlecaps=HG20%2Cbundle2%3DHG20%250Abookmarks%250Achangegroup%253D01%252C02%252C03%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps%250Arev-branch-cache%250Astream%253Dv2&cg=1&common=0000000000000000000000000000000000000000&heads=525251863cad618e55d483555f3d00a2ca99597e&listkeys=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$ partial-pull (glob) $LOCALIP - - [$LOGDATE$] "POST /subdir/mount/point/.git/info/lfs/objects/batch HTTP/1.1" 200 - (glob) $LOCALIP - - [$LOGDATE$] "GET /subdir/mount/point/.hg/lfs/objects/f03217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e HTTP/1.1" 200 - (glob) + $LOCALIP - - [05/Nov/2019 16:32:34] code 400, message Bad request version ('"download"}') Blobs that already exist in the usercache are linked into the repo store, even though the client doesn't send the blob. @@ -195,6 +197,7 @@ server2/.hg/store/lfs/objects/f0/3217a32529a28a42d03b1244fe09b6e0f9fd06d7b966d4d50567be2abe6c0e $ "$PYTHON" $RUNTESTDIR/killdaemons.py $DAEMON_PIDS $ cat $TESTTMP/errors.log + $LOCALIP - - [05/Nov/2019 16:32:34] code 400, message Bad request version ('"download"}') $ cat >> $TESTTMP/lfsstoreerror.py <<EOF > import errno (from https://ci.hg.gregoryszorc.com/job-info/hg-committed-ca3dca416f8d5863ca6f5a4a6a6bb835dcd5feeb-debian10-cpython-3.7-0)

py3: don't use bytes with vars() or __dict__ Inspired by D7227. These were all the remaining instances I could find. Differential Revision: https://phab.mercurial-scm.org/D7230

Added signature for changeset ca3dca416f8d

Added tag 5.2 for changeset ca3dca416f8d

py3: add inline comment about encoding issue of str(Abort())

py3: do not reimplement Abort.__str__() on Python 2 It isn't necessary on Python 2, and the default implementation should be better than our BaseException_str() clone.

tests: write out file using bytes I/O The encoding of sys.stdout varies between Python versions. So using a one-liner to write a file from a Unicode string is not deterministic. This commit writes out the file using bytes I/O to ensure we have exactly the bytes we want in the file. This change fixes a test failure in Python 3.5/3.6. Differential Revision: https://phab.mercurial-scm.org/D7226

import-checker: open all source files as utf-8 Before, we opened in text mode and used the default encoding to interpret the bytes within. This caused problems interpreting some byte sequences in some files. This commit changes things to always open files as UTF-8, which makes the error go away. test-check-module-imports.t now passes on Python 3.5 and 3.6 with this change. Differential Revision: https://phab.mercurial-scm.org/D7225

localrepo: use str for lookup in vars() vars() returns a dict of str. So always use a native str for the key lookup. Differential Revision: https://phab.mercurial-scm.org/D7227

automation: install python3-venv Debian package Debian's python install has a crippled venv by default, as it is lacking ensurepip. When you try to run `python3 -m venv` it tells you to install `python3-venv`. So this commit does that in our automation environment so we can fully test installing Mercurial using venv+pip with the system Python. Differential Revision: https://phab.mercurial-scm.org/D7229

tests: look for ensurepip before using venv Debian appears to cripple the venv module by default by removing the associated ensurepip functionality. (The module isn't present at all.) This caused test-install.t to fail when using the Debian python3 unless the python3-venv package was installed. This commit introduces a new hghave requirement for detecting ensurepip and makes the Python 3 install variant conditional on its presence. This should make test-install.t pass when using an incomplete Debian Python. Differential Revision: https://phab.mercurial-scm.org/D7228

automation: avoid '~' in the temp directory on Windows If a long-ish username is used, the environment variable ends up with a '~' to be 8.3 path compatible. That in turn causes a handful of tests (mostly ssh related) to add quotes around $TESTMP. I have no AWS experience, so I have no idea if this is the proper way to do it. But I've hit this problem locally, and redirecting the directory is a workaround. I don't recall if the directory is created on demand by the test harness, but presumably if this is configured before the machine boots, Windows will do it for us. Differential Revision: https://phab.mercurial-scm.org/D7130

tests: use venv on Python 3 This test was failing in some Python 3 environments because `$PYTHON -m virtualenv` was somehow resulting in Python 2 being used. Why, I'm not sure. Python 3 includes virtualenv in the standard library as the `venv` module. This commit changes test-install.t to use `$PYTHON -m venv` on Python 3 and `$PYTHON -m virtualenv` on Python 2 (if available). I chose to make some test output duplicated because we can't have nested conditionals and there is no easy way to express ORing of hghave checks. Differential Revision: https://phab.mercurial-scm.org/D7224

tests: remove HGALLOWPYTHON3 reference This variable was removed from setup.py in c3e10f705a6c. Differential Revision: https://phab.mercurial-scm.org/D7223

run-tests: use byte strings for inserted output We were inserting str on Python 3 which resulted in mixed str/bytes types on the list. This would later blow up when trying to write str to the .err file opened in bytes mode. Differential Revision: https://phab.mercurial-scm.org/D7222

contrib: require Python 3.7 for byteify-strings.py bb509f39d387 made an error, it's actually 3.7 that introduced token.COMMENT. Differential Revision: https://phab.mercurial-scm.org/D7220

hghave: fix bytes/string issue on Python 3

packaging: add support for CentOS 8 The resulting executable has not been tested in detail yet. I ran 'hg version' and 'hg clone', which worked fine (except for extensions acting up due to Python 3). Differential Revision: https://phab.mercurial-scm.org/D7216

packaging: allow choosing python version depending on centos version Differential Revision: https://phab.mercurial-scm.org/D7217

fsmonitor: use stringutil.forcebytestr() instead of str() on an exception Similar to 5fa8ac91190e / D7206, should get test-install.t passing on py3. Differential Revision: https://phab.mercurial-scm.org/D7218

py3: add a __str__ method to Abort This improves the rendering of some exceptions by avoiding raw bytestrings, especially when using --traceback option.

py3: add Python 3 exception output to test-lfs-serve-access.t Similar to a973a75e92bf or 3e9c6cef949b.

automation: install black This should unblock us from running the code formatting test in our automated environment. Differential Revision: https://phab.mercurial-scm.org/D7197

py3: use %d to format an int Avoids a TypeError under py3. Fortunately this is very much an edge case since it requires the user to have deliberately created a local tag of the form 'D\d+' that isn't truthful. Differential Revision: https://phab.mercurial-scm.org/D7215

fsmonitor: normalize exception types to bytes Unavailable.msg should now always be bytes. We also rename Unavailable.__str__ to __bytes__ as it always returns bytes. We make __str__ a simple wrapper that decodes that result to str. There's probably some excessive strutil.forcebytestr() in fsmonitor/__init__.py now. But at least the exceptions around type coercion should now be gone. Differential Revision: https://phab.mercurial-scm.org/D7214

fsmonitor: normalize clock value to bytes We normalize the value returned by watchman because we perform a number of compares with this value in code. So the easiest path forward is to normalize to bytes so we don't have to update many call sites. With this commit, the fsmonitor extension appears to be working with Python 3! Although there are still some failures in edge cases... Differential Revision: https://phab.mercurial-scm.org/D7213

fsmonitor: use next() instead of .next() This is needed for Python 3 compatibility. Differential Revision: https://phab.mercurial-scm.org/D7212

fsmonitor: normalize Watchman paths to bytes Otherwise it will be a str on Python 3 and operations below which operate in the bytes domain will fail. Differential Revision: https://phab.mercurial-scm.org/D7211

fsmonitor: handle unicode keys in tuples In Python 3, keys in the bset tuple are typically str, not bytes. PyBytes_AsString() would return NULL. But we weren't checking the return value and this would lead to a segfault. This commit makes the code type and Python version aware. The Python version specific code is to allow us to utilize a modern API for converting str -> char* without having to allocate an extra PyObject. FWIW I wanted to assume that keys were always str. However, there appear to be some bytes keys in some cases. I haven't debugged this further. Differential Revision: https://phab.mercurial-scm.org/D7210

fsmonitor: make _hashignore compatible with Python 3 The Hasher wants a bytes but we were feeding it a str. Let's use our repr() implementation to return bytes. In addition, the hexdigest() would return a str, which would be compared against a bytes and would always fail. Normalize to bytes so the compare works. Differential Revision: https://phab.mercurial-scm.org/D7209

fsmonitor: normalize hostname to bytes Without this, we get a str/bytes mismatching when using % formatting a few lines below. Differential Revision: https://phab.mercurial-scm.org/D7208

fsmonitor: access repo.root There is no repo._root. It looks like fsmonitor has been busted since this access was introduced in ab1900323b1 in July 2019! Differential Revision: https://phab.mercurial-scm.org/D7207

fsmonitor: coerce watchman exception to bytes Without this, we get errors due to passing str to a function which expects bytes. Differential Revision: https://phab.mercurial-scm.org/D7206

fsmonitor: fix str/bytes mismatch when accessing watchman version There were 2 bugs here. First, keys in the tuple are always str. Second, we needed to normalize the value to bytes to prevent a str/bytes mismatch on Python 3. With this commit, `hg debuginstall` with fsmonitor enabled now works on Python 3. Differential Revision: https://phab.mercurial-scm.org/D7205

fsmonitor: reapply b1f62cd39b5c The recent revendoring of pywatchman undid this changeset. Let's reapply it. This commit was generated by running `hg graft -f b1f62cd39b5c`. It applied cleanly. Differential Revision: https://phab.mercurial-scm.org/D7204

fsmonitor: reapply dd35abc409ee The recent revendoring of pywatchman undid this bug fix. Let's reapply it. Differential Revision: https://phab.mercurial-scm.org/D7203

fsmonitor: remove pywatchman from exclusion rule The recently vendored pywatchman code base is now formatted with black. We can now remove pywatchman from our black exclusion rule. Differential Revision: https://phab.mercurial-scm.org/D7202

fsmonitor: refresh pywatchman with upstream This commit vendors pywatchman commit 259dc66dc9591f9b7ce76d0275bb1065f390c9b1 from upstream without modifications. The previously vendored pywatchman from changeset 16f4b341288d was from Git commit c77452. This commit effectively undoes the following Mercurial changesets: * dd35abc409ee fsmonitor: correct an error message * b1f62cd39b5c fsmonitor: layer on another hack in bser.c for os.stat() compat (issue5811) * c31ce080eb75 py3: convert arguments, cwd and env to native strings when spawning subprocess * 876494fd967d cleanup: delete lots of unused local variables * 57264906a996 watchman: add the possibility to set the exact watchman binary location The newly-vendored code has support for specifying the binary location, so 57264906a996 does not need applied. But we do need to modify our code to specify a proper argument name. 876494fd967d is not important, so it will be ignored. c31ce080eb75 globally changed the code base to always pass str to subprocess. But pywatchman's code is Python 3 clean, so we don't need to do this. This leaves dd35abc409ee and b1f62cd39b5c, which will be re-applied in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D7201

py3: encode strings before setting rev summary in gnuarch converter

py3: use raw string to query EmailMessage in gnuarch converter

py3: use mail.parsebytes() in gnuarch catlog parser We drop 'catlogparser' attribute now unused.

tests: handle Message-Id line wrapping in test-notify-changegroup.t This fixes this test on Python 3 with a long hostname. See changeset 4128ffba4431 for details.

py3: define and use json.loads polyfill Python 3.5's json.loads() requires a str. Only Python 3.6+ supports passing a bytes or bytearray. This commit implements a json.loads() polyfill on Python 3.5 so that we can use bytes. The added function to detect encodings comes verbatim from Python 3.7.

contrib: require Python 3.6 for byteify-strings.py This script makes use of `token.COMMENT`, which apparently isn't present until Python 3.6. So make the script and its test conditional on Python 3.6.

demandimportpy3: only use lazy extension loader on Python 3.6+ There was an inline comment denoting a bug in the lazy extension loader on Python 3.5 which prevents it from working there. But the code was not conditional on the Python version. The result of this was a myriad of failures on Python 3.5 due to getattr() and friends not working on lazy extension modules. By making extension modules non-lazy on Python 3.5, we reduce the number of test failures from 48 to 22 on that Python version.

ui: flush before prompting for input with readline I was using `hg absorb` with Python 3 and noticed that the prompt was appearing without any output about what would be done. After I answered the prompt, the output was printed to stdout. This appears to be a buffering difference between Python 2 and Python 3. To work around it, this commit adds an explicit flush() before calling the raw input function when readline is used.

histedit: restore hex nodeids to be 12 digits long I accidentally switched from 12 digits to 40 digits while making the code py3-compatible. Thanks to Yuya for noticing. Differential Revision: https://phab.mercurial-scm.org/D7200

tests: fix typo "includfe" Differential Revision: https://phab.mercurial-scm.org/D7180

py3: fix fsmonitor's _watchmantofsencoding exception message encoding Differential Revision: https://phab.mercurial-scm.org/D7190

py3: use native strings as keys into **opts in chistedit Now you should be able to successfully confirm your histedit plan (at least in the case I tried). Even continuing after conflicts and finishing the histedit worked. Differential Revision: https://phab.mercurial-scm.org/D7186

py3: open chistedit file in binary mode using vfs We write bytes to the file, so it should be open in binary mode. Opening it via the vfs takes care of that for us. Now you'll get yet a different traceback if you try to confirm you histedit plan. Differential Revision: https://phab.mercurial-scm.org/D7185

py3: avoid another b''.format() in chistedit Now you'll get a different traceback if you try to confirm you histedit plan. Differential Revision: https://phab.mercurial-scm.org/D7184

py3: render message about conflicts in chistedit code Now you can also reorder commits that (potentially) conflict. Just don't try to confirm the changes yet (because then it crashes). Differential Revision: https://phab.mercurial-scm.org/D7183

py3: handle keypresses in chistedit Now you can navigate and change the action for a commit. You can also reorder commits, as long as that doesn't result in a conflict (then it crashes). Differential Revision: https://phab.mercurial-scm.org/D7182

py3: make chistedit render Now you can see the list of commits, but it crashes when you press a key. Differential Revision: https://phab.mercurial-scm.org/D7181

py3: fix exception display encoding in contrib/simplemerge.py Differential Revision: https://phab.mercurial-scm.org/D7191

py3: fix exception message check in test-linerange.py's testOutOfRange Differential Revision: https://phab.mercurial-scm.org/D7192

py3: fix exception message encoding in scmutil.py's simplekeyvaluefile.read Differential Revision: https://phab.mercurial-scm.org/D7193

py3: fix crecord.py's editpatchwitheditor exception message encoding Differential Revision: https://phab.mercurial-scm.org/D7194

py3: fix exception message encoding in infinitepush Differential Revision: https://phab.mercurial-scm.org/D7195

py3: fix fsmonitor's _handleunavailable exception message encoding Differential Revision: https://phab.mercurial-scm.org/D7196

packaging: update built-in Fedora support to Fedora 31 This is now quite easy ...

packaging: refactor "fedora29" target to a single more generic "fedora" target Fedora moves fast in version numbers, and often with Mercurial packaging being backwards compatible. Also, most people use the system package. There is thus much work and tech debt and little value in providing explicit built-in support for several versions. Thus, only aim for providing built-in support for latest Fedora version, and make it easy to update.

packaging: make dockerrpm fedora target more generic Fedora moves fast in version numbers, and often with Mercurial packaging being backwards compatible. Thus, only aim for providing built-in support for latest Fedora version, and make it easy to update. With this refactoring, 'dockerrpm fedora31' also works. 'dockerrpm fedora' will use the 'fedora:latest' Docker image.

packaging: use "python3" for fedora29 ... and as buildrpm default Change the buidrpm default. The CentOS targets explicitly use "python", and changing the default will only influence Fedora 29. A Python 3 package needs python3 dependencies, so pythonexe (and pythonver) is used for specifying dependencies. Other OS versions will keep using "python" as before ... or potentially change to explicit "python2". Fedora 29 packages can thus also still be built for Python 2 - just not in the docker image that is updated for Python 3.

packaging: use "--python python" for centos7 to avoid explicit "python2" This is a partial backout of 92a51a45d44c. We will need to be able to control whether package dependencies are python2 or python3. Generally (at least in recent Fedora), the package prefix match the name of the python executable ... but CentOS 7 doesn't use the python2 prefix in package name or alias for python-docutils yet, so just keep centos7 in the unversioned "python" world. Change the new (unused) buildrpm "--python3" option (introduced in a6dcac6454c1) to "--python python3" to get a more generic method for explicit control over whether we use python, python2 or python3.

packaging: fix docker-centos5 - use pythonexe and set to "python" as before Fix 92a51a45d44c .

packaging: move dockerrpm output directory creation to dockerrpm Avoid having to compute the directory in two places in different environments.

packaging: drop "support" for unsupported Fedora versions Fedora 31 has just been released, and Fedora 29 will be EOL in a month. Don't spend any time thinking about dead stuff.

mail: black wants to add this blank line I can't figure out how this got overlooked on previous runs, but here we are. It looks like the culprit change is already public?

hghave: verify we have a black that is new enough for our format We require what is currently the absolute latest black, so let's be paranoid.

contrib: fix up example fix configuration for our move to released black

phabricator: use True primitive instead of b'true' for phabupdate actions Something I'd missed in the creatediff port. This didn't matter before with the old PHP form style wireformat, but breaks with the new arcanist format. Differential Revision: https://phab.mercurial-scm.org/D7152

setup: allow py3 install without env vars 5.2 is the first release of Mercurial where py3 support is expected to be widely used, therefore we should allow installing it without hoop-jumping. Differential Revision: https://phab.mercurial-scm.org/D7151

formatting: drop `grey`, our custom black version Now that the official black has all we want, we can drop this.

formatting: using black to check for formatting

formatting: run black version 19.10b0 on the codebase The latest version of black is out and contains the change we needed. So we can start using it now. note: `test-check-format.t` will complains about this changes because it still use `grey` and need to be migrated to `black`. See next changesets for this.

packaging: fix buildrpm whitespace

packaging: drop outdated buildrpm "tested on" comment Packaging usually works on other versions too, and it is not efficient to maintain the list in repo. It is already out of sync with the Makefile targets.

packaging: also include hgweb.wsgi in rpms

packaging: introduce Python3 support as buildrpm --python3 Just overrule the HGPYTHON3 warning.

packaging: be explicit about Python version in rpm spec Fedora 31 has Python3 at /usr/bin/python ... but expect everybody to not just find python in $PATH but be explicit about whether they want python2 or python3. mercurial.spec just used 'python' and would fail when it unknowingly used Python 3 and ended up with Mercurial setup.py reporting "Python 3.7 detected." and talking about the HGPYTHON3 environment variable. For now, just be explicit about using system python2 as python executable when building rpms.

packaging: make python snippets in rpm building python3 compatible Fedora 31 has Python3 at /usr/bin/python, and buildrpm would fail on snippets that use python2 syntax. Instead of forcing python2, just accept for the future while staying backwards compatible.

py3: fix patchbomb to accept non-ASCII header value for email preview Since mail.headencode() is disabled by -n/--test, non-ASCII header value has to be allowed. Spotted by Denis Laxalde.

tests: check patchbomb with a non-ascii commit message This fails on Python 3 but gets fixed in the next changeset.

formatter: fix handling of None value in templater mapping For historical reasons, None in mapping dict means there's no such keyword, and falls back to b"". That's fine in log templates where mapping item is generally a callable returning a value (which may be None,) but the formatter directly puts an "evaluated" value in the mapping. So the None value has to be lifted to wrappedvalue(None) to avoid confusion in the template engine.

config: add support for defaultvalue of list of printable elements

config: fix -Tjson to not crash due to unsupported defaultvalue types Maybe it isn't great to ignore unsupported types at all, but otherwise "hg config -Tjson" would crash.

tests: handle Message-Id email header possible wrapping The "Message-Id" header will get wrapped with a new line when exceeding 75 characters on Python 3 (see changeset 7d4f2e4899c5 introducing usage of email.header.Header.encode and respective doc). This will occur in an unpredictable manner depending on the hostname's length. To make the test output consistent across Python versions and hostname configuration, we add a filter to unwrap this header value.

py3: leverage pycompat.long

packaging: remove version info from Breaks+Replaces in Debian package The versioned Breaks: and Replaces: cause problem when trying to install our package over the one in Debian. $ sudo apt install ./packages/debian-buster/mercurial_5.2~rc0+15-buster-a2ff3aff81d2_amd64.deb Reading package lists... Done Building dependency tree Reading state information... Done Note, selecting 'mercurial' instead of './packages/debian-buster/mercurial_5.2~rc0+15-buster-a2ff3aff81d2_amd64.deb' Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: mercurial : Breaks: mercurial-common (< 5.2~rc0+15-buster-a2ff3aff81d2) but 5.2~rc0-1 is to be installed E: Unable to correct problems, you have held broken packages. Removing version information resolves the situation.

py3: fix generated non-ascii message in test-notify.t The resulting "Subject:" header contains an encoded word in Python 3 so we have to add distinct outputs for Python 3 but underlying values are the same: >>> from email.header import decode_header >>> decode_header('=?utf-8?b?w6AuLi4=?=') [(b'\xc3\xa0...', 'utf-8')]

py3: decode encoding literal before passing to .decode() bytes.decode(<encoding>) wants an str as "encoding" parameter, it raises a TypeError if given a bytestring. encoding.encoding and encoding.fallbackencoding are bytes values.

py3: decode payload of notify email This fixes one UnicodeEncodeError in test-notify.t:422 when testing the notify hook with non-ascii content (there are more later). We only decode on Python 3, since it's not safe for sure on Python 2.

py3: decode email headers with mail.headdecode() in notify extension

py3: use stdlib's parseaddr() to get sender header in notify extension In Python 3, email headers are unicode string so using stringutil.email() will not work as it compares with bytestring. So let's use email.utils.parseaddr() from the stdlib which has a consistent behavior across Python versions. The same is done in patchbomb extension already.

py3: use a BytesParser in notify extension This is the first step to make the "long line" case in test-notify.t pass by fixing a UnicodeDecodeError on Python 3. We alias a parsebytes() in mail module, similarly as we already have a parse() function for Python 2 and Python 3 compatibility.

py3: fix headencode() with display=False We previously called str() on a email.header.Header object. On Python 2, this returns a bytestring and the __str__ method is actually an alias to .encode() method. On Python 3, __str__ does not perform encoding (and returns a unicode string). To keep a consistent behavior across Python versions, we explicitly use .encode() and we wrap the result with encoding.strtolocal() to get a bytestring in all cases. As a side effect of forcing bytes conversion, we need to decode back in _addressencode(). This is to make test-notify.t pass on Python 3. Also note that headers are now encoded in some patchbomb tests; this is because the charset is not always "us-ascii" ("iso-8859-1" otherwise) on Python 3.

mail: catch LookupError in headdecode() We already catch this exception in _encode() (called by headencode()). It gets raised when running test-notify.t with Python 3.

py3: account for extra line break in email headers in test-notify.t Long headers appears to be wrapped with new lines. In test-notify.t, we have a "filter.py" that replaces "\n" by " ", so we get an extra space in a Message-Id with a long value.

py3: use as_bytes() method of EmailMessage In Python 3, as_bytes() corresponds to as_string() in Python 2.

py3: use %d instead of %s when formatting an int into a bytestring The latter wasn't noticed before since no tests exercise --confirm at all, let alone on an existing DREV. The former is only hit during an intermittent network issue during amending at the end of a phabsend, so doesn't seem testable. Differential Revision: https://phab.mercurial-scm.org/D7153

packaging: ship only a single binary Debian package We merge the mercurial and mercurial-common binary packages into a single mercurial package. This is essentially to ease installation (and upgrade) using a simple "dpkg -i" command. This also simplifies debian/rules by removing arch (in)dependent cleanups during installation. We have the mercurial binary Breaks: and Replaces: mercurial-common so that the latter will be removed upon upgrade. Also note the change from "override_dh_install" to "override_dh_auto_install" in debian/rules: this is because we do not want "make install" to be run automatically as we need the --install-layout=deb of "setup.py install" (otherwise, files would end up in $DESTDIR/usr/local).

packaging: avoid running bare "make install" in debian/rules We change the "override_dh_install" target to "override_dh_auto_install" in debian/rules (see dh_auto_install(1) for details). This is because we do not want "make install" to be run automatically as we need the --install-layout=deb of "setup.py install". Otherwise, files end up in $DESTDIR/usr/local. At the moment, this is not a problem since files installed in debian/tmp (the default location) are not packed into binary packages (because there are debian/mercurial and debian/mercurial-common directories). This is cleaner to avoid running make more than needed.

packaging: distinguish clean and build steps from install in Debian

packaging: also move Debian .buildinfo file in output directory

packaging: upgrade packages distributed with Windows installers We like to use the latest versions of things. I added pywin32-ctypes to the explicit list of packages so running on !Windows will pull in the dependency.

automation: install Python 2.7.17, 3.7.5, and PyPy 7.2.0 These were all recently released and we should use them in automation.

contrib: install Python 2.7.17 and 3.7.5 in Windows environment These are the latest Python versions and we should use them.

packaging: use /usr/bin/python3 shebang for scripts in Debian "hg" script is already correct because it is handled by setup.py but "hg-ssh" will be rewritten by dh_python into "/usr/bin/python" which is not wanted as we target Python 3. By passing --shebang=/usr/bin/python3 to dh_python3, we force shebangs to be set with this value.

packaging: fix path where .deb files are looked for In builddeb script, the path where .deb files are looked for should simply be $OUTPUTDIR since the previous instruction moves those files there. This fixes "make deb".

packaging: let Debian package depends on sensible-utils This fixes the following Lintian error: E: mercurial-common: missing-depends-on-sensible-utils etc/mercurial/hgrc.d/default-tools.rc

packaging: upgrade Debian packaging to build with Python 3 Also drop the explicit "Depends: python" as debhelper will add it.

py3: decode bytes before logging in run-tests.py Avoids messages like "Found prerequisite b'diff' at b'/usr/bin/diff'" under Python 3.

Added signature for changeset 59338f956109

Added tag 5.2rc0 for changeset 59338f956109

merge default into stable for 5.2 release

grep: update docs to reflect new --all-files default These docs no longer even mention the old weird behaviour that was poorly understood and underutilised. I think this undocumentation effectively deprecates it.

grep: enable all-files by default (BC) This patch deprecates the `--all-files` flag and make the all-files behaviour as default. In test-grep.t, I removed '--all-files' from every command where it was used, to reflect that all-files behaviour is default and there is no change even after the removal. And other changes in test files are because of changed behaviour. Differential Revision: https://phab.mercurial-scm.org/D7000 .. bc:: `hg grep` now searches working copy file contents by default. We recognize this is a significant change from past behavior, but surveys of large bodies of users indicated nobody used (and almost nobody understood) the previous no-flags behavior of `hg grep`. The new behavior aligns with the behavior most users expected (including hg's maintainers), which also happens to be the behavior of `git grep`. Given that the old behavior was confusing to the point of being unusable, we were comfortable changing this behavior.

lfs: dedent documentation section about .hglfs file The leading ".hglfs::" is interpreted as a macro in generated man page and, as it is unknown, the whole section does not render. Also, having the section marked as preformatted (::) makes it render verbatim in HTML, which is not desired as the text contains formatting markers. So we just dedent the section and remove the ".hglfs::" line. The example file is still indented and rendered preformatted.

copies: simplify chain loop This is a mostly gratuitous change to make the code more direct. Differential Revision: https://phab.mercurial-scm.org/D7131

sidedatacopies: only fetch information once for merge Before this change, merge would result in reading the data from revlog twice. With this change, we keep the information in memory until we encounter the other parent. When looking at pypy, I see about 1/3 of the changesets with copy information being merge. Not doing duplicated fetch for them provide a significant speedup. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 0.767042 comb 0.760000 user 0.750000 sys 0.010000 (median of 11) after: ! wall 0.671162 comb 0.670000 user 0.650000 sys 0.020000 (median of 13) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.170169 comb 1.170000 user 1.130000 sys 0.040000 (median of 10) after: ! wall 1.030596 comb 1.040000 user 1.010000 sys 0.030000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.209846 comb 0.200000 user 0.200000 sys 0.000000 (median of 46) after: ! wall 0.170981 comb 0.170000 user 0.170000 sys 0.000000 (median of 56) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013248 comb 0.010000 user 0.010000 sys 0.000000 (median of 223) after: ! wall 0.013295 comb 0.020000 user 0.020000 sys 0.000000 (median of 222) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001672 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001666 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8010) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8007) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.168599 comb 0.160000 user 0.160000 sys 0.000000 (median of 58) after: ! wall 0.133316 comb 0.140000 user 0.140000 sys 0.000000 (median of 73) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.036052 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) after: ! wall 0.032558 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7127

copies: move from a copy on branchpoint to a copy on write approach Before this changes, any branch points results in a copy of the dictionary containing the copy information. This can be very costly for branchy history with few rename information. Instead, we take a "copy on write" approach. Copying the input data only when we are about to update them. In practice we where already doing the copying in half of these case (because `_chain` makes a copy), so we don't add a significant cost here even in the linear case. However the speed up in branchy case is very significant. Here are some timing on the pypy repository. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 1.399863 comb 1.400000 user 1.370000 sys 0.030000 (median of 10) after: ! wall 0.766453 comb 0.770000 user 0.750000 sys 0.020000 (median of 11) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.876748 comb 1.890000 user 1.870000 sys 0.020000 (median of 10) after: ! wall 1.167223 comb 1.170000 user 1.150000 sys 0.020000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.242457 comb 0.240000 user 0.240000 sys 0.000000 (median of 39) after: ! wall 0.211476 comb 0.210000 user 0.210000 sys 0.000000 (median of 45) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013193 comb 0.020000 user 0.020000 sys 0.000000 (median of 224) after: ! wall 0.013290 comb 0.010000 user 0.010000 sys 0.000000 (median of 222) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001673 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001677 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8023) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 7997) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.201898 comb 0.210000 user 0.200000 sys 0.010000 (median of 48) after: ! wall 0.167415 comb 0.170000 user 0.160000 sys 0.010000 (median of 58) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.036820 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) after: ! wall 0.035797 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) The extra cost in the linear case can be reclaimed later with some extra logic. Differential Revision: https://phab.mercurial-scm.org/D7124

copies: compute the exact set of revision to walk This change make the code clearer by removing the revision queue. It comes without very noticeable performance impact. However the simpler code will be easier to update in later changesets. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 1.430082 comb 1.430000 user 1.390000 sys 0.040000 (median of 10) after: ! wall 1.405192 comb 1.410000 user 1.390000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.971366 comb 1.970000 user 1.950000 sys 0.020000 (median of 10) after: ! wall 1.892541 comb 1.890000 user 1.870000 sys 0.020000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.252594 comb 0.250000 user 0.240000 sys 0.010000 (median of 38) after: ! wall 0.240075 comb 0.240000 user 0.240000 sys 0.000000 (median of 40) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013100 comb 0.010000 user 0.010000 sys 0.000000 (median of 226) after: ! wall 0.013247 comb 0.010000 user 0.010000 sys 0.000000 (median of 223) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001633 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001670 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000078 comb 0.000000 user 0.000000 sys 0.000000 (median of 11984) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 7982) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.207093 comb 0.210000 user 0.210000 sys 0.000000 (median of 47) after: ! wall 0.201551 comb 0.200000 user 0.200000 sys 0.000000 (median of 48) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.038462 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) after: ! wall 0.036578 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7076

copies: add an explicit test using multiple roots We explicitly add a test checking for copies of connected revision but involving a different roots. This make sure the copies coming purely from the new roots does not interfere with the merge. Differential Revision: https://phab.mercurial-scm.org/D7075

sidedatacopies: fast path data fetching if revision has no sidedata When using the side data mode, we know their won't be any copy information sidedata. Skipping revision restoration give an important speed boost. In the future, there will be other user of sidedata, reducing the efficiency of this. We should consider adding a dedicated flag in revlog V2 to preserve this optimisation. The current situation is good enough for now. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 2.401569 comb 2.400000 user 2.390000 sys 0.010000 (median of 10) after: ! wall 1.429294 comb 1.430000 user 1.410000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 3.519140 comb 3.520000 user 3.470000 sys 0.050000 (median of 10) after: ! wall 1.963332 comb 1.960000 user 1.960000 sys 0.000000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.593880 comb 0.600000 user 0.590000 sys 0.010000 (median of 15) after: ! wall 0.251679 comb 0.250000 user 0.250000 sys 0.000000 (median of 38) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013414 comb 0.020000 user 0.020000 sys 0.000000 (median of 220) after: ! wall 0.013222 comb 0.020000 user 0.020000 sys 0.000000 (median of 223) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.002711 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001631 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000077 comb 0.000000 user 0.000000 sys 0.000000 (median of 12208) after: ! wall 0.000078 comb 0.000000 user 0.000000 sys 0.000000 (median of 12012) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.410067 comb 0.410000 user 0.410000 sys 0.000000 (median of 23) after: ! wall 0.207786 comb 0.200000 user 0.200000 sys 0.000000 (median of 46) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.097004 comb 0.090000 user 0.090000 sys 0.000000 (median of 100) after: ! wall 0.038495 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7074

largefiles: use context manager for setting "lfstatus" on subrepos too It's safe since we now reset to the previous value. Differential Revision: https://phab.mercurial-scm.org/D7138

largefiles: reset "lfstatus" attribute to previous value in "finally" We were resetting it to False, which means nesting of these overrides would be incorrect. Differential Revision: https://phab.mercurial-scm.org/D7137

largefiles: add context manager for setting/clearing "lfstatus" attribute Differential Revision: https://phab.mercurial-scm.org/D7136

py3: adjust expected traceback in test-hook.t In Python 3, traceback.format_exception() displays the chain of exceptions so we get extra results from our grep. Also, ModuleNotFoundError is raised instead of ImportError from Python 3.6.

tests: use non-reverse grep in traceback in test-hook.t This will prepare for updating test output for Python 3.

py3: add Python 3 exception output to test-hook.t This is similar to, e.g., 3e9c6cef949b.

py3: ajust abort message in test-hook.t

rust-cpython: prepare for writing tests that require libpython What I wanted is to disable the "cpython/extension-module<ver>" feature while building tests executable, but that seems not doable. Instead, this patch adds new features dedicated for tests. The make rule is extracted so that we can easily run cargo tests. Added a minimal test of ref-sharing as an example. More tests will follow.

rust-cpython: make inner functions and structs of ref_sharing private Most of these methods were public because they had to be accessible from macro-generated functions. Some "unsafe" can be removed since we can guarantee the data consistency across non-public operations.

rust-cpython: keep Python<'a> token in PyRefMut This just clarifies that the GIL is obtained while PyRefMut is dereferenced, so there's no need of extra acquire_gil() to drop the reference.

rust-cpython: require GIL to borrow immutable reference from PySharedRefCell Since the inner value may be leaked, we probably need GIL to guarantee that there's no data race. inner(py).borrow() is replaced with inner_shared(py).borrow(), which basically means any PySharedRefCell data should be accessed through PySharedRef wrapper.

rust-cpython: make PyLeakedRef operations relatively safe This patch encapsulates the access to the leaked reference to make most leaked-ref operations safe. The only exception is leaked_ref.map(). I couldn't figure out how to allow arbitrary map operation safely over an unsafe static reference. See the docstring and inline comment for details. Now leak_immutable() can be safely implemented as the PyLeakedRef owns its inner data.

rust-cpython: put leaked reference in PyLeakedRef The next patch will make PyLeakedRef manage the lifetime of the underlying object. leak_handle.data.take() will be removed soon.

run-tests: make code coverage work on Python 3 This code path was obviously not tested on Python 3 because it blew up in several places due to str/bytes mismatch. For internal code, we normalize paths to bytes. For code calling into `coverage`, we normalize paths to str, which is what `coverage` seems to expect. After this, `run-tests.py -H` works on Python 3! Differential Revision: https://phab.mercurial-scm.org/D7133

py3: don't index into bytes in phabricator's _tokenize() `phabread`ing a stack using `hg phabread :D1234` under py3 will currently die with a KeyError because it will index into `b':D1234'` and return `58` instead of `b':'` as a token. Differential Revision: https://phab.mercurial-scm.org/D7129

rust-dirstate-status: use fast-path even with fsmonitor and sparse extensions When I initially ran the tests on my series, there were test failures related to those extensions. Now that the initial series has landed, I felt like going back to those issues because people with performance issues will often want to use fsmonitor and sparse. Either because of flaky tests or because the series has changed so much, I can't seem to reproduce these issues... let's widen the scope of the fast-path. Differential Revision: https://phab.mercurial-scm.org/D7128

rust-dirstatemap: remove additional lookups in dirstatemap We're copying this shortcut from the Python implementation, pretty standard for this codebase. Differential Revision: https://phab.mercurial-scm.org/D7117

tests: avoid chmod on windows in hgrc tests Follow up on changeset d201a637c971 introducing this test, which fails on Windows.

py3: fix sorting of obsolete markers during push This fixes similar errors as in 01e8eefd9434: TypeError: '<' not supported between instances of 'NoneType' and 'tuple'

exchange: extract a function to sort obsolete markers Follows up on changeset 01e8eefd9434, several other occurrences of sorted() need to be fixed. The _sortedmarkers() handles sorting obsmarkers with a None value as last element on Python 3.

py3: encode underlying error message during parse error of %include

convert: convert os.devnull to bytes before trying to join it with other bytes Together with the previous commit relating to emailparser this gets test-convert-tla.t passing under py3. Differential Revision: https://phab.mercurial-scm.org/D7062

rust-dirstate-status: add call to rust-fast path for `dirstate.status` The reasoning behind this patch is explained in the first patch of the series. Differential Revision: https://phab.mercurial-scm.org/D7060

rust-dirstate-status: rust-cpython bindings for `dirstate.status` The ref-sharing mechanism has improved, but its ergonomics still left a bit to be desired, as expected. Differential Revision: https://phab.mercurial-scm.org/D7059

rust-refsharing: add missing lifetime parameter in ref_sharing Differential Revision: https://phab.mercurial-scm.org/D7110

rust-dirstate-status: add first Rust implementation of `dirstate.status` Note: This patch also added the rayon crate as a Cargo dependency. It will help us immensely in making Rust code parallel and easy to maintain. It is a stable, well-known, and supported crate maintained by people on the Rust team. The current `dirstate.status` method has grown over the years through bug reports and new features to the point where it got too big and too complex. This series does not yet improve the logic, but adds a Rust fast-path to speed up certain cases. Tested on mozilla-try-2019-02-18 with zstd compression: - `hg diff` on an empty working copy: - c: 1.64(+-)0.04s - rust+c before this change: 2.84(+-)0.1s - rust+c: 849(+-)40ms - `hg commit` when creating a file: - c: 5.960s - rust+c before this change: 5.828s - rust+c: 4.668s - `hg commit` when updating a file: - c: 4.866s - rust+c before this change: 4.371s - rust+c: 3.855s - `hg status -mard` - c: 1.82(+-)0.04s - rust+c before this change: 2.64(+-)0.1s - rust+c: 896(+-)30ms The numbers are clear: the current Rust `dirstatemap` implementation is super slow, its performance needs to be addressed. This will be done in a future series, immediately after this one, with the goal of getting Rust to be at least to the speed of the Python + C implementation in all cases before the 5.2 freeze. At worse, we gate dirstatemap to only be used in those cases. Cases where the fast-path is not executed: - for commands that need ignore support (`status`, for example) - if subrepos are found (should not be hard to add, but winter is coming) - any other matcher than an `alwaysmatcher`, like patterns, etc. - with extensions like `sparse` and `fsmonitor` The next step after this is to rethink the logic to be closer to Jane Street's Valentin Gatien-Baron's Rust fast-path which does a lot less work when possible. Differential Revision: https://phab.mercurial-scm.org/D7058

share: unmark --relative as EXPERIMENTAL Differential Revision: https://phab.mercurial-scm.org/D7001

rust-cpython: removed now useless py_set() conversion In rust-cpython 0.3.0, HashSets implement the appropriate ToPythonObject, we can therefore get rid of this hacky conversion. There still remains an inefficiency in `MissingAncestors.bases()`: we have to clone, because `to_py_object()` requires full ownership. However: - the only use case outside of unit tests used to be from `setdiscovery.partialdiscovery` which is now fully implemented in Rust. - it's not worse than what `py_set()` used to do Differential Revision: https://phab.mercurial-scm.org/D7120

tests: stabilize test-eol-update on Windows These appeared in the recent eol fix series. I'm not sure about the identity mismatch warning- it seems like a general timing issue, because it happened on the first run but not when run with `--loop`. The buildbot also flagged this. Differential Revision: https://phab.mercurial-scm.org/D7112

tests: opt into python3 when testing virtualenv installation Otherwise it complains that Mercurial requires '~=2.7'. Since the existing linux py3 test doesn't flag this, I'm assuming that virtualenv simply isn't installed on that system. Differential Revision: https://phab.mercurial-scm.org/D7111

grep: put --diff in the first line of usage Differential Revision: https://phab.mercurial-scm.org/D7003

ui: option to preserve the progress bar Some interface like the fast-import format of git are meant for pipe processing. It can be still desirable to have progress bars in this case, but clearing the bar after each output operation is not desirable. Differential Revision: https://phab.mercurial-scm.org/D6843

phabricator: update hgmatcher to cope with the new data format The new conduit format can't be matched by the existing matcher since it shifts all the data into an urlencoded string of JSON, the order of which isn't stable between runs. Instead detect JSON values of params and load them into python dicts, which python will then naturally deep-equal compare. Differential Revision: https://phab.mercurial-scm.org/D7055

phabricator: treat non-utf-8 text files as binary as phabricator requires Phabricator can't cope with text files that are not UTF-8, so requires them to be submitted as binary files instead. This has the unfortunate effect of making them practically unreviewable in Phabricator since it will only display the separate versions of the file in other views, not a diff. `phabread`ing such submissions are similar, since it will just output the binary patch, but `hg import` copes with it fine and `hg diff` afterwards will show the actual changes. It is still a marked improvement over trying to submit them as text, which just leads to corruption (Phabricator will either output ? or HTML entities for non-UTF-8 characters, depending on context). Running decode on the whole file like this seems slightly unfortunate, but I'm not aware of a better way. Needs to be done to p1() version as well to detect conversions to UTF-8. Differential Revision: https://phab.mercurial-scm.org/D7054

phabricator: switch to the creatediff endpoint This lets the extension submit binary files, as well as set branch info so that it is exposed in the Phabricator interface. Differential Revision: https://phab.mercurial-scm.org/D7053

phabricator: change conduit data format to match arcanist The existing form isn't handled well by Phabricator for certain data types. Differential Revision: https://phab.mercurial-scm.org/D7052

phabricator: remove tests and all recordings The next commit is going to change the format of conduit API requests so none of the VCR recordings will match and all the tests will fail. Differential Revision: https://phab.mercurial-scm.org/D7051

phabricator: convert phabhunk and phabchange keys to bytes when finalising I thought I had included this code already but I'd missed it out. One of the disadvantages of attrs is that all the keys are unicode strings on py3, but we need them to be byte strings. Differential Revision: https://phab.mercurial-scm.org/D7066

phabricator: add addadded function This is the most complicated part of the new code, and is responsible for adding all added files, whether brand new, copied, or moved. This also includes creating the phabchanges for the original files that have been moved or copied from, since they might need their awayPaths and type updating if multiple copies are involved. Differential Revision: https://phab.mercurial-scm.org/D7050

sidedatacopies: directly fetch copies information from sidedata When using the sidedata mode, we don't need a complicated and expensive `context` object. Instead we directly fetch copies information from the sidedata (through a changelogrevision object). More optimisations coming. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 filelog: ! wall 3.679613 comb 3.680000 user 3.580000 sys 0.100000 (median of 3) base: ! wall 8.884369 comb 8.880000 user 8.850000 sys 0.030000 (median of 3) before: ! wall 4.681985 comb 4.680000 user 4.640000 sys 0.040000 (median of 3) after: ! wall 3.955894 comb 3.950000 user 3.940000 sys 0.010000 (median of 3) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 filelog: ! wall 0.003357 comb 0.010000 user 0.010000 sys 0.000000 (median of 781) base: ! wall 12.398524 comb 12.400000 user 12.330000 sys 0.070000 (median of 3) before: ! wall 6.459592 comb 6.470000 user 6.390000 sys 0.080000 (median of 3) after: ! wall 5.505774 comb 5.500000 user 5.410000 sys 0.090000 (median of 3) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 filelog: ! wall 2.754687 comb 2.760000 user 2.650000 sys 0.110000 (median of 4) base: ! wall 1.423166 comb 1.420000 user 1.400000 sys 0.020000 (median of 8) before: ! wall 0.961048 comb 0.960000 user 0.940000 sys 0.020000 (median of 11) after: ! wall 0.882950 comb 0.880000 user 0.880000 sys 0.000000 (median of 11) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f filelog: ! wall 1.552293 comb 1.550000 user 1.510000 sys 0.040000 (median of 6 base: ! wall 0.022662 comb 0.020000 user 0.020000 sys 0.000000 (median of 128) before: ! wall 0.021649 comb 0.020000 user 0.020000 sys 0.000000 (median of 135) after: ! wall 0.020951 comb 0.020000 user 0.020000 sys 0.000000 (median of 141) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 filelog: ! wall 1.500983 comb 1.500000 user 1.420000 sys 0.080000 (median of 7) base: ! wall 0.006956 comb 0.010000 user 0.010000 sys 0.000000 (median of 392) before: ! wall 0.004022 comb 0.000000 user 0.000000 sys 0.000000 (median of 735) after: ! wall 0.003988 comb 0.000000 user 0.000000 sys 0.000000 (median of 736) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e filelog: ! wall 0.011745 comb 0.020000 user 0.020000 sys 0.000000 (median of 250) base: ! wall 0.000156 comb 0.000000 user 0.000000 sys 0.000000 (median of 17180) before: ! wall 0.000118 comb 0.000000 user 0.000000 sys 0.000000 (median of 19170) after: ! wall 0.000097 comb 0.000000 user 0.000000 sys 0.000000 (median of 27276) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe filelog: ! wall 3.228230 comb 3.230000 user 3.110000 sys 0.120000 (median of 4) base: ! wall 0.997640 comb 1.000000 user 0.980000 sys 0.020000 (median of 10) before: ! wall 0.679500 comb 0.680000 user 0.680000 sys 0.000000 (median of 15) after: ! wall 0.596779 comb 0.600000 user 0.600000 sys 0.000000 (median of 17) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d filelog: ! wall 1.052501 comb 1.060000 user 1.040000 sys 0.020000 (median of 10 base: ! wall 0.214519 comb 0.220000 user 0.220000 sys 0.000000 (median of 45) before: ! wall 0.149675 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) after: ! wall 0.130786 comb 0.130000 user 0.130000 sys 0.000000 (median of 75) Differential Revision: https://phab.mercurial-scm.org/D7072

copies: use an unfiltered repository for the changeset centric algorithm Since the algorithm work form heads to ancestors, we don't need to check filtering for anything but the two entries. Using an unfiltered version is noticeably more efficient. Some timing on the pypy repo: revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 2.717861 comb 2.720000 user 2.700000 sys 0.020000 (median of 10) after: ! wall 2.582204 comb 2.580000 user 2.560000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 4.003146 comb 4.010000 user 3.970000 sys 0.040000 (median of 10) after: ! wall 3.814613 comb 3.810000 user 3.760000 sys 0.050000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.704204 comb 0.700000 user 0.700000 sys 0.000000 (median of 13) after: ! wall 0.657387 comb 0.650000 user 0.640000 sys 0.010000 (best of 14) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013493 comb 0.020000 user 0.020000 sys 0.000000 (median of 219) after: ! wall 0.013523 comb 0.020000 user 0.020000 sys 0.000000 (median of 218) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.003017 comb 0.000000 user 0.000000 sys 0.000000 (median of 985) after: ! wall 0.002876 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000073 comb 0.000000 user 0.000000 sys 0.000000 (median of 12672) after: ! wall 0.000082 comb 0.000000 user 0.000000 sys 0.000000 (median of 11456) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.478061 comb 0.470000 user 0.470000 sys 0.000000 (median of 19) after: ! wall 0.452420 comb 0.450000 user 0.450000 sys 0.000000 (median of 21) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.116015 comb 0.110000 user 0.110000 sys 0.000000 (median of 84) after: ! wall 0.109153 comb 0.100000 user 0.100000 sys 0.000000 (median of 90) Differential Revision: https://phab.mercurial-scm.org/D7123

copies: extract data extraction into a `revinfo` function The function is build once at the beginning of the algorithm and used fetch appropriate information for each revision. This abstracts some implementation details from the main algorithm and will help us to access the data more efficiently in future changesets. Differential Revision: https://phab.mercurial-scm.org/D7070

copies: avoid instancing more changectx to access parent revisions We just need to know the revision numbers of the parents, creating full context is needlessly expensive. This provide a small, but noticeable performance boost. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 2.885636 comb 2.900000 user 2.870000 sys 0.030000 (median of 10) after: ! wall 2.702270 comb 2.710000 user 2.690000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 4.298271 comb 4.290000 user 4.240000 sys 0.050000 (median of 10) after: ! wall 3.976610 comb 3.970000 user 3.920000 sys 0.050000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.773397 comb 0.770000 user 0.770000 sys 0.000000 (median of 11) after: ! wall 0.701634 comb 0.700000 user 0.700000 sys 0.000000 (median of 13) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013585 comb 0.010000 user 0.010000 sys 0.000000 (median of 217) after: ! wall 0.013550 comb 0.010000 user 0.010000 sys 0.000000 (median of 218) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.003202 comb 0.000000 user 0.000000 sys 0.000000 (median of 929) after: ! wall 0.002993 comb 0.010000 user 0.010000 sys 0.000000 (median of 992) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000077 comb 0.000000 user 0.000000 sys 0.000000 (median of 12060) after: ! wall 0.000072 comb 0.000000 user 0.000000 sys 0.000000 (median of 12804) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.510614 comb 0.500000 user 0.500000 sys 0.000000 (median of 18) after: ! wall 0.473681 comb 0.470000 user 0.470000 sys 0.000000 (median of 20) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.126552 comb 0.130000 user 0.130000 sys 0.000000 (median of 77) after: ! wall 0.115240 comb 0.110000 user 0.110000 sys 0.000000 (median of 85) Differential Revision: https://phab.mercurial-scm.org/D7122

copies: get copies information directly from _copies Performance measurement does not show any significant performance movement. This is not surprising since p1copies() code is `self._copies[0]`. Differential Revision: https://phab.mercurial-scm.org/D7121

copies: simplify the handling of merges Instead of stacking copies for both parent on the head, we move copies outside of the heap into a dedicated dictionary. The two side of merge can we merged sooner, making the algorithm simpler. This simplicity reflect in the heap structure and speed up the execution for copies involving a large amount of merges. Here are timing for perfpathcopies of multiple revision pairs. - filelog: timing using filelog (with the introrev condition dropped) - base: this series base - before: the parent of this changeset - after: this changeset revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 filelog: ! wall 3.679613 comb 3.680000 user 3.580000 sys 0.100000 (median of 3) base: ! wall 8.884369 comb 8.880000 user 8.850000 sys 0.030000 (median of 3) before: ! wall 8.443747 comb 8.420000 user 8.410000 sys 0.010000 (median of 3) after: ! wall 4.697917 comb 4.690000 user 4.660000 sys 0.030000 (median of 3) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 filelog: ! wall 0.003357 comb 0.010000 user 0.010000 sys 0.000000 (median of 781) base: ! wall 12.398524 comb 12.400000 user 12.330000 sys 0.070000 (median of 3) before: ! wall 10.852593 comb 10.850000 user 10.800000 sys 0.050000 (median of 3) after: ! wall 6.750832 comb 6.750000 user 6.640000 sys 0.110000 (median of 3) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 filelog: ! wall 2.754687 comb 2.760000 user 2.650000 sys 0.110000 (median of 4) base: ! wall 1.423166 comb 1.420000 user 1.400000 sys 0.020000 (median of 8) before: ! wall 1.068041 comb 1.060000 user 1.050000 sys 0.010000 (median of 10) after: ! wall 1.045916 comb 1.050000 user 1.040000 sys 0.010000 (median of 10) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f filelog: ! wall 1.552293 comb 1.550000 user 1.510000 sys 0.040000 (median of 6 base: ! wall 0.022662 comb 0.020000 user 0.020000 sys 0.000000 (median of 128) before: ! wall 0.021111 comb 0.020000 user 0.020000 sys 0.000000 (median of 139) after: ! wall 0.021577 comb 0.020000 user 0.020000 sys 0.000000 (median of 138) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 filelog: ! wall 1.500983 comb 1.500000 user 1.420000 sys 0.080000 (median of 7) base: ! wall 0.006956 comb 0.010000 user 0.010000 sys 0.000000 (median of 392) before: ! wall 0.004356 comb 0.010000 user 0.010000 sys 0.000000 (median of 675) after: ! wall 0.004329 comb 0.000000 user 0.000000 sys 0.000000 (median of 682) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e filelog: ! wall 0.011745 comb 0.020000 user 0.020000 sys 0.000000 (median of 250) base: ! wall 0.000156 comb 0.000000 user 0.000000 sys 0.000000 (median of 17180) before: ! wall 0.000100 comb 0.000000 user 0.000000 sys 0.000000 (median of 26912) after: ! wall 0.000105 comb 0.000000 user 0.000000 sys 0.000000 (median of 25689) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe filelog: ! wall 3.228230 comb 3.230000 user 3.110000 sys 0.120000 (median of 4) base: ! wall 0.997640 comb 1.000000 user 0.980000 sys 0.020000 (median of 10) before: ! wall 0.778291 comb 0.780000 user 0.780000 sys 0.000000 (median of 13) after: ! wall 0.706594 comb 0.710000 user 0.710000 sys 0.000000 (median of 15) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d filelog: ! wall 1.052501 comb 1.060000 user 1.040000 sys 0.020000 (median of 10 base: ! wall 0.214519 comb 0.220000 user 0.220000 sys 0.000000 (median of 45) before: ! wall 0.160804 comb 0.160000 user 0.160000 sys 0.000000 (median of 62) after: ! wall 0.163736 comb 0.160000 user 0.160000 sys 0.000000 (median of 60) Differential Revision: https://phab.mercurial-scm.org/D7069

rust-utils: introduce a debug util to print the python stack trace Differential Revision: https://phab.mercurial-scm.org/D7057

rust-cross-platform: remove `unimplemented!` to get compile-time errors We should get a compile-time error that there is missing functionality for the targeted platform instead of the program breaking at runtime. Differential Revision: https://phab.mercurial-scm.org/D7107

widening: pass in matchers instead of patterns This matches how it's done for the non-ellipsis case. The oldmatch is not used yet, but it should be used eventually when widening no longer resends manifests and files the client already has. Differential Revision: https://phab.mercurial-scm.org/D7099

widening: trust user to give full "known" set The new narrow_widen wire protocol command is supposed to return data for exactly the revisions that the client requested (in order for it to be strip-free). So we should not add ancestors the client didn't ask for. Differential Revision: https://phab.mercurial-scm.org/D7098

widening: remove "depth" argument since it's always None Differential Revision: https://phab.mercurial-scm.org/D7097

widening: remove unused "heads" argument Differential Revision: https://phab.mercurial-scm.org/D7096

widening: remove pointless code for second changegroup The caller passes in common==heads, so we will never find any nodes to visit in this code and there will therefore never be a second changegroup emitted. Differential Revision: https://phab.mercurial-scm.org/D7095

widening: remove always-true condition in widening code The function is now specifically about widening, so we don't need to check if we're widening. Differential Revision: https://phab.mercurial-scm.org/D7094

narrow: drop server support for widening using the getbundle command (BC) The client still supports widening using the getbundle, which we (Google) still depend on a for a little while more (we've started the migration to the new narrow_widen command, but we're not done yet). Differential Revision: https://phab.mercurial-scm.org/D7093

widening: duplicate generateellipsesbundle2() for widening The widening and the non-widening code are quite different. It will be clearer to have them as sepearate functions. To start with, I've just copied it exactly, so it's clearer over the next few patches how they're different. The new function should gradually become more similar to bundle2.widen_bundle(), and should perhaps eventually be merged with that function. However, I've left it in narrowbundle2.py for now since it still depends on constants like _KILLNODESIGNAL there. Differential Revision: https://phab.mercurial-scm.org/D7092

tests: show graph log before and after widening in more cases I'm about to make some changes to which revisions get sent during widening. Some more tests will make it clearer what changes. I've also switched to graph log so we easily notice if the graph shape changes. Differential Revision: https://phab.mercurial-scm.org/D7091

tracing: add a couple of trace points on obsolete and repoview I was seeing weird stalls in some fast commands, and these trace events help explain them. Differential Revision: https://phab.mercurial-scm.org/D6550

dirstate: add some traces on listdir calls I'm starting to spend some time trying to make `hg status` do less work, and knowing what's happening in here and what's expensive is an important start. Differential Revision: https://phab.mercurial-scm.org/D7109

cleanup: hgdemandimport.tracing accepts strings, not bytes This does mean that the resulting traces will have some b'' goo in them on Python 3, but I think that's a worthwile price to pay since it doesn't break any tests. Differential Revision: https://phab.mercurial-scm.org/D7108

tests: fix formatting test after move of black config Differential Revision: https://phab.mercurial-scm.org/D7102

rust-dependencies: pin the dependencies of hg-core to prevent breakage A new version of the `rand` crate broke two discovery tests. We should upgrade our dependencies periodically manually. Differential Revision: https://phab.mercurial-scm.org/D7115

automation: install Python 3.8.0 Python 3.8 has been released. Let's install it instead of the RC / development version. Differential Revision: https://phab.mercurial-scm.org/D7114

automation: schedule an EC2Launch run on next boot Without this, launching EC2 instances constructed from the AMI won't go through the normal EC2 instance launch machinery. This missing machinery does important things like set up network routes to use the instance metadata service and process any UserData. Since EC2Launch now runs on subsequent boots and UserData is processed, we needed to make setting of UserData conditional on bootstrapping mode. Differential Revision: https://phab.mercurial-scm.org/D7113

dirs: give formatting oversight to clang-format Differential Revision: https://phab.mercurial-scm.org/D7104

dirs: tag a struct as not being formattable Differential Revision: https://phab.mercurial-scm.org/D7103

sidedatacopies: only store an entry if it has values This will make for a shorter storage and help use to write faster code in simple case. This change already provided a speed boost for copy tracing. For example here is combined time of running copies tracing on a 6989 pairs of revision on the pypy repos: before: 771s after: 631s - 18% This also has a very positive impact on changelog size. For example here are the number for the `00changelog.d` file of pypy. before: 30449712 Bytes after: 24973718 Bytes - 18% This give an overall quite acceptable overhead for storing copies into the changelog: filelog-only: 23370586 sidedata: 24973718 Bytes + 7% Differential Revision: https://phab.mercurial-scm.org/D7068

sidedatacopies: only read from copies when in this mode If we know we expect data from sidedata, we read them from sidedata and nothing else. This avoid looking into extra for revision without sidedata entries. Such revision will be introduced in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D7067

dirs: remove mutable string optimization at all As far as I can see, the optimization trick has been dead since 42e89b87ca79 "dirs: speed up by storing number of direct children per dir". After 42e89b87ca79, the key variable is cleared to NULL at each iteration.

py3: prevent comparison with None in curses histedit (issue6196)

fix: match patterns relative to root I was surprised fixer patterns (used to determine which fixers to run) are applies to the parent directory, not the repo root directory. Danny Hooper (the author of the extension) seemed to agree that it's better to apply them to the repo root, so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D7101

tests: add test showing that fixer patterns are currently relative to $PWD Differential Revision: https://phab.mercurial-scm.org/D7100

black: also ignore grey.py Differential Revision: https://phab.mercurial-scm.org/D7090

black: use multiline exclude definition Our excludes list is pretty complicated, so let's give ourselves some readability. Black treats multiline exclude definitions as verbse regular expressions, so we can split this up and make it easier to consume. Differential Revision: https://phab.mercurial-scm.org/D7089

black: move remaining config knobs into toml file Differential Revision: https://phab.mercurial-scm.org/D7088

black: rename pyproject.toml to black.toml Black won't read this automatically (you'll have to specify --config), but having a pyproject.toml *at all* puts pip in PEP 517/518 mode which breaks us for obscure reasons I don't understand. Rather than waste a ton of time fighting with pip, let's just do this. Differential Revision: https://phab.mercurial-scm.org/D7087

fix: remove a never-true check for unset pattern in Fixer.affects() We don't create an instance of a Fixer if the pattern is None, so we don't need to check that in affects(). Differential Revision: https://phab.mercurial-scm.org/D7086

fix: warn when a fixer doesn't have a configured command It seems we currently produce an empty command line and then decide to not run it, but it seems better to skip that part too. Differential Revision: https://phab.mercurial-scm.org/D7085

fix: make Fixer initialization more explicit for clarity I found it quite confusing that Fixer accessed fields that seemed like they didn't exist. Differential Revision: https://phab.mercurial-scm.org/D7084

fix: don't pass in default value when looking up config The defaults are now registered correctly so we don't need to pass in the default value when we look up a config value. Differential Revision: https://phab.mercurial-scm.org/D7083

fix: fix registration of config item defaults Before this patch, because of the "(:<name>)?", all registered patterns would match and the default value would not be the one we thought we had registered (maybe it just took the default value for the first match?). This didn't matter because we didn't care about the default value; we used our own, intended default value in getfixers() anyway. We also have to look up each config item individually in order to not get developer warnings. Differential Revision: https://phab.mercurial-scm.org/D7082

automation: capture additional exception when formatting I started encountering this for some reason. Differential Revision: https://phab.mercurial-scm.org/D7078

contrib: update to latest Windows package versions Let's keep modern. Differential Revision: https://phab.mercurial-scm.org/D7077

rust-cpython: drop direct dependency on python(27|3)_sys We no longer use it.

rust-cpython: leverage upstreamed py_capsule_fn!() macro

rust-cpython: bump cpython crates to 0.3 Unblocks py_capsule_fn!().

rust-cpython: turn inline comments into non-doc comments

rust-cpython: fix signature of make_dirstate_tuple() Fortunately, the layout of PyObject {} is compatible with a C pointer, but we shouldn't rely on that. This also fixes the handling of NULL pointer.

rust-cpython: mark capsule function as unsafe

rust-cpython: add wrapper around decapsule_make_dirstate_tuple() There are a couple of safety issues. First, the returned function pointer must be unsafe. Second, its return value must be a raw pointer (i.e. python27_sys::PyObject), not a cpython::PyObject. The wrapper function will address these issues.

eol: don't fallback to use .hgeol from tip (BC) If no .hgeol were found in the current working directory, eol would fallback to use the one in tip. That could in some cases give very confusing or wrong behaviour when it applied wrong filters. It might be convenient to have plain 'clone' immediately apply 'native' encoding patterns in the cloned repo. But it is wrong to assume that this revision is tip, and even more wrong to also apply it when not cloning - for example when updating between history revisions. The encoding should always match the content of the current .hgeol . It should never use anything else.

eol: tweak test-eol-clone.t with better descriptions and logging Expose impact of changes coming next ...

eol: fix update - don't use and apply removed .hgeol patterns 'hg up -C' to revisions with different .hgeol patterns could leave dirty changes in the working directory. That could make deployment of new .hgeol filters tricky: they would "occasionally" apply also in branches where they shouldn't. Fixed by dropping all old patterns before applying new ones.