bookmarks: use cmdutil.check_at_most_one_arg() for action Differential Revision: https://phab.mercurial-scm.org/D7647

rebase: use cmdutil.check_at_most_one_arg() for action Here we also needed to know what the action was (if any), so I've updated the helper to return any specified option. Differential Revision: https://phab.mercurial-scm.org/D7640

releasenotes: extract helper for checking for incompatible arguments This patch extracts a new check_incompatible_arguments() function similar to check_at_most_one_arg(). The difference is that the new function is for checking for arguments that are disallowed together with some other argument but not mutually exclusive among themselves. Differential Revision: https://phab.mercurial-scm.org/D7639

fix: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7638

patchbomb: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7637

export: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7636

amend: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7635

commit: use cmdutil.check_at_most_one_arg() Differential Revision: https://phab.mercurial-scm.org/D7634

clone: extract helper for checking mutually exclusive args We have some duplicated code for aborting if the user provided mutually exclusive arguments. Extensions surely have more such code. We also have duplicated translations and inconsistent output in this area. This patch introduces a simpler helper for checking if more than one option among a given set was given on the command line. I've made the clone code call the function to show that it works. The function has no good way of checking arguments with hyphens in them. I'll add that later if necessary. The function still won't be applicable in all cases, but I think it's still better than nothing. Differential Revision: https://phab.mercurial-scm.org/D7633

dirstate: when calling rebuild(), avoid some N^2 codepaths I had a user repo with 200k files in it. Calling `hg debugrebuilddirstate` took tens of minutes (I didn't wait for it). In that situation, changedfiles==allfiles, and both are lists. This meant that we had to run an average of 100k comparisons, for each of 200k files, just to check whether a file needed to have normallookup called (it always did), or drop. While it's probably not a huge issue, in my very awkward synthetic benchmark I wrote (not using a benchmark library or anything), I was seeing some slowdowns for small-changedfiles and very-large-allfiles invocations, with an inflection somewhere around 10 items in changedfiles (regardless of the size of allfiles); above 10 items in changedfiles, the new code appears to always be faster. For the case of 50k files in changedfiles and the same items in allfiles, I'm seeing differences of 15s of just running comparisons vs. 0.003793s. I haven't bothered to run a comparison of 200k items in changedfiles and allfiles. :) Differential Revision: https://phab.mercurial-scm.org/D7665

rust-warnings: fix warnings in tests It turns out that I also missed those warnings inside tests. This should be the last of them. One day we will get rid of this interface anyway. Differential Revision: https://phab.mercurial-scm.org/D7678

relnotes: mention the merging of index and nodemap

crecord: remove toggleamend Previous commit removed its only calling site.

crecord: repurpose "a" key to toggle all selections (BC) I really don't like "a". I keep accidentally hitting it when I actually want "A", and then I'm suddenly in a state I don't want to be in. There's a big wall of text telling me that I've turned amend mode on or off (which one was I orginally in?), and this seems very useless. If I wanted to amend or not, I would have chosen that from the command-line, not change my mind after I've already started picking hunks apart. Furthermore, for most uses of the hunk selector (revert, uncommit, shelve/unshelve), this amend toggle doesn't make sense. It seems much better to repurpose this key to be a "weaker" version of "A". It toggles all selections. This is pretty harmless if hit accidentally, (can just hit "a" again to toggle everything and undo it), and has immediate visual feedback that something happened: all the x's and blank spaces get switched around. And unlike with amend, the current flipped state is also immediately visible without having to read a wall of text. I'm calling this a BC, however, because somewhere, someone out there has probably really fallen in love with the old use of "a" and will get angry that we took it away.

hgweb: fix error in docstring Despite the subtle semantic difference, this sentence really meant to say "overridden", not "overwritten".

crecord: rewrite help string to avoid mentioning "crecord" Despite its heritage, "crecord" is now mostly Mercurial-internal jargon. I find it better to call it "the curses hunk selector". Also slightly rewrote the part about which commands can use it. While I do believe that commit, shelve, and revert are the only commands in core that can use it, Evolve also adds at least amend and uncommit to the list.

update: add some tests for the status quo of morestatus on update conflicts Differential Revision: https://phab.mercurial-scm.org/D7666

typing: suppress a warning that mercurial.windows.checkosfilename is missing This function is used to check filename portability everwhere, so it isn't tucked into the windows.py module. I supposed the alternative is to move it and then alias it in `util`. I'm guessing it was done like this to save an import. Differential Revision: https://phab.mercurial-scm.org/D7677

typing: add an assertion to util.versiontuple Fixes the following warning: line 1177, in versiontuple: No attribute 'split' on None [attribute-error] In Optional[bytes] Differential Revision: https://phab.mercurial-scm.org/D7676

typing: add a couple of assertions to lrucachedict to help pytype Fixes the following warnings: line 1488, in pop: No attribute 'markempty' on None [attribute-error] In Optional[Union[Any, _lrucachenode, nothing]] line 1578, in popoldest: No attribute 'markempty' on None [attribute-error] In Optional[Union[Any, _lrucachenode, nothing]] Differential Revision: https://phab.mercurial-scm.org/D7674

util: rename a variable to avoid confusing pytype Fixes the following warning: line 1205, in f: No attribute 'append' on Dict[nothing, nothing] [attribute-error] In Union[Dict[nothing, nothing], list] Differential Revision: https://phab.mercurial-scm.org/D7673

pytype: suppress the import-error in util.py when importing re2 Differential Revision: https://phab.mercurial-scm.org/D7672

debugcommands: finish moving `extendeddateformats` from util to dateutil Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7671

ui: convert exception data to bytes when printing chained exception info Caught by pytype. Differential Revision: https://phab.mercurial-scm.org/D7670

ui: use a context manager to handle file streams in edit() Differential Revision: https://phab.mercurial-scm.org/D7669

graft: reuse cmdutl.resolvecommitoptions() Differential Revision: https://phab.mercurial-scm.org/D7632

rebase: make rebase not crash if p1 == p2 != nullid While this shouldn't happen normally, some historical bugs can have caused this kind of commit to exist. Instead of crashing and having it be unobvious how to recover, let's try to continue on. Without this, we get an error like "ValueError: min() arg is an empty sequence" Differential Revision: https://phab.mercurial-scm.org/D7664

merge: add commands.merge.require-rev to require an argument to hg merge This is related to commands.rebase.requiredest, commands.update.requiredest, and commands.push.require-revs. Since it isn't really a "destination", I went with require-rev to be similar to push's require-revs. Differential Revision: https://phab.mercurial-scm.org/D7620

tests: make test-config-env.py a little less hacky Overriding os.path.isdir to always returns False is very confusing. Let's make it more explicit by specifically not returning any defaultrc/ configs. Differential Revision: https://phab.mercurial-scm.org/D7623

help: get helptext/ data from `resources` module if available For PyOxidizer, we need to read configs using the `resources` module. This patch makes it so we use that module if available (i.e. Python >= 3.7). It does that by adding a new `open_resource()` function to our `resourceutil` module. Tested by running `$PYTHON ./hg help pager` for each $PYTHON in {python2, python3.6, python3.7}. Differential Revision: https://phab.mercurial-scm.org/D7622

pycompat: allow pycompat.sysbytes() even if input already is bytes pycompat.sysstr() on py3 accepts an input that's already str (i.e. unicode). This patch makes it so pycompat.sysbytes() on py3 accepts an input that's already bytes. Allowing that makes it possible to do pycompat.sysbytes(fp.name) where fp.name is either bytes or unicode, as we'll get when fp can come from either open() or resources.open_binary(). Differential Revision: https://phab.mercurial-scm.org/D7621

merge with stable

rust-dirs: handle forgotten `Result`s In 1fe2e574616e I introduced a temporary bugfix to align Rust code with a new behavior from C/Python and forgot about a few `Result`s (cargo's compiler cache does not re-emit warnings on cached modules). This fixes it. For the record, I am still unsure that this behavior change is a good idea. Note: I was already quite unhappy with the setters and getters for the `DirstateMap` and, indirectly, `Dirs`, and this only further reinforces my feelings. I hope we can one day fix that situation at the type level; Georges Racinet and I were just talking about devising a POC for using the builder pattern in the context of FFI with Python, we'll see what comes out of it. Differential Revision: https://phab.mercurial-scm.org/D7609

merge with stable

status: outputting structured unfinished-operation information This adds a new item in the json/template output for morestatus and added item types to all entries. Differential Revision: https://phab.mercurial-scm.org/D7595

chg: fix chg to work with py3.7+ "coercing" the locale When the environment is empty (specifically: it doesn't contain LC_ALL, LC_CTYPE, or LANG), Python will "coerce" the locale environment variables to be a UTF-8 capable one. It sets LC_CTYPE in the environment, and this breaks chg, since chg operates by: - start hg, using whatever environment the user has when chg starts - hg stores a hash of this "original" environment, but python has already set LC_CTYPE even though the user doesn't have it in their environment - chg calls setenv over the commandserver. This clears the environment inside of hg and sets it to be exactly what the environment in chg is (without LC_CTYPE). - chg calls validate to ensure that the environment hg is using (after the setenv call) is the one that the chg process has - if not, it is assumed the user changed their environment and we should use a different server. This will *never* be true in this situation because LC_CTYPE was removed. Differential Revision: https://phab.mercurial-scm.org/D7550

fuzz: add support for fuzzing under either Python 2 or 3 This was more of a hairball than I hoped, but it appears to work. The hg-py3 branch of my oss-fuzz fork on github has the remaining changes to switch us to Python 3, but we may as well retain Python 2 fuzzing support for at least a little while. Differential Revision: https://phab.mercurial-scm.org/D7592

phabricator: color the status in the "phabstatus" view I couldn't figure out strikethrough for "abandoned" like I've see with word diff. Differential Revision: https://phab.mercurial-scm.org/D7608

phabricator: add the "Changes Planned" status name I noticed this when coloring the status output. But it looks like status names are also used in the pseudo revset language as well, and abort when parsing a name not in this list. Differential Revision: https://phab.mercurial-scm.org/D7607

fuzz: fix mpatch_corpus to not have an overridden __repr__ on py3 Differential Revision: https://phab.mercurial-scm.org/D7606

formatting: fix some recent formatting regressions Differential Revision: https://phab.mercurial-scm.org/D7605

cext-revlog: fixed __delitem__ for uninitialized nodetree This is a bug in a code path that's seldom used, because in practice (at least in the whole test suite), calls to `del index[i:j]` currently just don't happen before the nodetree has been initialized. However, in our current work to replace the nodetree by a Rust implementation, this is of course systematic. In `index_slice_del()`, if the slice start is smaller than `self->length`, the whole of `self->added` has to be cleared. Before this change, the clearing was done only by the call to `index_invalidate_added(self, 0)`, that happens only for initialized nodetrees. Hence the removal was effective only from `start` to `self->length`. The consequence is index corruption, with bogus results in subsequent calls, and in particular errors such as `ValueError("parent out of range")`, due to the fact that parents of entries in `self->added` are now just invalid. This is detected by the rebase tests, under conditions that the nodetree of revlog.c is never initialized. The provided specific test is more direct. Differential Revision: https://phab.mercurial-scm.org/D7603

filemerge: fix a missing attribute usage Flagged by both pytype and VSCode. Differential Revision: https://phab.mercurial-scm.org/D7465

filemerge: drop a default argument to appease pytype The function slices and takes the length of this argument without internally setting it if not provided. There was no bug here because both callers passed the argument. Differential Revision: https://phab.mercurial-scm.org/D7464

fuzz: add a seed corpus for the dirs fuzzer I was hoping to trigger an asan violation under Python 3 that some internal tests at Google found, but for some reason that's beyond me I can't seem to manage. Differential Revision: https://phab.mercurial-scm.org/D7600

fuzz: clean up production of seed corpora This was getting out of hand. Differential Revision: https://phab.mercurial-scm.org/D7599

status: add template/json data about whether a file has unresolved conflicts Differential Revision: https://phab.mercurial-scm.org/D7594

status: split morestatus data loading from display This is a small refactoring in preparation for adding more morestatus functionality (notably for templated/JSON output) - the goal is to use the data inside the status display loop, as well as output the overall state in a templatable/structured way. Differential Revision: https://phab.mercurial-scm.org/D7593

phabricator: add a "phabstatus" template keyword We add a "phabstatus" template keyword, returning an object with "url" and "status" keys. This is quite similar to "phabreview" template keyword, but it queries phabricator for each specified revision so it's going to be slow (as compared to the "phabstatus" show view from previous changeset). Differential Revision: https://phab.mercurial-scm.org/D7507

phabricator: add a "phabstatus" show view We add a "phabstatus" show view (called as "hg show phabstatus") which renders a dag with underway revisions associated with a differential revision and displays their status. The revisions shown is a subset of that shown by "work" view, only including revisions with known by Phabricator. Differential Revision: https://phab.mercurial-scm.org/D7506

dirs: fix out-of-bounds access in Py3 The hack for mutating Python's variable-length integers that was ported to py3 in cb3048746dae (dirs: port PyInt code to work on Python 3, 2016-10-08) was reading from ob_digit[1] instead of ob_digit[0] for some reason. Space for ob_digit[1] would only be allocated for integers larger than 30 bits, so we ended up writing to unallocated memory. Also, we would write an integer that's 2^30 times too large, so we would never free these integers. Found by AddressSanitizer. Differential Revision: https://phab.mercurial-scm.org/D7597

manifestcache: add some test involving shares Some issue around shares have been reported. I am adding tests to better cover the share case. The test show that the code behave as expected so far. Differential Revision: https://phab.mercurial-scm.org/D7602

fuzz: fix test-fuzz-targets.t to run with python3 Differential Revision: https://phab.mercurial-scm.org/D7601

py3: fix a bytes vs str issue in remotefilelog extension Differential Revision: https://phab.mercurial-scm.org/D7598

rust-matchers: add doctests for `AlwaysMatcher` Differential Revision: https://phab.mercurial-scm.org/D7527

rust-hg-path: add method to get part of a path relative to a prefix This will be used in the next patch in this series. Differential Revision: https://phab.mercurial-scm.org/D7526

rust-matchers: improve `Matcher` trait ergonomics `VisitChildrenSet` has no need to own the set, this will save allocations. The `file_set` return type change is motivated by both ergonomics and... being able to compile code. The `AlwaysMatcher` does not store a `file_set`, which requires it to return an owned `HashSet`, which in turn would change our return type to `Cow<&HgPath>` (lifetimes omitted). This is both un-ergonomic and troublesome for more complex lifetime issues (especially with the upcoming `FileMatcher` in the following patch). Differential Revision: https://phab.mercurial-scm.org/D7525

rust-dirs-multiset: use `AsRef` instead of concrete types when possible I also renamed `vec` to `dirstate`, because it was not a great name. Differential Revision: https://phab.mercurial-scm.org/D7524

logcmdutil: call _exthook() in changesettemplater Class changesetprinter has an _exthook() method that is called in _show() before the patch is displayed. Call the method as well in changesettemplater._show(). Differential Revision: https://phab.mercurial-scm.org/D7505

phabricator: fix processing of tags/desc in getoldnodedrevmap() It seems that the previous logic was wrong (it essentially comes from changeset 3ab0d5767b54 where the result got accumulated instead of early returned). First of all, the "continue" in first "if m:" is useless because we're at the end of the loop. Then, the algorithm seems weird because we will process all predecessors of a node and possibly override `toconfirm[node]` for each of these having a tag (maybe this doesn't happen, but still). Finally, we would also override `toconfirm[node]` when the "Differential Revision: " is found in changeset description. Maybe this is not a big deal when there is no mix of local tag and changeset description update? The logic is changed so that the loop on predecessors stops upon first match of a tag and so that the changeset description is only checked if no tag was found. Therefore, `toconfirm[node]` is only set once. Differential Revision: https://phab.mercurial-scm.org/D7513

tests: replace [[]] bashism with portable [] invocation In this case nothing fancy is required. Differential Revision: https://phab.mercurial-scm.org/D7596

rust-hg-path: implement `Display` for `HgPath` and `HgPathBuf` This is useful when debugging, to get a human readable output instead of an array of `u8`. Differential Revision: https://phab.mercurial-scm.org/D7523

rust-performance: introduce FastHashMap type alias for HashMap Rust's default hashing is slow, because it is meant for preventing collision attacks. For all of the current Rust code, we don't care about those attacks, because if an person with bad intentions has write access to your repo, you have other issues. I've chosen to use the TwoXHash crate because it was made by a reputable member of the Rust community and has very good benchmarks. For now it does not seem to improve performance by much for the current code, but it's something else to not worry about when benchmarking code: in a previous experiment with copytracing in Rust, it accounted for more than 10% of the time of the entire script. Differential Revision: https://phab.mercurial-scm.org/D7116

mail: use procutil.shellsplit instead of bytes.split to parse command Differential Revision: https://phab.mercurial-scm.org/D7541

annotate: describe --skip as taking a revset It's obvious to me, but probably wouldn't be obvious to a novice user. Differential Revision: https://phab.mercurial-scm.org/D7557

automation: use latest Windows AMI as base It looks like the previous base image no longer exists. I guess Amazon expires them or something. Let's switch to the newest version of the equivalent image. Differential Revision: https://phab.mercurial-scm.org/D7571

tests: stabilize test-extdiff.t on Windows This goes with 765a9c299c44. I'm not sure if we care about the missing file being spelled `nul`, so I removed it from the glob to be explicit about it. The line needed to be special cased anyway because of the quoting on Windows. Differential Revision: https://phab.mercurial-scm.org/D7572

exchange: fix an attempt to format a list into bytes This goes with 27c6d6f53d46. Differential Revision: https://phab.mercurial-scm.org/D7583

status: remove pointless filtering by alwaysmatcher in morestatus The code has been like this since it was imported from FB's hg-experimental repo. It has been like that even since it was added in that repo. So I don't know why it looks that way. Perhaps the idea was to one day filter the unresolved paths by any patterns provided by the user. We can add a matcher back if we ever decide to do that. Differential Revision: https://phab.mercurial-scm.org/D7591

tests: add test for Rust formatting We enforce formatting for Python and C. It makes sense to do it for Rust as well. Since our rustfmt.toml relies on unstable rustfmt features, we need to use a Nightly rustfmt with --unstable-features in order for it to work. This is a bit hacky and I would prefer we remove this requirement. But for now, this commit assumes this is the way things must be and we go out of our way to detect and use the rustfmt from the "nightly" toolchain, as installed via rustup. We had to add some environment variables to the tests to make the Rust binaries happy. Otherwise when running rustfmt we get an error about no default toolchain being installed. Differential Revision: https://phab.mercurial-scm.org/D7579

rust: run rustfmt # skip-blame automated reformatting Differential Revision: https://phab.mercurial-scm.org/D7578

fuzz: clean up some repetition on building parsers.so fuzzers There was a lot of repetition here that I realized could be cleared out. I think there's some more work we could do here, but this is enough for now. Differential Revision: https://phab.mercurial-scm.org/D7568

fuzz: remove legacy setup for running fuzzers outside oss-fuzz We don't need this anymore. Differential Revision: https://phab.mercurial-scm.org/D7567

tests: finally fix up test-fuzz-targets.t It's been failing on my workstation for a while, since I have a new enough LLVM that I had the fuzzer goo, but not so new that I actually had FuzzedDataProvider. This is a better solution all around in my opinion. I _believe_ this should let us run these tests on most systems, even those using GCC instead of clang. That said, my one attempt to test this on my macOS laptop failed miserably, and I don't feel like doing more work on this right now. Differential Revision: https://phab.mercurial-scm.org/D7566

fuzz: make standalone_fuzz_target_runner call LLVMFuzzerInitialize Otherwise some of our fuzzers crash when they try and use Python. Differential Revision: https://phab.mercurial-scm.org/D7565

fuzz: use a more standard approach to allow local builds of fuzzers This is taken from the (improved since we started fuzzing) guide on ideal integrations. Rather than have our own wonky targets for building outside the fuzzer universe, we have a driver program we carry along and use when we're not using LibFuzzer. This will let us jettison a fair amount of goo. contrib/fuzz/standalone_fuzz_target_runner.cc is https://github.com/google/oss-fuzz/ file projects/example/my-api-repo/standalone from git revision c4579d9358a73ea5dbcc99cb985de1f2bf76dcf7, reformatted with out clang-format settings and a no-check-code comment added. It allows running a single test input through a fuzzer, rather than performing ongoing fuzzing as libfuzzer would. contrib/fuzz/FuzzedDataProvider.h is https://github.com/llvm/llvm-project/ file /compiler-rt/include/fuzzer/FuzzedDataProvider.h from git revision a44ef027ebca1598892ea9b104d6189aeb3bc2f0, reformatted with our clang-format settings and a no-check-code comment added. We can discard this if we instead want to add an hghave check for a new enough llvm that includes FuzzedDataProvder.h in the fuzzer headers. Differential Revision: https://phab.mercurial-scm.org/D7564

fuzz: use a variable to allow specifying python-config to use Eventually we should probably default this to just `python-config` and have the oss-fuzz build.sh script specify the sanpy python-config, but for now this lets us make progress. Differential Revision: https://phab.mercurial-scm.org/D7563

fuzz: suppress deprecated-register warnings in our compile These come from the Python.h headers still using the `register` keyword and our use of C++17. I think this will go away when we're using Python 3 for our fuzzing, but that can come later. Differential Revision: https://phab.mercurial-scm.org/D7562

fuzz: follow modern guidelines and use LIB_FUZZING_ENGINE This will make our lives easier in an upcoming change, but it's also how we're supposed to set things up anyway. Differential Revision: https://phab.mercurial-scm.org/D7561

fuzz: always define LLVMFuzzerInitialize() even if we don't need it This will make it easier to test our fuzzers outside oss-fuzz. Differential Revision: https://phab.mercurial-scm.org/D7560

fuzz: remove debug prints from revlog_corpus.py Differential Revision: https://phab.mercurial-scm.org/D7559

fuzz: fix an unused result on getcwd() in pyutil clang was rightly complaining about this, so let's just fix it. Differential Revision: https://phab.mercurial-scm.org/D7558

upgraderepo: add a config option for parallel computation The option is put to use to compute new copy tracing side data in parallel. It use the multiprocessing module as it had the appropriate primitive for what we needed. Gregory Szorc had concerned on windows so we disabled it there. See inline comment for details on the parallel implementation.

upgrade-repo: colorize some of the output Having clear color for requirement added and removed is useful.

tests: remove hardcoded errno values Fixes test failures on hurd-i386 Ref: https://bugs.debian.org/946178 Differential Revision: https://phab.mercurial-scm.org/D7556

tests: expect return status 255 on exception for test-blackbox.t with chg Differential Revision: https://phab.mercurial-scm.org/D7554

tests: fix command name in test-blackbox.t to be bytes This command is only used when tests are run with --with-chg, so this was missed before. Differential Revision: https://phab.mercurial-scm.org/D7553

tests: fix deprecation warning about regex flags not at beginning of expr This may only show up when running the tests under python3.6+. Currently the only test that does this is test-patchbomb-tls.t, and it only uses (?i), so that's all that's handled at the moment. Differential Revision: https://phab.mercurial-scm.org/D7552

tests: fix test-chg.t to work with py3 (no setprocname) Differential Revision: https://phab.mercurial-scm.org/D7551

amend: check for file modifications when updating dirstate (issue6233) Previously, we called dirstate.normal(f), which would put information into the dirstate claiming that the file on disk is what it "should be" for the current checkout, and it would have the size and timestamp of the most recent modification to the file (which is not necessarily the one we just committed). If the file was modified while the commit message editor was open, we would put incorrect information into the dirstate. Differential Revision: https://phab.mercurial-scm.org/D7521

graft: never set both parents equal in the dirstate (issue6098) merge.graft() can set both parents equal in the dirstate when keepparent=True. We don't seem to set that in core, but the evolve extension does use it. So I couldn't figure out a way to add a test for this patch in core. Differential Revision: https://phab.mercurial-scm.org/D7549

graft: extract `repo['.']` to local variable It's used in two places and I'm about to use it more. Differential Revision: https://phab.mercurial-scm.org/D7548

graft: rename `pctx` argument to `base` since that's what it is The new name better matches the docstring. It also frees up `pctx` to be used for something else (next patch). Differential Revision: https://phab.mercurial-scm.org/D7547

exchange: replace a "not x in ys" by more Pythonic "x not in ys" Found by one of our (Google-)internal tools. Differential Revision: https://phab.mercurial-scm.org/D7546

packaging: include defaultrc/*.rc instead of default.d/*.rc Differential Revision: https://phab.mercurial-scm.org/D7555

merge with stable

py3: make a pycompat.osdevnull, use it in extdiff Differential Revision: https://phab.mercurial-scm.org/D7545

subrepo: use pycompat.open directly instead of importing open from pycompat Differential Revision: https://phab.mercurial-scm.org/D7544

rust-dirs: address failing tests for `dirs` impl with a temporary fix https://phab.mercurial-scm.org/D7252 (5d40317d42b7083b49467502549e25f144888cb3) introduced a regression in Rust tests. This is a temporary fix that replicates the behavior of the C and Python impl, pending the resolution of the discussion (in the phabricator link) about how we actually want to solve this problem. Differential Revision: https://phab.mercurial-scm.org/D7503

cleanup: fix docstring formatting This is just removing the b'' prefix (except demandimportpy2), and making sure it is triple quoted. I skipped the mapping.py module in zope because that's 3rd party code. Differential Revision: https://phab.mercurial-scm.org/D7539

copies: split the combination of the copies mapping in its own function In some case, this part take up to 95% of the copy tracing that take about a hundred second. This poor performance comes from the fact we keep duplciating and merging dictionary that are mostly similar. I want to experiment with smarter native code to do this, so I need to isolate the function first.

copies: do not initialize the dictionary with root in changeset copies We don't strictly need the dict to be prefiled with `{}`. Not doing so will make the next changeset simpler. This part of a preparation to use native code for this part of copy tracing.

copies: expand `_chain` variable name to make the function easier to read This came up while explaining what the function is about. I find the function easier to follow that way.

copies: show we can read data from the "first" mapping In practice this make no difference since `t` is a copy of `a`. Having this change before hand will make the next cleanup changeset clearer. This is work on the road to using some native code for some performance critical part of copy tracing.

copies: return consistent type from revinfo The return is expected to be a list, so we should return a list. This make the use of this function from type-checked language (eg: rust) much simpler.

dateutil: correct default for Ymd in parsedate The code uses `0` for the default value of Ymd (year, month, and day), which seems suboptimal. For example, these will fail to parse: dateutil.parsedate('2000', formats=dateutil.extendeddateformats) dateutil.parsedate('Jan 2000', formats=dateutil.extendeddateformats) Fix it by providing sane defaults (1 instead of 0) for year, month, and day. The suboptimal behavior was introduced by 91bc001a592 (2010-12-29, "date: fix matching of underspecified date ranges"), which does not seem to justify the current behavior. Note end-users should not notice the subtle issue, because there are no formats in `defaultdateformats` that allow an explicit year with omitted month, or an explicit month with omitted day. Differential Revision: https://phab.mercurial-scm.org/D7520

test-doctest: include dateutil `mercurial.utils.dateutil` has dostrings that contain doctests. Include them. Differential Revision: https://phab.mercurial-scm.org/D7519

revlog: fix revset in reachableroots docstring `reachableroots` will only return a subset of `roots` when `includepath` is False. For example, given the following linear DAG: 2 | 1 | 0 Using roots=0+2, heads=1, the definition in the docstring does not match what `reachableroots` actually does: ipdb> repo.changelog.reachableroots(0, roots=[0,2],heads=[1]) [0] ipdb> repo.revs('heads(::(0+2) & (0+2)::1)') <baseset+ [1]> The fix is to do `heads & ::roots` (or `heads & heads::roots`) first, then select their ancestors: ipdb> repo.revs('heads(::((0+2) & (0+2)::1))') <baseset+ [0]> The docstring was introduced by fd92bfbbe02d9 (2015-06-19 "revset: rename revsbetween to reachableroots and add an argument"), which introduced the `includepath=False` behavior for graphlog grandparents use-case. I believe the docstring instead of the code should be changed because changing the code to match the docstring can result in suboptimal graphlog like: o :\ : o : : :/ o As opposite to the current "linearized" graphlog: o | o : o Differential Revision: https://phab.mercurial-scm.org/D7518

lock: pass "success" boolean to _afterlock callbacks This lets the callback decide if it should actually run or not. I suspect that most callbacks (and hooks) *should not* run in this scenario, but I'm trying to not break any existing behavior. `persistmanifestcache`, however, seems actively dangerous to run: we just encountered an exception and the repo is in an unknown state (hopefully a consistent one due to transactions, but this is not 100% guaranteed), and the data we cache may be based on this unknown state. This was observed by our users since we wrap some of the functions that persistmanifestcache calls and it expects that the repo object is in a certain state that we'd set up earlier. If the user hits ctrl-c before we establish that state, we end up crashing there. I'm going to make that extension resilient to this issue, but figured it might be a common issue and should be handled here as well instead of just working around the issue. Differential Revision: https://phab.mercurial-scm.org/D7459

relnotes: add note about changes to match.{explicit,reverse}dir Differential Revision: https://phab.mercurial-scm.org/D7508

graphlog: change state dict to attr struct This should help static analysis.

status: fix default value of status struct The default argument isn't overloaded. Before, the default constructor would create a struct having 7 list type objects.

typing: fix return type of logcmdutil.getrevs() Fixes the following errors: Invalid type annotation "'Tuple[smartset.BaseSet, changesetdiffer]'" [invalid-annotation] No attribute 'BaseSet' on module 'mercurial.smartset' getrevs: bad option in return type [bad-return-type] Expected: Tuple[mercurial.smartset.abstractsmartset, changesetdiffer] Actually returned: Tuple[mercurial.smartset.baseset, None]

typing: consolidate "if not globals():" trick Removes redundant inline comments. I think pycompat is good place to host this kind of constants.

rust-cpython: do not convert warning pattern to utf-8 bytes On Unix, both Rust Path and Mercurial expect a locale-dependent bytes, and we don't support Windows yet.

rust-cpython: import utils::files::* function at module level IIRC, it's common in Rust to call functions with the module prefix.

exchange: guard against method invocation on `b2caps=None` args I couldn't figure out how these are called, but the value is pretty obviously set at least for the cases that have tests. Differential Revision: https://phab.mercurial-scm.org/D7512

exchange: eliminate some bytes.format() calls Differential Revision: https://phab.mercurial-scm.org/D7511

windows: suppress pytype warnings for Windows imports and functions This should allow the modules to not be excluded, and not generate complaints on non Windows platforms. Differential Revision: https://phab.mercurial-scm.org/D7510

changectx: mark the parents of the working copy as non filtered If we successfully accessed the working copy, its parents are not filtered. Differential Revision: https://phab.mercurial-scm.org/D7491

repoview: add an explicit set of all filter that show the wc parents The `visible` set will always show the working copy parents. We record this in a specific set. This will allow to fast path some access. Differential Revision: https://phab.mercurial-scm.org/D7490

localrepo: introduce a `_quick_access_changeid` property Having faster access to `null` is cute… but limited. We want to speedup access to more useful revision, like `.`. We start with turning the fast path for `null` into something more generic. Differential Revision: https://phab.mercurial-scm.org/D7488

changectx: use unfiltered changelog to access parents of unfiltered revs If a revision is not filtered, we know that its parents are not either. So we can take a shortcut. This shortcut avoid the computation of all filtered revs in some cases. Differential Revision: https://phab.mercurial-scm.org/D7487

locarepo: also fastpath `nullid` lookup in __getitem__ We already use that fastpath for `"null"` and `nullrev`, using it for `nullid` is similar. Differential Revision: https://phab.mercurial-scm.org/D7486

repoview: add more tests to track operation not supposed to trigger filtering This test is useful to confirm we removed filtering trigger and to prevent it to come back without us noticing. The commands tested in the test were initially introduced one by one. However, on Martin von Zweigbergk request, we are adding them all at once. Differential Revision: https://phab.mercurial-scm.org/D7514

localrepo: mark nullrev has never filtered All repository have a null, and it cannot be filtered. Differential Revision: https://phab.mercurial-scm.org/D7484

changectx: add a "maybe filtered" filtered attribute There are changeset that we know not to be filtered (eg: `null`). In this case, we could access some information without triggering changelog filtering. We add an attribute to changectx that track this property. Differential Revision: https://phab.mercurial-scm.org/D7483

repoview: add a test that access actual changeset data of `null` Ideally, we would not trigger filtering here. However some work needs to happens first. Differential Revision: https://phab.mercurial-scm.org/D7482

localrepo: recognize trivial "null" queries in `anyrev` Bypassing the revset logic for trivial "null" queries means we can avoid to trigger the filtering logic in some cases. Differential Revision: https://phab.mercurial-scm.org/D7481

localrepo: also fastpath `nullrev` in __getitem__ As explained earlier, nullrev will exist in all repository, we do not need any special checking. Differential Revision: https://phab.mercurial-scm.org/D7480

repoview: add a test to track operation not supposed to trigger filtering This test will be useful to confirm we removed filtering trigger and to prevent it to come back without us noticing. Differential Revision: https://phab.mercurial-scm.org/D7479

repoview: display stack trace along side the debug message When a filter computation is triggered, If --traceback is provided, we will display a traceback in addition to the debug message. Differential Revision: https://phab.mercurial-scm.org/D7478

util: add an optional `prefix` argument to debugstacktrace This is useful when using it in a specific context. Differential Revision: https://phab.mercurial-scm.org/D7477

repoview: add a 'devel.debug.repo-filter' option If set, there will be debug message when a filter computation is triggered. This is going to be useful to remove various filtering trigger and to test they do not get reintroduced. Differential Revision: https://phab.mercurial-scm.org/D7476

localrepo: extract handling of some special value in __getitem__ The value "tip" should always be accessible, otherwise there is some problematic bug in the lower level. So we can access this outside of the general try/catch. If it fails some wider and actual big is in play. Differential Revision: https://phab.mercurial-scm.org/D7475

localrepo: extract handling of some special value in __getitem__ The value "null" will always be present in a repository. So this lookup should always succeed and do not need to be in the general try/catch. Differential Revision: https://phab.mercurial-scm.org/D7474

localrepo: add some basic comment for block in __getitem__ There are different early processing before getting to the core of the function. We highlight that fact. Differential Revision: https://phab.mercurial-scm.org/D7473

windows: further build fixes for the WiX installer With these fixes in place, the .msi actually builds for me again. Differential Revision: https://phab.mercurial-scm.org/D7509

cleanup: update references to /help/ that should now be /helptext/ I noticed this only because the WiX build was broken after 2e017696181f. I haven't tested this change with WiX, but it seems pretty obvious by inspection this should fix it. Differential Revision: https://phab.mercurial-scm.org/D7472

merge with stable

tests: byteify a few more things in simplestorerepo.py Differential Revision: https://phab.mercurial-scm.org/D7471

profiling: add a missing argument to the ProgrammingError constructor Differential Revision: https://phab.mercurial-scm.org/D7470

profiling: disable the import-error warning for the flamegraph module The module being missing is properly handled, but pytype was complaining. Differential Revision: https://phab.mercurial-scm.org/D7469

phases: make `allphases` a list on py3 also Pytype complained that `range` doesn't support slicing, so make this a list like it was on py2. Differential Revision: https://phab.mercurial-scm.org/D7468

wireprotov1server: capture Abort type before accessing the `hint` attribute Although the previous code worked, pytypes complained because `exc` is caught above as `BundleValueError, Abort, PushRaced`, and the other two don't have this attribute. Differential Revision: https://phab.mercurial-scm.org/D7463

tests: add more tests for "hg shelve --delete" It appears that the only tests for "hg shelve --delete" concern command errors (e.g. incompatible command options). Adding some more to check that non-existent names are handled and a success case. Differential Revision: https://phab.mercurial-scm.org/D7460

scmutil: add assertions to help pytype Differential Revision: https://phab.mercurial-scm.org/D7456

revset: add an assertion to help pytype Differential Revision: https://phab.mercurial-scm.org/D7455

import: add a --secret option Similarly to "hg commit", we add a --secret option to "hg import" for committing with the secret phase. The option has no short form since there already is a "-s" for "--similarity". .. feature:: ``hg import`` has a new --secret option for committing with the secret phase.

hgweb: add diffs to the json changeset template While it's possible to get the diff for each file individually via an obscure url that I'm not even sure is documented, (e.g. diff/{rev}/{filename}?style=json ), it is more convenient to provide the full diff from the changeset view. This is already normally computed for other non-JSON templates, so it seems like an oversight to omit it for the JSON one. The output format is a bit unwieldy but it's consistent with the existing format output by the aforementioned obscure url.

hgweb: add files to the json changeset template As far as I can tell, it's currently impossible via JSON to figure out what files are touched by a particular revision. Reusing the filenodelink and filenolink templates from the other hgweb templates should provide this information.

hgweb: add a status property to file list context The web templates merely indicate if files touched by this revision are in the commit or not, i.e. if they are removed. It would be helpful to have more context and also indicate whether the files are added, modified, or removed.

match: remove explicitdir attribute No one sets it or reads it anymore. Differential Revision: https://phab.mercurial-scm.org/D7441

dirstate: stop caring about match.explicitdir No one passes it in anymore. Differential Revision: https://phab.mercurial-scm.org/D7440

dirstate: include explicit matches in match.traversedir calls The caller should not have to register to get both explicit and non-explicit matches. Differential Revision: https://phab.mercurial-scm.org/D7439

commit: drop unused "vdirs" argument from repo.checkcommitpatterns() Differential Revision: https://phab.mercurial-scm.org/D7438

commit: rewrite check for `hg ci <path>` being a directory The matcher API is complicated by match.bad, match.explicitdir, and match.traversedir. We already have very few users of match.explicitdir. By rewriting this check we get close to being able to remove match.explicitdir. This may make the check slower, but I think that will be very marginal. Disclosure: I actually wrote this patch to solve a bug we've seen with an internal extension. The internal extension overrides the dirstate walk to only walk the files that our FUSE tells us are modified. That led to "vdirs" not getting populated as this code expected. I have wanted to get rid of match.explicitdir for a very long time, though. Differential Revision: https://phab.mercurial-scm.org/D7437

lock: fix race in lock-breaking code With low frequency, I see hg pulls fail with output like: abort: no such file or directory: .hg/store/lock I think what happens is, in lock.py, in: def _testlock(self, locker): if not self._lockshouldbebroken(locker): return locker # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: with lock(self.vfs, self.f + b'.break', timeout=0): self.vfs.unlink(self.f) except error.LockError: return locker if a lock is breakable on disk, and two hg processes concurrently get to the "if locker dead" comment, a possible interleaving is: process1 finishes executing the function and then process2 finishes executing the function. If that happens, process2 will either get ENOENT in self.vfs.unlink (resulting in the spurious failure above), or break a valid lock and potentially cause repository corruption. The fix is simple enough: make sure the lock is breakable _inside_ the critical section, because only then can we know that no other process can invalidate our knowledge on the lock on disk. I don't think there are tests for this. I've tested this manually with: diff --git a/mercurial/lock.py b/mercurial/lock.py --- a/mercurial/lock.py +++ b/mercurial/lock.py @@ -351,6 +351,8 @@ class lock(object): if not self._lockshouldbebroken(locker): return locker + import random + time.sleep(1. + random.random()) # if locker dead, break lock. must do this with another lock # held, or can race and break valid lock. try: @@ -358,6 +360,7 @@ class lock(object): self.vfs.unlink(self.f) except error.LockError: return locker + time.sleep(1) def testlock(self): """return id of locker if lock is valid, else None. and I see this change of behavior before/after this commit: $ $hg init repo $ cd repo $ ln -s $HOSTNAME/effffffc:987654321 .hg/wlock $ touch a $ $hg commit -Am_ & $hg commit -Am _; wait -abort: No such file or directory: '/tmp/repo/.hg/wlock' adding a +warning: ignoring unknown working parent 679a8959a8ca! +nothing changed Differential Revision: https://phab.mercurial-scm.org/D7199

lock: refactor in preparation for next commit Differential Revision: https://phab.mercurial-scm.org/D7198

extensions: suppress a pytype failure due to a typeshed bug Bug filed upstream, suppress the failure here so we can move on. Differential Revision: https://phab.mercurial-scm.org/D7410

dispatch: add some assertions to give pytype a helping hand Differential Revision: https://phab.mercurial-scm.org/D7409

extensions: hide two confusing import statements from pytype Differential Revision: https://phab.mercurial-scm.org/D7408

debugcommands: add assertions to convince pytype peer is not None This function is moderately annoyingly defined, and peer is set up iff we're not in raw-proto mode. That's fine, but it confuses pytype. Adding these assertions is a low-overhead way to convince pytype we're doing reasonable things. Differential Revision: https://phab.mercurial-scm.org/D7407

debugcommands: suppress import errors for pytype Differential Revision: https://phab.mercurial-scm.org/D7385

state: add a pytype annotation Differential Revision: https://phab.mercurial-scm.org/D7383

logcmdutil: add a type annotation This wasn't actually required in the end, as there was a real bug found by pytype, but the annotation helped me figure that out. We can drop this patch if that's the preference. Differential Revision: https://phab.mercurial-scm.org/D7382

cmdutil: add a pytype annotation to help out some callsites I think we could constrain the values here more than Any, but this lets us move forward with typechecking commands.py. Differential Revision: https://phab.mercurial-scm.org/D7381

hghave: add a check for the `xz` compression utility This isn't install by default on Mac, which causes a test failure. The logic for avoiding the command is a little goofy, but nested `#if` isn't supported, and it still seems worth running the hg command to see if anything explodes. With this, the py3 tests run (almost) cleanly on 10.14.6: # Ran 835 tests, 58 skipped, 1 failed. Alas, the mac-packaging test is skipped because it's slow. The failure here is in test-releasenotes-merging.t, complaining about not being able to import the `fuzzywuzzy` module. I have it installed on py3 (thus the test isn't skipped), but not on py2. So there must be some unintended cross pollination here when running `hg` commands. Differential Revision: https://phab.mercurial-scm.org/D7374

py3: use '%d' for formatting a revnum in `hg co --date` code Differential Revision: https://phab.mercurial-scm.org/D7429

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7454

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7453

templates: make {indent("", " ")} be empty indent() is documented to indent all non-empty lines, but it made an exception for the first line, which always got indented. I also made indent() not indent the first line even if an indent override was given for the first line. I think that is what one would usually want. Differential Revision: https://phab.mercurial-scm.org/D7432

tests: document behavior of indent() with empty first line indent() is documented to indent all non-empty lines, but it still indents the first line even if it's empty. Let's have tests for that. Differential Revision: https://phab.mercurial-scm.org/D7431

tests: check for Windows line endings We shouldn't generally be using Windows line endings in files under version control. I've accidentally committed a few files with Windows line endings recently. So let's add a test for this. Differential Revision: https://phab.mercurial-scm.org/D7448

tests: verify that Python scripts are blackened Scripts are Python too. They should be blackened. (As suggested by Pierre-Yves in D7420.) Differential Revision: https://phab.mercurial-scm.org/D7447

black: blacken scripts These scripts weren't blackened. I found these as part of adding script checking to test-check-format.t. # skip-blame black Differential Revision: https://phab.mercurial-scm.org/D7446

doc: don't use mutable default arguments It appears our mutable default arguments checker doesn't find arguments not on the first line of a function definition :( The arguments aren't used, so the default value is irrelevant. I found this when blackening this script in a future commit. Differential Revision: https://phab.mercurial-scm.org/D7445

tags: don't check for clean files when checking for dirty working copy We didn't request status for clean files so there should never be any entries in the list of clean files (at least not since dd773340a085), but it's misleading to check the list. Differential Revision: https://phab.mercurial-scm.org/D7428

branchmap: make "closed" a set from beginning instead of converting from list Differential Revision: https://phab.mercurial-scm.org/D7427

debugextensions: gracefully handle missing __file__ attributes This was crashing PyOxidizer. Differential Revision: https://phab.mercurial-scm.org/D7452

debuginstall: gracefully handle missing __file__ attributes This was crashing PyOxidizer. While here, point "Python lib" and "installed modules" to the oxidized binary when read from memory instead of pretending their location is unknown. Differential Revision: https://phab.mercurial-scm.org/D7451

typing: add pseudo localstr.__init__() to help pytype Apparently, pytype failed to parse localstr.__new__()? This fixes the following errors: line 126, in __hash__: No attribute '_utf8' on localstr [attribute-error] line 188, in tolocal: Function localstr.__init__ was called with the wrong arguments [wrong-arg-types] Expected: (self, string: str, ...) Actually passed: (self, string: bytes, ...)

typing: fix argument type of encoding.localstr() It takes both utf-8 and local strings in bytes type.

typing: suppress error of py2 encoding.strtolocal() and .strfromlocal() I don't know how to conditionally get rid of these py2/py3 overloads from .pyi file. Instead, this patch makes pytype ignore the false-positives: line 271, in strtolocal: bad option in return type [bad-return-type] Expected: bytes Actually returned: str

typing: constrain argument/return types of encoding.toutf8b()

typing: fix argument type of encoding.tolocal() and .fromutf8b() Fixes various pytype errors.

typing: fix forward reference in _Tlocalstr type bound AFAIK, a quoted string in type position is translated to its ForwardRef. https://github.com/python/typing/issues/34 This fixes the following error: File "mercurial/encoding.py", line 38, in <module>: Name 'localstr' is not defined [name-error]

encoding: make getcols() raise exception explicitly This helps pytype. File "mercurial/encoding.py", line 329, in getcols: bad option in return type [bad-return-type] Expected: bytes Actually returned: None

setup: conditionalize access to `sys.dllhandle` when building extensions This code is only run on Windows, and was crashing PyOxidizer when running in `setup-py-install` mode. Now an oxidized binary can be built by simply pointing to setup.py. Something is slightly different now that it's not being built from a virtualenv. Previously, `hg version` could print to the screen, but now it aborts saying "Incorrect function". But I can see the output if redirected to a file, and it's not complaining about missing C extensions, so I think those are loading now (unlike from the virtualenv). The interesting this about this incorrect function output is that it failed when initially built. I then went back and did a `make clean` and `make local` with py3 and then py2 to ensure I didn't break the existing code. At that point I ran the oxidized executable again and it was able to print to the screen normally! So I ran `pyoxidizer build` again, it only output the following, and then running the executable failed to output again: (pyO2_venv) C:\Users\Matt\hg3\hg_pyO2>pyoxidizer build Finished dev [unoptimized + debuginfo] target(s) in 0.12s packaging application into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug purging C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug copying C:/Users/Matt/hg3/hg_pyO2\build\target\x86_64-pc-windows-msvc\debug\hg_pyO2.exe to C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe resolving packaging state... writing license for [...] hg_pyO2 packaged into C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug executable path: C:/Users/Matt/hg3/hg_pyO2\build\apps\hg_pyO2\x86_64-pc-windows-msvc\debug\hg_pyO2.exe Differential Revision: https://phab.mercurial-scm.org/D7444

setup: use bytes for assumed python version Caught by PyOxidizer when hacking on Windows. # skip-blame for b'' prefixing Differential Revision: https://phab.mercurial-scm.org/D7443

setup: combine two contiguous string literals Differential Revision: https://phab.mercurial-scm.org/D7442

util: remove datapath and swith users over to resourceutil The util module no longer needs the datapath for injecting it into the i18n module, so we can remove it from there and update other users. Differential Revision: https://phab.mercurial-scm.org/D7436