tests: make test-config-env.py a little less hacky Overriding os.path.isdir to always returns False is very confusing. Let's make it more explicit by specifically not returning any defaultrc/ configs. Differential Revision: https://phab.mercurial-scm.org/D7623

help: get helptext/ data from `resources` module if available For PyOxidizer, we need to read configs using the `resources` module. This patch makes it so we use that module if available (i.e. Python >= 3.7). It does that by adding a new `open_resource()` function to our `resourceutil` module. Tested by running `$PYTHON ./hg help pager` for each $PYTHON in {python2, python3.6, python3.7}. Differential Revision: https://phab.mercurial-scm.org/D7622

pycompat: allow pycompat.sysbytes() even if input already is bytes pycompat.sysstr() on py3 accepts an input that's already str (i.e. unicode). This patch makes it so pycompat.sysbytes() on py3 accepts an input that's already bytes. Allowing that makes it possible to do pycompat.sysbytes(fp.name) where fp.name is either bytes or unicode, as we'll get when fp can come from either open() or resources.open_binary(). Differential Revision: https://phab.mercurial-scm.org/D7621

merge with stable

rust-cpython: do not convert warning pattern to utf-8 bytes On Unix, both Rust Path and Mercurial expect a locale-dependent bytes, and we don't support Windows yet. Differential Revision: https://phab.mercurial-scm.org/D7614

rust-cpython: import utils::files::* function at module level IIRC, it's common in Rust to call functions with the module prefix. Differential Revision: https://phab.mercurial-scm.org/D7613

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7612

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7611

rust-threads: force Rayon to respect the worker count in config As per the inline comment, this is a workaround because Rust code does not yet know how to read config files. Differential Revision: https://phab.mercurial-scm.org/D7610

rust-dirs: handle forgotten `Result`s In 1fe2e574616e I introduced a temporary bugfix to align Rust code with a new behavior from C/Python and forgot about a few `Result`s (cargo's compiler cache does not re-emit warnings on cached modules). This fixes it. For the record, I am still unsure that this behavior change is a good idea. Note: I was already quite unhappy with the setters and getters for the `DirstateMap` and, indirectly, `Dirs`, and this only further reinforces my feelings. I hope we can one day fix that situation at the type level; Georges Racinet and I were just talking about devising a POC for using the builder pattern in the context of FFI with Python, we'll see what comes out of it. Differential Revision: https://phab.mercurial-scm.org/D7609

merge with stable

status: outputting structured unfinished-operation information This adds a new item in the json/template output for morestatus and added item types to all entries. Differential Revision: https://phab.mercurial-scm.org/D7595

chg: fix chg to work with py3.7+ "coercing" the locale When the environment is empty (specifically: it doesn't contain LC_ALL, LC_CTYPE, or LANG), Python will "coerce" the locale environment variables to be a UTF-8 capable one. It sets LC_CTYPE in the environment, and this breaks chg, since chg operates by: - start hg, using whatever environment the user has when chg starts - hg stores a hash of this "original" environment, but python has already set LC_CTYPE even though the user doesn't have it in their environment - chg calls setenv over the commandserver. This clears the environment inside of hg and sets it to be exactly what the environment in chg is (without LC_CTYPE). - chg calls validate to ensure that the environment hg is using (after the setenv call) is the one that the chg process has - if not, it is assumed the user changed their environment and we should use a different server. This will *never* be true in this situation because LC_CTYPE was removed. Differential Revision: https://phab.mercurial-scm.org/D7550

fuzz: add support for fuzzing under either Python 2 or 3 This was more of a hairball than I hoped, but it appears to work. The hg-py3 branch of my oss-fuzz fork on github has the remaining changes to switch us to Python 3, but we may as well retain Python 2 fuzzing support for at least a little while. Differential Revision: https://phab.mercurial-scm.org/D7592

phabricator: color the status in the "phabstatus" view I couldn't figure out strikethrough for "abandoned" like I've see with word diff. Differential Revision: https://phab.mercurial-scm.org/D7608

phabricator: add the "Changes Planned" status name I noticed this when coloring the status output. But it looks like status names are also used in the pseudo revset language as well, and abort when parsing a name not in this list. Differential Revision: https://phab.mercurial-scm.org/D7607

fuzz: fix mpatch_corpus to not have an overridden __repr__ on py3 Differential Revision: https://phab.mercurial-scm.org/D7606

formatting: fix some recent formatting regressions Differential Revision: https://phab.mercurial-scm.org/D7605

remotefilelog: remove the `ensurestart` usage The option does not seems to do what we want since it does not wait on the background process actually starting (eg: taking the lock). And the race have been dealt with another way anyway. This effectively back out changeset 2c74337e6483. Differential Revision: https://phab.mercurial-scm.org/D7590

remotefilelog: remove most of the debugwait* command in the tests Since we use a dedicated developer config, we do not need them anymore. Differential Revision: https://phab.mercurial-scm.org/D7589

remotefilelog: use the correct redirection in tests The test were using '2>%1' which just redirect stderr in the "%1" file. The intend was '2>&1' which redirect stderr on stdin. This impact tests because some commit were picking up this '%1' files. Differential Revision: https://phab.mercurial-scm.org/D7588

remotefilelog: remove most sleep in tests Now that we removed most of the races, we can remove the infamous sleeps. Differential Revision: https://phab.mercurial-scm.org/D7587

remotefilelog: have command wait for background task in the tests We leverage the feature introduced in the two previous commit to have the main command wait for the background task to be finished before returning. All test still pass and instability seems to have gone away. Most of the sleep and explicit wait are now useless and will be cleaned up in later changesets. Differential Revision: https://phab.mercurial-scm.org/D7586

remotefilelog: add a developer option to wait for background processes In order to block the main command on the subprocess exiting, we ensure the repo's ui object will call the subprocess.wait() method to ensure the top-level hg process doesn't exit until all background processes have also done so. Currently, in the tests, most operation spawning background process as followed by commands waiting for these operations to complete. However this waiting is racy. First because it seems like we can start waiting before the background operation actually start, in which case it is prematurely detected as "done". Second, because some commands may spawn multiple background operation for the same operation (eg: rebase can apparently trigger multiple prefetch). The current approach could be updated to maybe handle the first issue, but the second one will never be properly handled. In most case, we do not care that the bg process keep running after the command end. (Since we explicitly wait for them to end before doing anything else). So we add an option to wait on the background process before exiting the command. We'll put it in use in the next changeset. Differential Revision: https://phab.mercurial-scm.org/D7585

procutil: add a option to not fully detach background process When writing test, it is useful to make code pretending to spawn a fully detached process while still actually waiting for it to finish. We add the option to `procutil.runbgcommand`, to do so. We will use this in remote-filelog tests. Differential Revision: https://phab.mercurial-scm.org/D7584

cext-revlog: fixed __delitem__ for uninitialized nodetree This is a bug in a code path that's seldom used, because in practice (at least in the whole test suite), calls to `del index[i:j]` currently just don't happen before the nodetree has been initialized. However, in our current work to replace the nodetree by a Rust implementation, this is of course systematic. In `index_slice_del()`, if the slice start is smaller than `self->length`, the whole of `self->added` has to be cleared. Before this change, the clearing was done only by the call to `index_invalidate_added(self, 0)`, that happens only for initialized nodetrees. Hence the removal was effective only from `start` to `self->length`. The consequence is index corruption, with bogus results in subsequent calls, and in particular errors such as `ValueError("parent out of range")`, due to the fact that parents of entries in `self->added` are now just invalid. This is detected by the rebase tests, under conditions that the nodetree of revlog.c is never initialized. The provided specific test is more direct. Differential Revision: https://phab.mercurial-scm.org/D7603

filemerge: fix a missing attribute usage Flagged by both pytype and VSCode. Differential Revision: https://phab.mercurial-scm.org/D7465

filemerge: drop a default argument to appease pytype The function slices and takes the length of this argument without internally setting it if not provided. There was no bug here because both callers passed the argument. Differential Revision: https://phab.mercurial-scm.org/D7464

fuzz: add a seed corpus for the dirs fuzzer I was hoping to trigger an asan violation under Python 3 that some internal tests at Google found, but for some reason that's beyond me I can't seem to manage. Differential Revision: https://phab.mercurial-scm.org/D7600

fuzz: clean up production of seed corpora This was getting out of hand. Differential Revision: https://phab.mercurial-scm.org/D7599

status: add template/json data about whether a file has unresolved conflicts Differential Revision: https://phab.mercurial-scm.org/D7594

status: split morestatus data loading from display This is a small refactoring in preparation for adding more morestatus functionality (notably for templated/JSON output) - the goal is to use the data inside the status display loop, as well as output the overall state in a templatable/structured way. Differential Revision: https://phab.mercurial-scm.org/D7593

phabricator: add a "phabstatus" template keyword We add a "phabstatus" template keyword, returning an object with "url" and "status" keys. This is quite similar to "phabreview" template keyword, but it queries phabricator for each specified revision so it's going to be slow (as compared to the "phabstatus" show view from previous changeset). Differential Revision: https://phab.mercurial-scm.org/D7507

phabricator: add a "phabstatus" show view We add a "phabstatus" show view (called as "hg show phabstatus") which renders a dag with underway revisions associated with a differential revision and displays their status. The revisions shown is a subset of that shown by "work" view, only including revisions with known by Phabricator. Differential Revision: https://phab.mercurial-scm.org/D7506

dirs: fix out-of-bounds access in Py3 The hack for mutating Python's variable-length integers that was ported to py3 in cb3048746dae (dirs: port PyInt code to work on Python 3, 2016-10-08) was reading from ob_digit[1] instead of ob_digit[0] for some reason. Space for ob_digit[1] would only be allocated for integers larger than 30 bits, so we ended up writing to unallocated memory. Also, we would write an integer that's 2^30 times too large, so we would never free these integers. Found by AddressSanitizer. Differential Revision: https://phab.mercurial-scm.org/D7597

manifestcache: add some test involving shares Some issue around shares have been reported. I am adding tests to better cover the share case. The test show that the code behave as expected so far. Differential Revision: https://phab.mercurial-scm.org/D7602

fuzz: fix test-fuzz-targets.t to run with python3 Differential Revision: https://phab.mercurial-scm.org/D7601

py3: fix a bytes vs str issue in remotefilelog extension Differential Revision: https://phab.mercurial-scm.org/D7598

rust-matchers: add doctests for `AlwaysMatcher` Differential Revision: https://phab.mercurial-scm.org/D7527

rust-hg-path: add method to get part of a path relative to a prefix This will be used in the next patch in this series. Differential Revision: https://phab.mercurial-scm.org/D7526

rust-matchers: improve `Matcher` trait ergonomics `VisitChildrenSet` has no need to own the set, this will save allocations. The `file_set` return type change is motivated by both ergonomics and... being able to compile code. The `AlwaysMatcher` does not store a `file_set`, which requires it to return an owned `HashSet`, which in turn would change our return type to `Cow<&HgPath>` (lifetimes omitted). This is both un-ergonomic and troublesome for more complex lifetime issues (especially with the upcoming `FileMatcher` in the following patch). Differential Revision: https://phab.mercurial-scm.org/D7525

rust-dirs-multiset: use `AsRef` instead of concrete types when possible I also renamed `vec` to `dirstate`, because it was not a great name. Differential Revision: https://phab.mercurial-scm.org/D7524

logcmdutil: call _exthook() in changesettemplater Class changesetprinter has an _exthook() method that is called in _show() before the patch is displayed. Call the method as well in changesettemplater._show(). Differential Revision: https://phab.mercurial-scm.org/D7505

phabricator: fix processing of tags/desc in getoldnodedrevmap() It seems that the previous logic was wrong (it essentially comes from changeset 3ab0d5767b54 where the result got accumulated instead of early returned). First of all, the "continue" in first "if m:" is useless because we're at the end of the loop. Then, the algorithm seems weird because we will process all predecessors of a node and possibly override `toconfirm[node]` for each of these having a tag (maybe this doesn't happen, but still). Finally, we would also override `toconfirm[node]` when the "Differential Revision: " is found in changeset description. Maybe this is not a big deal when there is no mix of local tag and changeset description update? The logic is changed so that the loop on predecessors stops upon first match of a tag and so that the changeset description is only checked if no tag was found. Therefore, `toconfirm[node]` is only set once. Differential Revision: https://phab.mercurial-scm.org/D7513

tests: replace [[]] bashism with portable [] invocation In this case nothing fancy is required. Differential Revision: https://phab.mercurial-scm.org/D7596

rust-hg-path: implement `Display` for `HgPath` and `HgPathBuf` This is useful when debugging, to get a human readable output instead of an array of `u8`. Differential Revision: https://phab.mercurial-scm.org/D7523

rust-performance: introduce FastHashMap type alias for HashMap Rust's default hashing is slow, because it is meant for preventing collision attacks. For all of the current Rust code, we don't care about those attacks, because if an person with bad intentions has write access to your repo, you have other issues. I've chosen to use the TwoXHash crate because it was made by a reputable member of the Rust community and has very good benchmarks. For now it does not seem to improve performance by much for the current code, but it's something else to not worry about when benchmarking code: in a previous experiment with copytracing in Rust, it accounted for more than 10% of the time of the entire script. Differential Revision: https://phab.mercurial-scm.org/D7116

mail: use procutil.shellsplit instead of bytes.split to parse command Differential Revision: https://phab.mercurial-scm.org/D7541

annotate: describe --skip as taking a revset It's obvious to me, but probably wouldn't be obvious to a novice user. Differential Revision: https://phab.mercurial-scm.org/D7557

automation: use latest Windows AMI as base It looks like the previous base image no longer exists. I guess Amazon expires them or something. Let's switch to the newest version of the equivalent image. Differential Revision: https://phab.mercurial-scm.org/D7571

tests: stabilize test-extdiff.t on Windows This goes with 765a9c299c44. I'm not sure if we care about the missing file being spelled `nul`, so I removed it from the glob to be explicit about it. The line needed to be special cased anyway because of the quoting on Windows. Differential Revision: https://phab.mercurial-scm.org/D7572

exchange: fix an attempt to format a list into bytes This goes with 27c6d6f53d46. Differential Revision: https://phab.mercurial-scm.org/D7583

status: remove pointless filtering by alwaysmatcher in morestatus The code has been like this since it was imported from FB's hg-experimental repo. It has been like that even since it was added in that repo. So I don't know why it looks that way. Perhaps the idea was to one day filter the unresolved paths by any patterns provided by the user. We can add a matcher back if we ever decide to do that. Differential Revision: https://phab.mercurial-scm.org/D7591

tests: add test for Rust formatting We enforce formatting for Python and C. It makes sense to do it for Rust as well. Since our rustfmt.toml relies on unstable rustfmt features, we need to use a Nightly rustfmt with --unstable-features in order for it to work. This is a bit hacky and I would prefer we remove this requirement. But for now, this commit assumes this is the way things must be and we go out of our way to detect and use the rustfmt from the "nightly" toolchain, as installed via rustup. We had to add some environment variables to the tests to make the Rust binaries happy. Otherwise when running rustfmt we get an error about no default toolchain being installed. Differential Revision: https://phab.mercurial-scm.org/D7579

rust: run rustfmt # skip-blame automated reformatting Differential Revision: https://phab.mercurial-scm.org/D7578

fuzz: clean up some repetition on building parsers.so fuzzers There was a lot of repetition here that I realized could be cleared out. I think there's some more work we could do here, but this is enough for now. Differential Revision: https://phab.mercurial-scm.org/D7568

fuzz: remove legacy setup for running fuzzers outside oss-fuzz We don't need this anymore. Differential Revision: https://phab.mercurial-scm.org/D7567

tests: finally fix up test-fuzz-targets.t It's been failing on my workstation for a while, since I have a new enough LLVM that I had the fuzzer goo, but not so new that I actually had FuzzedDataProvider. This is a better solution all around in my opinion. I _believe_ this should let us run these tests on most systems, even those using GCC instead of clang. That said, my one attempt to test this on my macOS laptop failed miserably, and I don't feel like doing more work on this right now. Differential Revision: https://phab.mercurial-scm.org/D7566

fuzz: make standalone_fuzz_target_runner call LLVMFuzzerInitialize Otherwise some of our fuzzers crash when they try and use Python. Differential Revision: https://phab.mercurial-scm.org/D7565

fuzz: use a more standard approach to allow local builds of fuzzers This is taken from the (improved since we started fuzzing) guide on ideal integrations. Rather than have our own wonky targets for building outside the fuzzer universe, we have a driver program we carry along and use when we're not using LibFuzzer. This will let us jettison a fair amount of goo. contrib/fuzz/standalone_fuzz_target_runner.cc is https://github.com/google/oss-fuzz/ file projects/example/my-api-repo/standalone from git revision c4579d9358a73ea5dbcc99cb985de1f2bf76dcf7, reformatted with out clang-format settings and a no-check-code comment added. It allows running a single test input through a fuzzer, rather than performing ongoing fuzzing as libfuzzer would. contrib/fuzz/FuzzedDataProvider.h is https://github.com/llvm/llvm-project/ file /compiler-rt/include/fuzzer/FuzzedDataProvider.h from git revision a44ef027ebca1598892ea9b104d6189aeb3bc2f0, reformatted with our clang-format settings and a no-check-code comment added. We can discard this if we instead want to add an hghave check for a new enough llvm that includes FuzzedDataProvder.h in the fuzzer headers. Differential Revision: https://phab.mercurial-scm.org/D7564