lfs: add a switch to `hg verify` to ignore the content of blobs Trying to validate the fulltext of an external revision causes missing blobs to be downloaded and cached. Since the downloads aren't batch prefetched[1] and aren't compressed, this can be expensive both in terms of time and space. I made this a tri-state instead of a simple bool because there's an existing (undocumented) config to handle this, and it would be weird if `hg verify` were to suddenly start ignoring that config but an `hg recover` initiated verify honors it. Since this uses the same config setting, it too will skip rename verification (which requires fulltext, but not for LFS). [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-April/116118.html Differential Revision: https://phab.mercurial-scm.org/D7708

revlog: run rustfmt nightly I'm a little nervous about folding this back (might be nightly rustfmt mismatches?) so I want someone to review this. Differential Revision: https://phab.mercurial-scm.org/D7813

examples: specify rustfmt nightly using a $() construct This is ugly, but it's how we have to configure rustfmt for now as we require nightly rustfmt. Differential Revision: https://phab.mercurial-scm.org/D7812

hg-core: rustfmt path.rs The file as vendored does not conform to our source formatting conventions. Let's reformat it so it does. # skip-blame automated code reformatting Differential Revision: https://phab.mercurial-scm.org/D7580

hg-core: vendor Facebook's path utils module The added file was imported from https://github.com/facebookexperimental/eden/blob/d1d8fb939a39aa331ae7f162c39cbcaa511d474b/eden/scm/lib/util/src/path.rs without modifications. The file is not yet integrated into our project. This will be done in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D7573

revlog-native: introduced ABI version in capsule Concerns that an inconsistency could arise between the actual contents of the capsule in revlog.c and the Rust consumer have been raised after the switch to the array of data and function pointers in f384d68d8ea8. It has been suggested that the `version` from parsers.c could be use for this. In this change, we introduce instead a separate ABI version number, which should have the following advantages: - no need to change the consuming Rust code for changes that have nothing to do with the contents of the capsule - the version number in parsers.c is not explicitely flagged as ABI. It's not obvious to me whether an ABI change that would be invisible to Python would warrant an increment The drawback is that developers now have to consider two version numbers. We expect the added cost of the check to be negligible because it occurs at instantiation of `CIndex` only, which in turn is tied to instantiation of Python objects such as `LazyAncestors` and `MixedIndex`. Frequent calls to `Cindex::new` should also probably hit the CPU branch predictor. Differential Revision: https://phab.mercurial-scm.org/D7856

phases: make phasecache._phasesets immutable Previously, some code paths would mutate the cache itself, which could give weird results if multiple revsets got evaluated through that path. Differential Revision: https://phab.mercurial-scm.org/D7854

phases: reduce code duplication in phasecache.getrevset This is a functional NOP other than reducing some of the duplication in that method. Differential Revision: https://phab.mercurial-scm.org/D7853

scmutil: fix an unbound variable with progressbar debug enabled Differential Revision: https://phab.mercurial-scm.org/D7852

hgext: replace references to hashlib.sha1 with hashutil.sha1 When in a non-pure build of Mercurial, this will provide protections against SHA1 collision attacks. Differential Revision: https://phab.mercurial-scm.org/D7851

sslutil: migrate to hashutil.sha1 instead of hashlib.sha1 This is a straight-line replacement like the others, but I split it out since it's used in a network context and I'm not sure this is appropriate (we should probably drop support for sha1 fingerprints over TLS) and wanted this to be easily dropped. Differential Revision: https://phab.mercurial-scm.org/D7850

core: migrate uses of hashlib.sha1 to hashutil.sha1 Differential Revision: https://phab.mercurial-scm.org/D7849

hashutil: new package for hashing-related features Right now this just tries to use our sha1dc and if it's missing (eg a --pure build) we fall back to hashlib. I imagine in the future we'll want some other things in here for detecting what hasher is in use as we transition off sha1. Differential Revision: https://phab.mercurial-scm.org/D7848

sha1dc: initial implementation of Python extension A future change will use this when available to avoid sha1 collision issues until we can get moved to something else. Differential Revision: https://phab.mercurial-scm.org/D7815

sha1dc: import latest version from github After the recent SHA1 news, the attacks are serious enough we should be more proactive. This code will at least allow detection of attacks early. It's already widely deployed in Git. This is git revision 855827c583bc30645ba427885caa40c5b81764d2 of the sha1collisiondetection repo[0], with most of the files omitted. A follow-up change will introduce Python bindings for this code. 0: https://github.com/cr-marcstevens/sha1collisiondetection Differential Revision: https://phab.mercurial-scm.org/D7814

transaction: add a `hasfinalize` method The method allow code to check if an existing callback exists. It allow them to skip potentially expensive setup for a callback. Differential Revision: https://phab.mercurial-scm.org/D7832

changelog: fix the diverted opener to accept more kwargs The current code prevent the use of `atomictemp` file with the changelog opener. I do not see a good reason for this limitation. Differential Revision: https://phab.mercurial-scm.org/D7831

revlog: reorder a conditionnal about revlogio if we are using REVLOGV0, we will not use a rust based index. This small line movement make it clearer. Differential Revision: https://phab.mercurial-scm.org/D7830

rebase: delete seemingly unnecessary needupdate() This seemed to be about checking that the user hasn't updated away when we asked them to resolve merge conflicts. These days we call `cmdutil.checkunfinished()` and refuse to update, so the user shouldn't be able to get into this state. `test-rebase-interruptions.t` actually has some tests where it disables the rebase extension in order to be allowed to do some of these updates. That still passes, but I wouldn't personally haved cared if that failed. Differential Revision: https://phab.mercurial-scm.org/D7825

workingctx: move setparents() logic from localrepo to mirror overlayworkingctx It would be nice to later be able to call `wctx.setparents()` whether `wctx` is a `workingctx` or an `overlayworkingctx`. Differential Revision: https://phab.mercurial-scm.org/D7823

overlayworkginctx: implement a setparents() to mirror dirstate.setparents() This lets us make the in-memory and on-disk code a bit more similar. I'll soon also implement setparents() on the regular workingctx. Differential Revision: https://phab.mercurial-scm.org/D7822

overlayworkingctx: default branch to base context's branch This matches what the dirstate does (it reuses working copy parent's branch unless told otherwise). By moving the default out of `rebase.commitmemorynode()`, it will let us clean that up better later. Differential Revision: https://phab.mercurial-scm.org/D7821

grep: speed up `hg grep --all-files some/path` by using ctx.matches(match) ctx.matches(match) avoids walking unintersting parts of the tree when using tree manifests. That can make a very big difference when grepping in a small subset of the tree (2.0s -> 0.7s in my case, but can of course be made more extreme by picking a smaller subset of files). Differential Revision: https://phab.mercurial-scm.org/D7820

fix: fix grammar/typos in hg help -e fix

py3: byteify the opener option to use `rust.index` to allow Rust revlogs It looks like this corresponds to the byteified key written in localrepo in 8042856c90b6. Differential Revision: https://phab.mercurial-scm.org/D7818

graft: use revset for intersecting with ancestor set This addresses a TODO added in a1381eea7c7d (graft: do not use `.remove` on a smart set (regression), 2014-04-28). Differential Revision: https://phab.mercurial-scm.org/D7806

graft: don't remove from a list in a loop This addresses a TODO added in a1381eea7c7d (graft: do not use `.remove` on a smart set (regression), 2014-04-28). I couldn't measure any speedup. Differential Revision: https://phab.mercurial-scm.org/D7805

tests: avoid grafting the same change over and over The test case added in a1381eea7c7d (graft: do not use `.remove` on a smart set (regression), 2014-04-28) added a test case that grafted the same change (renaming 'a' to 'b') three times over. It had description "graft works on complex revset", but AFACT, all that it cared about was that some ancestor of the working copy was in the set of revisions to graft. So this patch changes the test to do that instead. (I plan to later make it so that grafting these renames on top of each won't create the empty commits they currently create.) Differential Revision: https://phab.mercurial-scm.org/D7804

py3: byteify some `ui.configbool()` parameters This popped up in 8042856c90b6. Differential Revision: https://phab.mercurial-scm.org/D7817

rust-discovery: type alias for random generator seed It just makes our life easier Differential Revision: https://phab.mercurial-scm.org/D7715

tests: split out another ~1/2 of test-graft.t The tests involving renames were also quite independent from the rest. Differential Revision: https://phab.mercurial-scm.org/D7803

tests: split out ~1/3 of test-graft.t test-graft.t is ~2400 lines and takes 34s to run. This patch moves the last ~1/3 of it to a separate file. The parts now run in 22s + 13s. On top of that, we can remove the #testcases from the old file, so it's only 22s + 2*13s instead of the 2*34s it was before. Differential Revision: https://phab.mercurial-scm.org/D7802

overlwayworkingctx: remove doubly bad reference to wrapped ctx for copies `_wrappedctx` lives on overlwayworkingctx, not on the repo object, so we should access it as `._wrappedctx`, not `._repo._wrappedctx`. More importantly, the overlayworkingctx is relative to its base, not including it, so the copies returned should not include copies made in the base. Differential Revision: https://phab.mercurial-scm.org/D7801

movedirstate: get copies from dirstate before setting parents Setting dirstate parents can modify the copies recorded in the dirstate when there are two dirstate parents. I don't think we ever call movedirstate() when there is more than one parent, but it seems clearer to get the copies from the dirstate first anyway. Differential Revision: https://phab.mercurial-scm.org/D7800

fix: convert clang-format-ignorelist to use wildcards It's very brittle to specify the exact filenames for these vendored/thirdparty libraries; when there's an upgrade, it's likely not going to be updated. Differential Revision: https://phab.mercurial-scm.org/D7619

packaging: update Ubuntu docker build dependencies to Python 3 Changeset 7574ccd87200f02e updated the Debian docker builds to Python 3. In doing so, the Ubuntu docker-based builds broke, as the Dockerfile does not include the new dependencies as specified in `contrib/packaging/debian/control`. This commit changes the dependencies in the Ubuntu Dockerfile template to their Python 3 equivalents, fixing `make docker-ubuntu-bionic`. Differential Revision: https://phab.mercurial-scm.org/D7810

mmap: add a size argument to mmapread With this argument, we can control the size of the mmap created. (previously it was always the whole file. Differential Revision: https://phab.mercurial-scm.org/D7808

revlogutils: move the NodeMap class in a dedicated nodemap module We will introduce more nodemap related code, so it make sense to move the existing code in that module first. Differential Revision: https://phab.mercurial-scm.org/D7809

progress: flush stderr after clearing On python3, ui.stderr is buffered, it seems, so we need to flush it to actually get the progress bar off the screen. This is important since ui.write() will call into progbar.clear() if it thinks there's a progress bar on the screen, with the intent that the next thing it outputs is at the beginning of the line (instead of at the end of the progress bar line). Without the flush, we buffer up the clearing of the screen, and we get some really weird/corrupt output. Differential Revision: https://phab.mercurial-scm.org/D7784

rcutil: drop the `defaultrcpath()` method (API) The resource based code can service py2, py3 and an oxidized executable, so there's no reason to leave this around. It was flagged as an API change when it was introduced, so flagging it again. Differential Revision: https://phab.mercurial-scm.org/D7777

ui: add the ability to apply `defaultrc` configs from resources We will want the ability to cat out these resources, but the same would apply to templates, so I'm going to wait for the dust to settle on that. Reading the default config directly from the filesystem is still in place for now. Differential Revision: https://phab.mercurial-scm.org/D7776

ui: refactor `readconfig()` into a form that can consume resources The old form can't completely go away, because files outside of packages still need to be read. The name passed in here is a tuple of `package name, resource` as needed by the resource API. I like the idea of stating the config file is embedded in the executable by listing is as `exe!package.resource`. This would be consistent with how `debuginstall` points to the executable for the python executable, lib, and installed modules. While in practice the filesystem path is available from the backing ResourceReader when the resource is opened, it is a relative path on py2 and absolute on py3. Further, while this would show in the `hg config` output for each option if set as such here, it doesn't show in the `reading from...` line when `--debug` is used. The file isn't actually open where that prints, so there's no way I see to get that info there. So I opted for the simple prefix to distinguish resources from files. Differential Revision: https://phab.mercurial-scm.org/D7775

resourceutil: implement `contents()` to iterate over resources in a package Differential Revision: https://phab.mercurial-scm.org/D7774

resourceutil: implement `is_resource()` This will be needed when iterating resources. Differential Revision: https://phab.mercurial-scm.org/D7773

resourceutil: don't limit resources to the `mercurial` package This should make things a little clearer, in that it now requires the full package name to access a resource. But the real motivation is that `extensions._disabledpaths()` walks the `hgext` directory looking for bundled extensions. This in turn feeds, among other things: 1) Listing disabled extensions in `hg help extensions` 2) Indicating that an unknown command is in a non-enabled extension 3) Displaying help for non-enabled extensions 4) Generating documentation 5) Announcing LFS is auto-enabled (or not) when cloning from an LFS source The filesystem based ResourceReader will happily return *.py and *.pyc, but the one supplied by PyOxidizer doesn't. Presumably we can change that. The only other idea I had here is for setup.py to generate a text file containing the list of extensions, but that doesn't seem great when running from source. Differential Revision: https://phab.mercurial-scm.org/D7772

setup: include `defaultrc` in the package list This is needed to access the config files via the resourceutil framework, which is needed to embed the config files when uing PyOxidizer. Differential Revision: https://phab.mercurial-scm.org/D7771

tests: update test-releasenotes-formatting.t output The output seems to be left outdated because running this test requires fuzzy-wuzzy which is not generally installed. Differential Revision: https://phab.mercurial-scm.org/D7723

tests: fix a copy/paste name duplication in storage.py Flagged by PyCharm as a redeclared name without usage, because the name matches the function immediately preceding it. But since this differs by doing regular reads instead of raw reads, I assume we still want it. But I don't see either function being called anywhere. Differential Revision: https://phab.mercurial-scm.org/D7769

commit: change default `editor` parameter to None Flagged by PyCharm as a boolean not being callable later where it is used. There's no actual bug here because of `if editor` checks before using. Differential Revision: https://phab.mercurial-scm.org/D7768

cleanup: drop redundant character escapes outside of `[]` Flagged by PyCharm. `@`, `:`, `<`, `>`, and `{` aren't special characters. (I'm a bit surprised that it doesn't also want to unescape `}` in schemes.py.) Differential Revision: https://phab.mercurial-scm.org/D7767

cleanup: drop redundant character escapes from `[]` character sets Flagged by PyCharm. Most special characters lose their special meaning inside square brackets. The `-` in util.py doesn't need to be escaped because it is the last character in the set. Differential Revision: https://phab.mercurial-scm.org/D7766

cleanup: replace contiguous spaces in regex patterns with an explicit count Flagged by PyCharm, this form should be more readable. Differential Revision: https://phab.mercurial-scm.org/D7765

mq: avoid using `__file__` to compare modules This fixes ~70 tests when using an oxidized executable, bringing the current failure count down to 92 (with 110 skips) when templates/ is copied next to the executable. Differential Revision: https://phab.mercurial-scm.org/D7778

util: avoid referencing `time.clock()` on Windows when missing (issue6238) It's been removed in 3.8, and issues a deprecation warning since 3.3. Differential Revision: https://phab.mercurial-scm.org/D7780

py3: replace `time.clock()` with `time.perf_counter()` The former was removed in 3.8. The util function uses `time.clock()` if the latter is unavailable. Differential Revision: https://phab.mercurial-scm.org/D7779

tests: fix a "naked exception" issue in test-remotefilelog-prefetch.t Differential Revision: https://phab.mercurial-scm.org/D7811