hgext: replace references to hashlib.sha1 with hashutil.sha1 When in a non-pure build of Mercurial, this will provide protections against SHA1 collision attacks. Differential Revision: https://phab.mercurial-scm.org/D7851

sslutil: migrate to hashutil.sha1 instead of hashlib.sha1 This is a straight-line replacement like the others, but I split it out since it's used in a network context and I'm not sure this is appropriate (we should probably drop support for sha1 fingerprints over TLS) and wanted this to be easily dropped. Differential Revision: https://phab.mercurial-scm.org/D7850

core: migrate uses of hashlib.sha1 to hashutil.sha1 Differential Revision: https://phab.mercurial-scm.org/D7849

hashutil: new package for hashing-related features Right now this just tries to use our sha1dc and if it's missing (eg a --pure build) we fall back to hashlib. I imagine in the future we'll want some other things in here for detecting what hasher is in use as we transition off sha1. Differential Revision: https://phab.mercurial-scm.org/D7848

sha1dc: initial implementation of Python extension A future change will use this when available to avoid sha1 collision issues until we can get moved to something else. Differential Revision: https://phab.mercurial-scm.org/D7815

sha1dc: import latest version from github After the recent SHA1 news, the attacks are serious enough we should be more proactive. This code will at least allow detection of attacks early. It's already widely deployed in Git. This is git revision 855827c583bc30645ba427885caa40c5b81764d2 of the sha1collisiondetection repo[0], with most of the files omitted. A follow-up change will introduce Python bindings for this code. 0: https://github.com/cr-marcstevens/sha1collisiondetection Differential Revision: https://phab.mercurial-scm.org/D7814

transaction: add a `hasfinalize` method The method allow code to check if an existing callback exists. It allow them to skip potentially expensive setup for a callback. Differential Revision: https://phab.mercurial-scm.org/D7832

changelog: fix the diverted opener to accept more kwargs The current code prevent the use of `atomictemp` file with the changelog opener. I do not see a good reason for this limitation. Differential Revision: https://phab.mercurial-scm.org/D7831

revlog: reorder a conditionnal about revlogio if we are using REVLOGV0, we will not use a rust based index. This small line movement make it clearer. Differential Revision: https://phab.mercurial-scm.org/D7830

rebase: delete seemingly unnecessary needupdate() This seemed to be about checking that the user hasn't updated away when we asked them to resolve merge conflicts. These days we call `cmdutil.checkunfinished()` and refuse to update, so the user shouldn't be able to get into this state. `test-rebase-interruptions.t` actually has some tests where it disables the rebase extension in order to be allowed to do some of these updates. That still passes, but I wouldn't personally haved cared if that failed. Differential Revision: https://phab.mercurial-scm.org/D7825

workingctx: move setparents() logic from localrepo to mirror overlayworkingctx It would be nice to later be able to call `wctx.setparents()` whether `wctx` is a `workingctx` or an `overlayworkingctx`. Differential Revision: https://phab.mercurial-scm.org/D7823

overlayworkginctx: implement a setparents() to mirror dirstate.setparents() This lets us make the in-memory and on-disk code a bit more similar. I'll soon also implement setparents() on the regular workingctx. Differential Revision: https://phab.mercurial-scm.org/D7822

overlayworkingctx: default branch to base context's branch This matches what the dirstate does (it reuses working copy parent's branch unless told otherwise). By moving the default out of `rebase.commitmemorynode()`, it will let us clean that up better later. Differential Revision: https://phab.mercurial-scm.org/D7821

grep: speed up `hg grep --all-files some/path` by using ctx.matches(match) ctx.matches(match) avoids walking unintersting parts of the tree when using tree manifests. That can make a very big difference when grepping in a small subset of the tree (2.0s -> 0.7s in my case, but can of course be made more extreme by picking a smaller subset of files). Differential Revision: https://phab.mercurial-scm.org/D7820

py3: byteify the opener option to use `rust.index` to allow Rust revlogs It looks like this corresponds to the byteified key written in localrepo in 8042856c90b6. Differential Revision: https://phab.mercurial-scm.org/D7818

graft: use revset for intersecting with ancestor set This addresses a TODO added in a1381eea7c7d (graft: do not use `.remove` on a smart set (regression), 2014-04-28). Differential Revision: https://phab.mercurial-scm.org/D7806

graft: don't remove from a list in a loop This addresses a TODO added in a1381eea7c7d (graft: do not use `.remove` on a smart set (regression), 2014-04-28). I couldn't measure any speedup. Differential Revision: https://phab.mercurial-scm.org/D7805

tests: avoid grafting the same change over and over The test case added in a1381eea7c7d (graft: do not use `.remove` on a smart set (regression), 2014-04-28) added a test case that grafted the same change (renaming 'a' to 'b') three times over. It had description "graft works on complex revset", but AFACT, all that it cared about was that some ancestor of the working copy was in the set of revisions to graft. So this patch changes the test to do that instead. (I plan to later make it so that grafting these renames on top of each won't create the empty commits they currently create.) Differential Revision: https://phab.mercurial-scm.org/D7804

py3: byteify some `ui.configbool()` parameters This popped up in 8042856c90b6. Differential Revision: https://phab.mercurial-scm.org/D7817

rust-discovery: type alias for random generator seed It just makes our life easier Differential Revision: https://phab.mercurial-scm.org/D7715

tests: split out another ~1/2 of test-graft.t The tests involving renames were also quite independent from the rest. Differential Revision: https://phab.mercurial-scm.org/D7803

tests: split out ~1/3 of test-graft.t test-graft.t is ~2400 lines and takes 34s to run. This patch moves the last ~1/3 of it to a separate file. The parts now run in 22s + 13s. On top of that, we can remove the #testcases from the old file, so it's only 22s + 2*13s instead of the 2*34s it was before. Differential Revision: https://phab.mercurial-scm.org/D7802

overlwayworkingctx: remove doubly bad reference to wrapped ctx for copies `_wrappedctx` lives on overlwayworkingctx, not on the repo object, so we should access it as `._wrappedctx`, not `._repo._wrappedctx`. More importantly, the overlayworkingctx is relative to its base, not including it, so the copies returned should not include copies made in the base. Differential Revision: https://phab.mercurial-scm.org/D7801

movedirstate: get copies from dirstate before setting parents Setting dirstate parents can modify the copies recorded in the dirstate when there are two dirstate parents. I don't think we ever call movedirstate() when there is more than one parent, but it seems clearer to get the copies from the dirstate first anyway. Differential Revision: https://phab.mercurial-scm.org/D7800

fix: convert clang-format-ignorelist to use wildcards It's very brittle to specify the exact filenames for these vendored/thirdparty libraries; when there's an upgrade, it's likely not going to be updated. Differential Revision: https://phab.mercurial-scm.org/D7619

packaging: update Ubuntu docker build dependencies to Python 3 Changeset 7574ccd87200f02e updated the Debian docker builds to Python 3. In doing so, the Ubuntu docker-based builds broke, as the Dockerfile does not include the new dependencies as specified in `contrib/packaging/debian/control`. This commit changes the dependencies in the Ubuntu Dockerfile template to their Python 3 equivalents, fixing `make docker-ubuntu-bionic`. Differential Revision: https://phab.mercurial-scm.org/D7810

mmap: add a size argument to mmapread With this argument, we can control the size of the mmap created. (previously it was always the whole file. Differential Revision: https://phab.mercurial-scm.org/D7808

revlogutils: move the NodeMap class in a dedicated nodemap module We will introduce more nodemap related code, so it make sense to move the existing code in that module first. Differential Revision: https://phab.mercurial-scm.org/D7809

rcutil: drop the `defaultrcpath()` method (API) The resource based code can service py2, py3 and an oxidized executable, so there's no reason to leave this around. It was flagged as an API change when it was introduced, so flagging it again. Differential Revision: https://phab.mercurial-scm.org/D7777

ui: add the ability to apply `defaultrc` configs from resources We will want the ability to cat out these resources, but the same would apply to templates, so I'm going to wait for the dust to settle on that. Reading the default config directly from the filesystem is still in place for now. Differential Revision: https://phab.mercurial-scm.org/D7776

ui: refactor `readconfig()` into a form that can consume resources The old form can't completely go away, because files outside of packages still need to be read. The name passed in here is a tuple of `package name, resource` as needed by the resource API. I like the idea of stating the config file is embedded in the executable by listing is as `exe!package.resource`. This would be consistent with how `debuginstall` points to the executable for the python executable, lib, and installed modules. While in practice the filesystem path is available from the backing ResourceReader when the resource is opened, it is a relative path on py2 and absolute on py3. Further, while this would show in the `hg config` output for each option if set as such here, it doesn't show in the `reading from...` line when `--debug` is used. The file isn't actually open where that prints, so there's no way I see to get that info there. So I opted for the simple prefix to distinguish resources from files. Differential Revision: https://phab.mercurial-scm.org/D7775

resourceutil: implement `contents()` to iterate over resources in a package Differential Revision: https://phab.mercurial-scm.org/D7774

resourceutil: implement `is_resource()` This will be needed when iterating resources. Differential Revision: https://phab.mercurial-scm.org/D7773

resourceutil: don't limit resources to the `mercurial` package This should make things a little clearer, in that it now requires the full package name to access a resource. But the real motivation is that `extensions._disabledpaths()` walks the `hgext` directory looking for bundled extensions. This in turn feeds, among other things: 1) Listing disabled extensions in `hg help extensions` 2) Indicating that an unknown command is in a non-enabled extension 3) Displaying help for non-enabled extensions 4) Generating documentation 5) Announcing LFS is auto-enabled (or not) when cloning from an LFS source The filesystem based ResourceReader will happily return *.py and *.pyc, but the one supplied by PyOxidizer doesn't. Presumably we can change that. The only other idea I had here is for setup.py to generate a text file containing the list of extensions, but that doesn't seem great when running from source. Differential Revision: https://phab.mercurial-scm.org/D7772

setup: include `defaultrc` in the package list This is needed to access the config files via the resourceutil framework, which is needed to embed the config files when uing PyOxidizer. Differential Revision: https://phab.mercurial-scm.org/D7771

tests: update test-releasenotes-formatting.t output The output seems to be left outdated because running this test requires fuzzy-wuzzy which is not generally installed. Differential Revision: https://phab.mercurial-scm.org/D7723

tests: fix a copy/paste name duplication in storage.py Flagged by PyCharm as a redeclared name without usage, because the name matches the function immediately preceding it. But since this differs by doing regular reads instead of raw reads, I assume we still want it. But I don't see either function being called anywhere. Differential Revision: https://phab.mercurial-scm.org/D7769

commit: change default `editor` parameter to None Flagged by PyCharm as a boolean not being callable later where it is used. There's no actual bug here because of `if editor` checks before using. Differential Revision: https://phab.mercurial-scm.org/D7768

cleanup: drop redundant character escapes outside of `[]` Flagged by PyCharm. `@`, `:`, `<`, `>`, and `{` aren't special characters. (I'm a bit surprised that it doesn't also want to unescape `}` in schemes.py.) Differential Revision: https://phab.mercurial-scm.org/D7767

cleanup: drop redundant character escapes from `[]` character sets Flagged by PyCharm. Most special characters lose their special meaning inside square brackets. The `-` in util.py doesn't need to be escaped because it is the last character in the set. Differential Revision: https://phab.mercurial-scm.org/D7766

cleanup: replace contiguous spaces in regex patterns with an explicit count Flagged by PyCharm, this form should be more readable. Differential Revision: https://phab.mercurial-scm.org/D7765

mq: avoid using `__file__` to compare modules This fixes ~70 tests when using an oxidized executable, bringing the current failure count down to 92 (with 110 skips) when templates/ is copied next to the executable. Differential Revision: https://phab.mercurial-scm.org/D7778

util: avoid referencing `time.clock()` on Windows when missing (issue6238) It's been removed in 3.8, and issues a deprecation warning since 3.3. Differential Revision: https://phab.mercurial-scm.org/D7780

py3: replace `time.clock()` with `time.perf_counter()` The former was removed in 3.8. The util function uses `time.clock()` if the latter is unavailable. Differential Revision: https://phab.mercurial-scm.org/D7779

tests: fix a "naked exception" issue in test-remotefilelog-prefetch.t Differential Revision: https://phab.mercurial-scm.org/D7811

rust-index: add a `experimental.rust.index` option to use the wrapper Now we can start putting this wrapper on the test and benchmark grill. Differential Revision: https://phab.mercurial-scm.org/D7660

rust-index: use the new method in shortesthexnodeidprefix This code can now run with both a Rust or a C index. Differential Revision: https://phab.mercurial-scm.org/D7659

rust-index: expose a method to retrieve the C index The code for `shortesthexnodeidprefix` need to access the actual C index. For now we grant its wish and expose a method to do so. Once we have the nodemap in Rust, we will be able to implement the same feature from rust and we will be able to drop this method. Differential Revision: https://phab.mercurial-scm.org/D7658

rust-index: handle `MixedIndex` in `pyindex_to_graph` On the long run we will want to implement the Graph trait directly in Rust, but for now we take the path with the least amount of change to focus on the coming persistent NodeMap code. We test this new code through with the lazy ancestors code. Differential Revision: https://phab.mercurial-scm.org/D7657

rust-index: make it possible to clone the struct referencing the C index If we are to hand over the C index object to other code, we need to be able to create a new python reference to it. Differential Revision: https://phab.mercurial-scm.org/D7656

match: resolve filesets against the passed `cwd`, not the current one This allows filesets to be resolved relative to `repo.root`, the same as other patterns are since f02d3c0eed18. The current example in contrib/ wasn't working from the tests directory because of this. Differential Revision: https://phab.mercurial-scm.org/D7570

fix: correct the clang-format example hgrc so that it actually works There are three changes here: * Remove -i from `command`, it causes fix to eat your file and empty it out * Add `set:` to pattern, otherwise this is interpreted as just a glob * Switch `listfile:` to `include:`; `listfile:` is relative to the current working directory, while `include:` is relative to the repo root. This makes it so that you don't receive errors when running outside of the repo root about being unable to find the file. Differential Revision: https://phab.mercurial-scm.org/D7618

inno: drop support for Windows 95/98/ME Differential Revision: https://phab.mercurial-scm.org/D7696

rust-matchers: fixing cargo doc Block quotes marked as `ignore` are still parsed as Rust source examples. Differential Revision: https://phab.mercurial-scm.org/D7783

rust-core: extracted a revlog submodule This moves fundamental definitions from the top of the crate to the newly created `revlog` submodule and reexports them for easy compatibility. As we are about to add new features to this crate, we felt it will improve clarity, and moreso if `ancestors` and `dagops` would become submodules of `revlog`. Differential Revision: https://phab.mercurial-scm.org/D7782

phases: make the working directory consistently a draft Before this change, `hg log -r 'wdir() and public()'` would return it. Differential Revision: https://phab.mercurial-scm.org/D7705

rust-core: updated copyright notice The intention here is to put an email address that actually works (I did have full personal copyrights with my former affiliation). It is also an opportunity to acknowledge that I've not been the only one to work on this file. Differential Revision: https://phab.mercurial-scm.org/D7781

tests: fix failing doctest in match.py by adding dummy auditor It was failing with "OSError: [Errno 13] Permission denied: '/root/sub'". It has been failing since 8b1a9ba375e5 (match: make sure `root` argument is always an absolute path (API), 2019-12-13). I don't know why I didn't notice it before sending that patch. Differential Revision: https://phab.mercurial-scm.org/D7731

remotefilelog: actually fix (and test this time) a bytes vs str issue I attempted this in 94670e124d29 but we didn't actually have any tests for this function it seems, and I mistook "it works now" for it being fixed (when what had actually happened was that my cache had populated enough that I didn't trigger the minimum threshold on my reattempts). Differential Revision: https://phab.mercurial-scm.org/D7785

merge with stable

revset: drop some unused code in the `remote` revset PyCharm flagged the `revs = [..]` as an unused assignment. But then neither `revs` nor `checkout` is used, and I can't see where `hg.addbranchrevs()` has external side effects. Differential Revision: https://phab.mercurial-scm.org/D7764

tests: avoid using a list comprehension to fill a list with fixed values Flagged by PyCharm as an unused assignment for the variable in the list. Differential Revision: https://phab.mercurial-scm.org/D7763

histedit: avoid using a list comprehension to fill a list with fixed values Flagged by PyCharm as an unused assignment for the variable in the list. Differential Revision: https://phab.mercurial-scm.org/D7762

zstandard: vendor python-zstandard 0.13.0 Version 0.13.0 of the package was just released. It contains an upgraded zstd C library which can result in some performance wins, official support for Python 3.8, and a blackened code base. There were no meaningful code or functionality changes in this release of python-zstandard: just reformatting and an upgraded zstd library version. So the diff seems much larger than what it is. Files were added without modifications. The clang-format-ignorelist file was updated to reflect a new header file in the zstd distribution. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D7770

hgweb: delete a local variable instead of setting to `None` The previous code was flagged by PyCharm as an unused variable assignment. Differential Revision: https://phab.mercurial-scm.org/D7761

tests: drop unused local variable assignments in linelog Flagged by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7760

ui: delete local variables instead of setting to `None` The previous code was flagged by PyCharm as an unused variable assignment. Differential Revision: https://phab.mercurial-scm.org/D7759

tests: drop unused local variable assignments in remotefilelog Flagged by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7758

fastannotate: drop unused local variable assignments Flagged by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7757

fsmonitor: drop an unused local variable assignment Flagged by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7756

mq: drop an unused variable assignment Flagged by PyCharm. This variable is the index of a `for` loop below. Differential Revision: https://phab.mercurial-scm.org/D7755

hgweb: delete local variable instead of setting it to `None` The previous code was flagged by PyCharm as an unused variable assignment. Differential Revision: https://phab.mercurial-scm.org/D7754

annotate: avoid using a list comprehension to fill a list with fixed values Flagged by PyCharm as an unused assignment for the variable in the list. Differential Revision: https://phab.mercurial-scm.org/D7753

fsmonitor: drop an unused local variable Flagged by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7752

absorb: avoid using a list comprehension to fill a list with fixed values Flagged by PyCharm as an unused assignment for the variable in the list. Differential Revision: https://phab.mercurial-scm.org/D7751

run-tests: ensure the script exits when it fails to change directories Caught by PyCharm as an unused variable assignment. This regressed in c496e8c14b9e. Differential Revision: https://phab.mercurial-scm.org/D7750

shelve: drop an unused variable assignment Caught by PyCharm. This stopped being used in a3b285882724. Differential Revision: https://phab.mercurial-scm.org/D7749

hgweb: drop an unused variable assignment Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7748

revlog: drop an unused variable assignment Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7747

rebase: drop an unused variable assignment Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7746

polib: drop an unused local function Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7745

perf: drop an unused variable assignment Caught by PyCharm. A different formatter is created below in `_displaystats()`. Differential Revision: https://phab.mercurial-scm.org/D7744

perf: drop an unnecessary `pass` Differential Revision: https://phab.mercurial-scm.org/D7743

perf: fix the time measurement for pathcopies relative to p2 PyCharm flagged this as unused. But it seems more likely that it was intended to reset the time for the p2 path. Differential Revision: https://phab.mercurial-scm.org/D7742

githelp: drop unused variable assignments Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7741

debug: drop unused variable assignments Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7740

changegroup: drop an unused variable assignment Caught by PyCharm. The variable is assigned again below before it is used. Differential Revision: https://phab.mercurial-scm.org/D7739

bugzilla: drop an unused variable assignment Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7738

ancestor: drop another unused variable assignment Also caught by PyCharm. The only time this is used is immediately after another assignment below this. Differential Revision: https://phab.mercurial-scm.org/D7737

ancestor: drop an unused local variable assignment Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7736

cleanup: drop unused import from pywatchman Caught by PyCharm. Differential Revision: https://phab.mercurial-scm.org/D7735

narrow: move `testedwith` after module imports PyCharm flagged this as a PEP 8 violation, for module level imports not being at the top of the file. Differential Revision: https://phab.mercurial-scm.org/D7734

match: don't util.normpath() cwd The problem here is that `util.normpath()` calls `util.pconvert()`, which switches to Unix style separators. That results in two test failures like this since 5685ce2ea3bf: --- c:/Users/Matt/hg/tests/test-globalopts.t +++ c:/Users/Matt/hg/tests/test-globalopts.t.err @@ -89,7 +89,7 @@ [255] $ hg -R b ann a/a abort: a/a not under root '$TESTTMP/b' - (consider using '--cwd b') + (consider using '--cwd ..\$TESTTMP\b') [255] $ hg log abort: no repository found in '$TESTTMP' (.hg not found)! ERROR: test-globalopts.t output changed Martin originally had `os.path.normpath()` (which *would* work here too), but changed it during review. He didn't remember why he thought any form is needed here. Most uses simply pass '' or `repo.getcwd()`, so these should generally be in local format anyway. It seems better if `cwd` and `root` use consistent styles here. Differential Revision: https://phab.mercurial-scm.org/D7725

tests: convert the `root` arg of matchmod.match() to local path separators This fixes tests that broke with 8b1a9ba375e5, complaining that "X not under root /repo". The vast majority of real uses are to pass `repo.root`, which is normalized by `wdirvfs.base` being set to the result of `os.path.realpath()`. Failure to convert looks like this: --- c:/Users/Matt/hg/tests/test-match.py.out +++ c:/Users/Matt/hg/tests/test-match.py.err @@ -0,0 +1,48 @@ +ERROR: testVisitchildrensetGlob (__main__.IncludeMatcherTests) + +Traceback (most recent call last): + File "c:\Users\Matt\hg\tests\test-match.py", line 180, in testVisitchildrensetGlob + m = matchmod.match(b'/repo', b'', include=[b'glob:dir/z*']) + File "c:\Users\Matt\hg\mercurial\match.py", line 271, in match + kindpats = normalize(include, b'glob', root, cwd, auditor, warn) + File "c:\Users\Matt\hg\mercurial\match.py", line 322, in _donormalize + pat = pathutil.canonpath(root, cwd, pat, auditor=auditor) + File "c:\Users\Matt\hg\mercurial\pathutil.py", line 251, in canonpath + _(b"%s not under root '%s'") % (myname, root), hint=hint +Abort: dir/z* not under root '/repo' +ERROR: testVisitdirGlob (__main__.IncludeMatcherTests) Differential Revision: https://phab.mercurial-scm.org/D7724

commitablectx: fix the default phase The code was ignoring the config option. Since the config option will control the actual phase of the committed result, we better take it in account early. Differential Revision: https://phab.mercurial-scm.org/D7726

fix: fix handling of merge commits by using overlayworkingctx Most of this code was conceptually copied from what rebase does, with one small difference: hgext.rebaserev.rebase uses branchmerge=True, while I had to use branchmerge=False, or else it got really confused about updating to the same revision in some situations. I believe that the difference is that rebase is always dealing with *some* form of update - it never gets to mergemod.update if the source and destination are the same, while we can encounter that situation with fix. This may imply that this code has some issues with named branches that should be investigated. Differential Revision: https://phab.mercurial-scm.org/D7703