rust: sort dependencies entries in Cargo.toml I should probably write a test to enforce this... Differential Revision: https://phab.mercurial-scm.org/D6398

py3: make contrib/testparseutil.py to work on str(unicodes) contrib/check-code work on unicodes and call functions from testparseutil.py which before this patch used to work on bytes. This path removes that inconsistency and make testparseutil.py work on unicodes. This makes test-check-code.t and test-contrib-check-code.t work on Python 3 again. Differential Revision: https://phab.mercurial-scm.org/D6391

rust-filepatterns: call new Rust implementations from Python This change adds the import to the `rust-cpython` bindings and uses them when appropriate. A wrapper function has been defined in the case of `_regex` to keep this patch simple. Differential Revision: https://phab.mercurial-scm.org/D6273

rust-filepatterns: add `rust-cpython` bindings for `filepatterns` This change adds the `rust-cpython` interface for top-level functions and exceptions in the filepatterns module. Contrary to the Python implementation, this tries to have finer-grained exceptions to allow for better readability and flow control down the line. Differential Revision: https://phab.mercurial-scm.org/D6272

rust-filepatterns: add a Rust implementation of pattern-related utils This change introduces Rust implementations of two functions related to pattern handling, all located in `match.py`: - `_regex` - `readpatternfile` These utils are useful in the long-term effort to improve `hg status`'s performance using Rust. Experimental work done by Valentin Gatien-Baron shows very promising improvements, but is too different from the current Mercurial core code structure to be used "as-is". This is the first - albeit very small - step towards the code revamp needed down the line. Two dependencies were added: `regex` and `lazy_static`. Both of them will be useful for a majority of the Rust code that will be written, are well known and maintained either by the Rust core team, or by very frequent contributors. Differential Revision: https://phab.mercurial-scm.org/D6271

exchange: don't take wlock if bookmarks are stored in .hg/store/ If bookmarks are stored in .hg/store/, there is no need for the wlock(). Differential Revision: https://phab.mercurial-scm.org/D6388

bookmarks: keep bookmarks in .hg/store if new config set Bookmarks storage consists of two parts: (1) the set of bookmarks and their positions, and (2) the current bookmark. The former can get updated by exchange, while the latter cannot. However, they are both stored in directly .hg/ and protected by repo.wlock(). As a result, ugly workarounds were needed. This patch introduces a new config option to store the set of bookmarks and their positions in .hg/store/ but still storing the current bookmark directory in .hg/. The config option only takes effect at repo creation time. It results in a new requirement being set. Differential Revision: https://phab.mercurial-scm.org/D6387

bookmark: also make bookmark cache depends of the changelog Since the changelog is also used during the parsing of bookmark data, it should be listed as a file cache dependency. This fix the race condition we just introduced a test for. This is a simple fix that might lead bookmark data to be invalidated more often than necessary. We could have more complicated code to deal with this race in a more "optimal" way. I feel it would be unsuitable for stable. In addition, the performance impact of this is probably minimal and I don't foresee the more advanced fix to actually be necessary.

localrepo: grab mixedrepostorecache class from 526750cdd02d On default, Martin von Zweigbergk <martinvonz@google.com> introduced a more advance filecache decorator. I need this decorator to fix a bug on stable. So I am grafting the relevant part of 526750cdd02d.

bookmark: add a test for a race condition on push Bookmark pointing to unknown nodes are ignored. Later these ignored bookmarks are dropped when writing the file back on disk. On paper, this behavior should be fine, but with the current implementation, it can lead to unexpected bookmark deletions. In theory, to make sure writer as a consistent view, taking the lock also invalidate bookmark data we already loaded into memory. However this invalidation is incomplete. The data are stored in a `filecache` that preserve them if the bookmark related file are untouched. In practice, the bookmark data in memory also depends of the changelog content, because of the step checking if the bookmarks refers to a node known to the changelog. So if the bookmark data were loaded from an up to date bookmark file but filtered with an outdated changelog file this go undetected. This condition is fairly specific, but can occurs very often in practice. We introduce a test recreating the situation. The test comes in an independant changeset to show it actually reproduce the situation. The fix will come soon after. A large share of the initial investigation of this race condition was made by Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>.

test: properly gate a zstd section This part of the test can't run if zstd is not available. This was caught by --pure test (who don't support zstd).

test: update test for expected test output In 1fac9b931d46 as new test session was introduced. It did not take in account some part that only ran for pure. The test is now fixed.

log: flag topo-sorted set as such This isn't required right now, but revs.istopo() should return True.

copies: fix duplicatecopies() with overlay context The reasoning for this check is in 78d760aa3607 (duplicatecopies: do not mark items not in the dirstate as copies, 2013-03-28). The check was then moved to workingfilectx in 754b5117622f (context: add workingfilectx.markcopied, 2017-10-15) and no corresponding check was added later when overlayworkingfilectx was added. Rather than adding the check there, this patch adds a more generic check on the callers side and removes the check in workingfilectx.markcopied(). Differential Revision: https://phab.mercurial-scm.org/D6380

tests: demonstrate crash when rebasing across copy with --collapse As reported by timeless. Differential Revision: https://phab.mercurial-scm.org/D6379

exthelper: add some semi-useful trace logs It'd be nice to make the trace functions a little better-named in the output, but I'm not sure how much better we can do without overhead. This at least lets you see if a single reposetup function is eating all the time or if it's spread over all of them. I needed this because Google's uber-extension has a long load time and I wasn't sure where the problem was. Differential Revision: https://phab.mercurial-scm.org/D6381

help: add missing blank line, making "revlog-compression" show up Differential Revision: https://phab.mercurial-scm.org/D6386

tests: fix share test to actually share the repo "repo2" is clearly meant to be a share from "repo1" but without sharing bookmarks. However, `hg unshare` was called in the repo, so it had become completely unrelated and thus not testing what it was supposed to test. Differential Revision: https://phab.mercurial-scm.org/D6385

tests: separate out bookmarks tests from test-share.t Differential Revision: https://phab.mercurial-scm.org/D6384

bookmarks: use vfs.tryread() instead of reimplementing it Differential Revision: https://phab.mercurial-scm.org/D6383

bookmarks: use context manager when writing files Differential Revision: https://phab.mercurial-scm.org/D6382

bisect: do not crash with rewritten commits

log: add config for making `hg log -G` always topo-sorted I (and everyone else at Google) have an log alias that adds graph mode and templating. I have another one that builds on the first and also restricts the set of revisions to only show those I'm most likely to care about. This second alias also adds topological sorting. I still sometimes use the first one. When I do, it very often bothers me that it's not topologically sorted (branches are interleaved). This patch adds a config option for always using topological sorting with graph log. The revision set is sorted eagerly, which seems like a bad idea, but it doesn't seem to make a big difference in the hg repo (150ms). I initially tried to instead wrap the user's revset in sort(...,topo), but that seemed much harder. Differential Revision: https://phab.mercurial-scm.org/D6331

log: remove an unnecessary "and opts.get('rev')" condition As Yuya pointed out, the condition is unnecessary since revs.isdescending() would be true if --follow without --rev. Differential Revision: https://phab.mercurial-scm.org/D6372

graphmod: remove support for graph lines mixing parent/grandparent styles (BC) Currently, if the configuration for a graph edge draw style has multiple bytes (at least on python2), it is interpreted as "this is a request to draw the line partially in the style of the parent, partially in the style of the grandparent". This precludes the configuration handling unicode characters (which trigger the `len > 1` check, at least on python2), and I believe was part of the reason that beautifygraph was written the way it was. Talking with the person who implemented this, it appears to have been to achieve feature parity with the rendering of the smartlog extension. I suspect that this isn't actually used outside of that situation, so I think that we can remove it without much issue. This will make it so that multi-character edges are possible, and render any existing configuration that uses this feature with these multiple characters. This is *not* going to adjust the width of everything to make it line up correctly, please see the test that's being modified in this changeset for an example of how the previous configuration now renders. Note also that the previous configuration seems to have been broken, or at least it was behaving in a really non-obvious way - it was avoiding the grandparent character(s) when it should have been displaying them! This is why so many "!" characters changed to "3."; I don't know if this was intentional. Differential Revision: https://phab.mercurial-scm.org/D5112

py3: add 5 new passing tests Differential Revision: https://phab.mercurial-scm.org/D6378

py3: add a r'' to prevent transformer adding b'' # skip-blame because just r'' prefix Differential Revision: https://phab.mercurial-scm.org/D6377

rust-dirstate: call parse/pack bindings from Python A future patch will need to address the issue of Rust module policy, to avoid having ugly duplicate imports and conditionals all over the place. As the rewrite of dirstate in Rust progresses, we will need fewer of those "contact points". Differential Revision: https://phab.mercurial-scm.org/D6350

rust-dirstate: add rust-cpython bindings to the new parse/pack functions This allows for Python code to call `parse/pack_dirstate` transparently. These bindings are heavy given the relatively simple task, as they are bound to implementation details of both the C and Python code. They will be slimmed down in future patches and eventually completely removed once more of the dirstate code has been refactored/rewritten in Rust. Both functions emulate the mutate-on-loop style of the Python and C implementations by looping over changed items in the compatibility layer, instead of at the core functions. Differential Revision: https://phab.mercurial-scm.org/D6349

rust-dirstate: add rust implementation of `parse_dirstate` and `pack_dirstate` Working towards the goal of having a complete Rust implementation of `hg status`, these two utils are a first step of many to be taken to improve performance and code maintainability. Two dependencies have been added: `memchr` and `byteorder`. Both of them have been written by reputable community members and are very mature crates. The Rust code will often need to use their byte-oriented functions. A few unit tests have been added and may help future development and debugging. In a future patch that uses `parse_dirstate` to stat the working tree in parallel - which neither the Python nor the C implementations do - actual performance improvements will be seen for larger repositories. Differential Revision: https://phab.mercurial-scm.org/D6348

changelog: define changelogrevision.p[12]copies for null revision Looks like I missed these in 5382d8f8530b (changelog: parse copy metadata if available in extras, 2017-12-27). `hg debugp[12]copies -r null` fails before this patch. Differential Revision: https://phab.mercurial-scm.org/D6376

copies: write empty entries in changeset when also writing to filelog When writing to both changeset and filelog (during transition), we don't want the reader to waste time by falling back to reading from the filelog when there is no copy metadata. Let's write out empty copy metadata instead (the read path is already prepared for this case). Thanks to Greg for pointing this out. Differential Revision: https://phab.mercurial-scm.org/D6306

rebase: hide help for revisions.Predicates._destautoorphanrebase

unshelve: add space to help

context: default to using branch from dirstate only in workingctx Same reasoning as previous commits: only the workingctx should know about the dirstate. committablectx now seems free of dirstate references. Differential Revision: https://phab.mercurial-scm.org/D6374

context: let caller pass in branch to committablectx.__init__() committablectx.__init__() currently looks up the branch from the dirstate unless it's passed in the extras. memctx.__init__() has a branch argument, but since committablectx.__init__() doesn't accept it, it lets that constructor look up the branch from the dirstate before it overwrites it, which seems awkward. Differential Revision: https://phab.mercurial-scm.org/D6366

context: move contents of committablectx.markcommitted() to workingctx Same reasoning as previous commits: this function updates the dirstate. By not updating the dirstate here, we also fix the close-head test. Differential Revision: https://phab.mercurial-scm.org/D6365

tests: demonstrate that close-head command updates working copy The help text for the command says "...it doesn't change the working directory", so I don't think this is intentional. Differential Revision: https://phab.mercurial-scm.org/D6364

context: move walk() and match() overrides from committablectx to workingctx Same reasoning as previous commit: these functions update the dirstate. Differential Revision: https://phab.mercurial-scm.org/D6363

context: move flags overrides from committablectx to workingctx These read from the dirstate, so they shouldn't be used in other subclasses. Differential Revision: https://phab.mercurial-scm.org/D6362

context: reuse changectx._copies() in all but workingctx This moves the dirstate-specific _copies() implementation from committablectx into workingctx where it should be (I think all dirstate-specific stuff should be moved into workingctx). The part of changectx._copies() that is for producing changeset-wide copy dicts from the filectxs is moved into basectx so it's reused by the other subclasses. The part of changectx._copies() that's about reading copy information from the changeset remains there. This fixes in-memory rebase (and makes `hg convert` able to write copies to changesets). Differential Revision: https://phab.mercurial-scm.org/D6219

overlayworkingctx: don't include added-then-deleted files in memctx If a file (such as a .orig file) is temporarily added to the overlayworkingctx and then deleted, it's still going to be in the _cache dict. In tomemctx(), we created the list of files from _cache.keys(), so the memctx.files() would include the temporary file. That was fine because the list of files was only used in localrepo.commitctx() (I think), where there's an extra filtering of incorrectly removed files (annotated with an inaccurate "update manifest" comment). I'd like to call memctx.files() in another case, but first we need to make it accurate. Differential Revision: https://phab.mercurial-scm.org/D6361

tests: demonstrate loss of changeset copy metadata on rebase Differential Revision: https://phab.mercurial-scm.org/D6360

overlaycontext: allow calling copydata() on clean context We should just report no copy if the context is clean. Differential Revision: https://phab.mercurial-scm.org/D6358

tests: demonstrate another failure with in-memory rebase and copies This is a similar to dd1ab72be983 (test: demonstrate crash with in-memory rebase and copies, 2019-03-14). The new failure started with 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). It happens in the call to mergemod.update() on rebase.py:1268 where we call mergemod.update() to graft a node. Since the mergecopies() rewrite, that calls _related() with the filectx from the overlaywctx instead of a filectx from the changectx where the file was last modified. Either should be fine, so I don't think that's a bug. Differential Revision: https://phab.mercurial-scm.org/D6357

commit: fix a typo ("form p1" -> "from p1") Differential Revision: https://phab.mercurial-scm.org/D6375

automation: initial support for running Linux tests Building on top of our Windows automation support, this commit implements support for performing automated tasks on remote Linux machines. Specifically, we implement support for running tests on ephemeral EC2 instances. This seems to be a worthwhile place to start, as building packages on Linux is more or less a solved problem because we already have facilities for building in Docker containers, which provide "good enough" reproducibility guarantees. The new `run-tests-linux` command works similarly to `run-tests-windows`: it ensures an AMI with hg dependencies is available, provisions a temporary EC2 instance with this AMI, pushes local changes to that instance via SSH, then invokes `run-tests.py`. Using this new command, I am able to run the entire test harness substantially faster then I am on my local machine courtesy of access to massive core EC2 instances: wall: 16:20 ./run-tests.py -l (i7-6700K) wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge wall: 8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge wall: 8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge wall: 3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge wall: 3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge ~3 minute wall time to run pretty much the entire test harness is not too bad! The AMIs install multiple versions of Python. And the run-tests-linux command specifies which one to use: automation.py run-tests-linux --python system3 automation.py run-tests-linux --python 3.5 automation.py run-tests-linux --python pypy2.7 By default, the system Python 2.7 is used. Using this functionality, I was able to identity some unexpected test failures on PyPy! Included in the feature is support for running with alternate filesystems. You can simply pass --filesystem to the command to specify the type of filesystem to run tests on. When the ephemeral instance is started, a new filesystem will be created and tests will run from it: wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs wall: 4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs wall: 4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4 We also support multiple Linux distributions: $ automation.py run-tests-linux --distro debian9 total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4% $ automation.py run-tests-linux --distro ubuntu18.04 total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4% $ automation.py run-tests-linux --distro ubuntu18.10 total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9% $ automation.py run-tests-linux --distro ubuntu19.04 total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0% Debian and Ubuntu are supported because those are what I use and am most familiar with. It should be easy enough to add support for other distros. Unlike the Windows AMIs, Linux EC2 instances bill per second. So the cost to instantiating an ephemeral instance isn't as severe. That being said, there is some overhead, as it takes several dozen seconds for the instance to boot, push local changes, and build Mercurial. During this time, the instance is largely CPU idle and wasting money. Even with this inefficiency, running tests is relatively cheap: $0.15-$0.25 per full test run. A machine running tests as efficiently as these EC2 instances would cost say $6,000, so you can run the test harness a >20,000 times for the cost of an equivalent machine. Running tests in EC2 is almost certainly cheaper than buying a beefy machine for developers to use :) # no-check-commit because foo_bar function names Differential Revision: https://phab.mercurial-scm.org/D6319

automation: move image operations to own functions An upcoming commit will need this functionality with slightly different values and it is enough code to not want to duplicate. Let's refactor into standalone functions so it can be reused. Differential Revision: https://phab.mercurial-scm.org/D6318

automation: add --version argument to build-all-windows-packages This lets us pass a version string through when building all Windows packages, just like we can do with the individual commands which produce installers. Differential Revision: https://phab.mercurial-scm.org/D6317

automation: do a force push to synchronize We don't know what the state of the remote is. Force pushing will be more resilient. Differential Revision: https://phab.mercurial-scm.org/D6316

automation: add check that hg source directory is a repo Synchronizing from e.g. source distributions is not yet supported. Let's add a check so we fail with an error message indicating such. Differential Revision: https://phab.mercurial-scm.org/D6315

automation: shore up rebooting behavior There was a race condition in the old code. Use instance.stop()/instance.start() to eliminate it. As part of debugging this, I also found another race condition related to PowerShell permissions after the reboot. Unfortunately, I'm not sure the best way to work around it. I've added a comment for now. Differential Revision: https://phab.mercurial-scm.org/D6288

automation: wait longer for WinRM connection I got a few timeouts waiting for only 120s for the WinRM connection to become available. Increasing to 180s seems to fix. I guess AWS isn't as consistent as I would like :( Differential Revision: https://phab.mercurial-scm.org/D6287

automation: wait for instance profiles and roles Otherwise there is a race condition between creating the resources and us attempting to use them / them becoming available. The role waiter API was recently introduced, so we had to upgrade the boto3 package to get it. Other packages were also updated to latest versions just because. Even with this change, I still run into issues with the IAM instance profile not being available when we attempt to create an EC2 instance using a just-created profile. I'm not sure what's going on. Possibly a bug on Amazon's end. But the new behavior is "more correct." Differential Revision: https://phab.mercurial-scm.org/D6286

automation: don't create resources when deleting things Otherwise running these commands can result in resources being created. In the case of `purge-ec2-resources`, we will create resources only to delete them immediately afterwards! With this change, `purge-ec2-resources` now no-ops if no resources exist. # no-check-commit because foo_bar function name Differential Revision: https://phab.mercurial-scm.org/D6285

automation: detach policies before deleting role You can't delete an IAM role that has attached policies. With this change, the purge-ec2-resources command now works. Differential Revision: https://phab.mercurial-scm.org/D6284

automation: only iterate over our AMIs We can't delete AMIs that we don't own. Iterating over other AMIs won't work and slows down execution. Differential Revision: https://phab.mercurial-scm.org/D6283

remotefilelog: move most setup from onetimesetup() to uisetup() All the wrappers moved in this patch check if remotefilelog is enabled before they change behavior, so it's safe to always wrap. Differential Revision: https://phab.mercurial-scm.org/D6334

remotefilelog: move most functions in onetimeclientsetup() to top level This is how most extensions seem to do it. It makes sure we don't accidentally depend on the captured ui instance. Differential Revision: https://phab.mercurial-scm.org/D6333

tests: avoid the word "dirty" to mean "not a descendant of merge base" The term "dirty" is no longer used in the code since 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). Differential Revision: https://phab.mercurial-scm.org/D6373

releasenotes: add a file in which to record release notes I've just spent a few very boring hours going through the changelog for the 5.0 release (829 commits). We only had 5 commits that used the syntax that the release notes extension expects. This commit adds a file in which we can record important changes. The file should preferably be edited in the patch that makes the important change, but it can also be edited after (I think this is an important benefit compared to the release notes extension). I'm thinking that we can rename the file from "next" to "5.1" or something when it's time, and then we'd create a new "next" file on the default branch. I've used the syntax that we use on the our wiki in the template, but I don't care much that we use any valid syntax at all. The idea is mostly to record important changes when they happen. I expect that some copy editing will be needed at release time anyway. Differential Revision: https://phab.mercurial-scm.org/D6332

record: avoid modifying the matcher passed as a method parameter No problem observed, but I remember the previous pattern causing problems with largefiles and/or subrepos. This special matcher was added in 419ac63fe29c, so directly modifying the `fail` callback was probably an oversight in 44611ad4fbd9. Differential Revision: https://phab.mercurial-scm.org/D6371

sslutil: add support for SSLKEYLOGFILE to wrapsocket I recently learned of a Firefox/Chrome feature that allows wiresharking otherwise-TLS'd network connections. Gloriously, there's a pypi module that enables this same feature on Python, so let's add support for it to Mercurial in case we need to wireshark some HTTPs connections. Differential Revision: https://phab.mercurial-scm.org/D6343

phabricator: add custom vcr matcher to match request bodies Currently when the phabricator extension's conduit output changes the tests don't notice since the default vcr matcher only matches on 'method' and 'uri', not the body. Add a custom matcher that checks the same params are in the body (ignoring ordering). vcr's in-built body matcher can't be used since it fails under py3 with a "UnicodeEncodeError" on the "€ in commit message" tests. The DREV ids have decreased since the recordings were generated against a different phabricator instance to avoid spamming mercurial-devel. Differential Revision: https://phab.mercurial-scm.org/D6347

merge with stable

absorb: be more specific when erroring out on merge commit When you have a merge commit checked out and run `hg absorb`, it would tell you abort: no mutable changeset to change That makes it sound like the problem is public commits when isn't really. Let's be more specific in this case. There was already a test case that test this, so that now prints the new message. I added a new test case that shows the old message (when a public commit is checked out). Differential Revision: https://phab.mercurial-scm.org/D6354

remotefilelog: log when we're about to fetch files I'm debugging a slow client situation and knowing how many files are in the batch request would be a nice thing. Differential Revision: https://phab.mercurial-scm.org/D6353

revset: populate wdir() by its hash or revision number It belongs to the same category as the null hash/revision, and we do handle these virtual identifiers in id()/rev() predicates. Let's do that more consistently.

sslutil: fsencode path returned by certifi (issue6132) By inspection, this is the only codepath that could be returning a string instead of a bytes on Python 3.

revset: extract private constant of {nullrev, wdirrev} set I'll add a few more users of this constant to get around wdir identifiers.

help: suggest merge() revset instead of -m/--only-merges Suggested by Dr Rainer Woitok.

tests: update annotate tests to work around simplemerge bug test-annotate.t and test-fastannotate.hg were failing with --pure since 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). It turned out to be because the pure file merge code behaved differently. I'm guessing it's the mdiff.get_matching_blocks() that behaves differently, but I haven't confirmed that. With this content in the base: a a a And this on the local side: a z a And this on the other side: a a a b4 c b6 It produced this conflict: a z a <<<<<<< working copy: b80e3e32f75a - test: c ||||||| base a ======= a b4 c b5 >>>>>>> merge rev: 64afcdf8e29e - test: mergeb I don't care enough about the pure Python code to fix it, so this patch just updates the tests to manually resolve the conflict. Differential Revision: https://phab.mercurial-scm.org/D6351

copies: delete misplaced comment The comment was added in 78d760aa3607 (duplicatecopies: do not mark items not in the dirstate as copies, 2013-03-28). It became misplaced in 3666331164bb (cmdutil: add copy-filtering support to duplicatecopies, 2014-06-07). Then the relevant code was moved far away in 754b5117622f (context: add workingfilectx.markcopied, 2017-10-15). Differential Revision: https://phab.mercurial-scm.org/D6352

phabricator: include branch in the phabread output Depends on D6301 Differential Revision: https://phab.mercurial-scm.org/D6302

phabricator: fallback to reading metadata from diff for phabread Differential Revision: https://phab.mercurial-scm.org/D6301

phabricator: include commit (node) and parent in the local:commits metadata Differential Revision: https://phab.mercurial-scm.org/D6298

copies: remove redundant filtering of ping-pong renames in _chain() We already handle the ping-pong rename case in the filtering step, so there's very little point in doing it in the chaining loop (ping-pong renames are very rare, so I'm not worried about the cost of adding it and then removing it again). Differential Revision: https://phab.mercurial-scm.org/D6344

repair: reword comments that I noticed while working on source formatting I think this is clearer, and one will also keep us from upsetting check-code when other formatting cleanups happen. Differential Revision: https://phab.mercurial-scm.org/D6339

commit: allow --interactive to work again when naming a directory (issue6131)

gendoc: nest command headers under category headers Differential Revision: https://phab.mercurial-scm.org/D6329

minirst: support subsubsubsubsections (header level 5) with marker '''' Differential Revision: https://phab.mercurial-scm.org/D6328

gendoc: guarantee that all commands were processed The new logic renders the commands belonging to each category in turn. Commands with an unregistered category are at risk of getting skipped because their category is not in the list. By comparing the list of all commands to a log of processed commands, we can detect commands with unregistered categories and fail with an error message. Differential Revision: https://phab.mercurial-scm.org/D6327

gendoc: group commands by category in man page and HTML help Make Mercurial's man page and HTML help group commands by category, and present the categories in a helpful order. `hg help` already does this; this patch uses the same metadata. This patch uses the same header level for command categories and for commands. A subsequent patch will push the command headers down one level. Differential Revision: https://phab.mercurial-scm.org/D6326

gendoc: indent loop to make next patch more legible Differential Revision: https://phab.mercurial-scm.org/D6325

contrib: have byteify-strings explode if run in Python 2 Differential Revision: https://phab.mercurial-scm.org/D6341

repair: reword comment about bookmarks logic Again, this will help auto-formatting shortly. Differential Revision: https://phab.mercurial-scm.org/D6340

monotone: fix a bogus _() wrapper that was caught when formatting code There was a spurious space after `debug`, which hid the _() inside ui.debug() from check-code. Sigh. While here, wrap things more concisely. Differential Revision: https://phab.mercurial-scm.org/D6338

commit: add ability to print file status after each successful invocation When commands.commit.post-status is enabled, `hg commit` will effectively run `hg status -mardu` after committing. It can help catch mistakes like not committing all needed files or not adding unknown files that should've been part of the just created commit.

tests: flatten repo structure in test-commit.t Let's move to parent directory before `hg init` repos, since they don't need to be nested. It makes amend/strip messages that include full path to the backup bundle shorter, for instance.

lfs: add a TODO file This is a cleaned up and reorganized list of items I sent out about a year ago. But tracking this in the repo (like the narrow extension) gives more visibility in case anyone wants to help out.

copies: make "limit" argument to _tracefile() mandatory We always pass a limit. I think the fact that it was optional was also the reason we checked ">=limit" before we used it. So now we can remove that condition too. Differential Revision: https://phab.mercurial-scm.org/D6335

localrepo: don't use defaults arguments that will never be overridden The commithook() callback will be called when the lock is released. lock.release() calls the callback without arguments, so it was quite confusing to me that this function declared extra arguments. We can just close on the variables in the outer scope instead. Differential Revision: https://phab.mercurial-scm.org/D6336

tags: avoid double-reversing a list Differential Revision: https://phab.mercurial-scm.org/D6337

updatecaches: also warm hgtagsfnodescache Now that a full update of this cache run in a reasonable amount of time, we can warm everything when during a full update.

hgtagsfnodescache: inherit fnode from parent when possible If a changeset does not update the content of `.hgtags`, it means it will use the same file-node (for `.hgtags`) as its parents. In this case we can directly reuse the parent's file-node. We use this property when updating the `hgtagsfnodescache` taking a faster path if we already have a cached value for the parents of the node we are looking at. Doing so provides a large performance boost when looking at a lot of fnodes, especially on repository with very large manifest: timing for `tagsmod.fnoderevs(ui, repo, repo.changelog.revs())` mercurial: (41907 revisions, 1923 files) before: 6.9 seconds after: 2.7 seconds (-54%) pypy: (96266 revisions, 5198 files) before: 80 seconds after: 20 seconds (-75%) mozilla-central: (463411 revisions, 272080 files) before: 7166.4 seconds after: 47.8 seconds (-99%, x150 speedup) On a copy of mozilla-try with about 35K heads ans 1.7M changesets, this moves the computation from many hours to a couple of minutes, making it more interesting to do a full warm up of this cache before computing tags (from a cold cache). There seems to be other performance low hanging fruits, like avoiding the use of changectx or a more revision centric logic. However, the new code is fast enough for my needs right now.

hgtagsfnodescache: handle nullid lookup The null revision is empty, so it `.hgtags` content is `nullid` in regards with the `hgtagsfnodescache`. Dealing with `nullid` will help with the next changeset. Before this change, feeding `nullid` to `hgtagsfnodescache.getfnode` would return a wrong result (fnode for tip).

help: register the 'gpg' command category and give it a description help.py expects extensions to register their command category in the CATEGORY_ORDER and CATEGORY_NAMES variables. Once gendoc.py orders commands by category, in the next patch, it'll assume this registration (and raise an exception on encountering any unregistered categories). Luckily, gpg is the only bundled extension with an unregistered custom category, so let's fix it. Differential Revision: https://phab.mercurial-scm.org/D6324

histedit: Speed up scrolling in patch view mode Store patchcontents into the mode state, avoiding the expensive call to ui for computing the patchcontents. Before this change in large repos histedit patch view mode can be very irresponsive.

histedit: Show file names in multiple line format

parser: fix crash by parsing "()" in keyword argument position A tree node can be either None or a tuple because x=("group", None) is reduced to x[1].

repoview: introduce a `experimental.extra-filter-revs` config The option define revisions to additionally filter out of all repository "view". The end goal is to provide and easy to way to serve multiple subset of the same repository using multiple "shares". The simplest use case of this feature is to have one view serving the public changesets and one view also serving the draft. This is currently achievable using the new `server.view` option introduced recently by Joerg Sonnenberger. However, more advanced use cases need more advanced definitions. For example some needs a view dedicated to some release branches, or view that hides security fixes to be released. Joerg Sonnenberger and I discussed this topic at the recent mini-sprint and the both of us have seen real life use cases for this. (This series got written during the same mini-sprint). The feature is fully functional, and use similar cache-fallback mechanism to ensure decent performance. However,there remaining room to ensure each share caches and hooks collaborate with each others. This will come at a later time once users start to actually test this feature on real usecase.

copies: filter out copies from non-existent source later in _chain() _changesetforwardcopies() repeatedly calls _chain(). That is very expensive because _chain() does lookups in the manifest. I hope to split up the function in two parts: 1) simple chaining, not considering end points, and 2) filter out files that don't exist in the end points (and ping-pong copies/renames). This patches gets us closer to that by moving the check for non-existent source later in the function. Now there are no more checks for "src" and "dst" in the first loop; all the filtering of invalid copies is done in the second loop. The code also looks much more consistent now. No measureable impact on `hg debugpathcopies 4.0 4.8`. That shouldn't be surprising since the only case we're doing more checks now is in case of chained copies/renames, which are quire rare in practice. Differential Revision: https://phab.mercurial-scm.org/D6277

copies: clarify mutually exclusive cases in _chain() with a s/if/elif/ If the 'b' dict has a rename from 'x' to 'y', it shouldn't be possible for 'x' to be both (a key) in 'a' and in 'src'. That would mean that 'x' is a file in the source commit and also a rename destination in the intermediate commit. But we currently don't allow renaming files onto existing files, so that shouldn't happen. So let's clarify that by using an "elif" instead of an "if". And if we did allow renaming files onto existing files, we should prefer to use the rename destination in the intermediate commit as source anyway. Differential Revision: https://phab.mercurial-scm.org/D6276

copies: delete a redundant cleanup step in _chain() The check is redundant since d5edb5d3a337 (copies: filter out copies when target is not in destination manifest, 2019-02-14). To test that hypothesis, I made this change in the commit that commit, but all tests still passed. I think the case was necessary before then, we just didn't have tests for it. Differential Revision: https://phab.mercurial-scm.org/D6275

copies: document cases in _chain() Differential Revision: https://phab.mercurial-scm.org/D6274

copies: ignore heuristics copytracing when using changeset-centric algos Differential Revision: https://phab.mercurial-scm.org/D6269

copies: move check for experimental.copytrace==<falsy> earlier I'm going to ignore experimental.copytrace when changeset-centric algorithms are required. This little refactoring makes that easier to add. Differential Revision: https://phab.mercurial-scm.org/D6268

copies: replace .items() by .values() where appropriate As pointed out by Pierre-Yves. Differential Revision: https://phab.mercurial-scm.org/D6266

copies: inline _computenonoverlap() in mergecopies() We now call pathcopies() from the base to each of the commits, and that calls _computeforwardmissing(), which does file prefetching (in the remotefilelog override). So the call to _computenonoverlap() is now pointless (the sets of files from _computenonoverlap() are subsets of the sets of files from _computeforwardmissing()). This somehow also fixes a broken remotefilelog test. Differential Revision: https://phab.mercurial-scm.org/D6256

copies: calculate mergecopies() based on pathcopies() When copies are stored in changesets, we need a changeset-centric version of mergecopies() just like we have a changeset-centric version of pathcopies(). I think the natural way of thinking about mergecopies() is in terms of pathcopies() from the base to each of the commits. So if we can rewrite mergecopies() based on two such pathcopies() calls, we'll get the changeset-centric version for free. That's what this patch does. A nice bonus is that it ends up being a lot simpler. mergecopies() has accumulated a lot of technical debt over time. One good example is the code for dealing with grafts (the "partial/incomplete/dirty" stuff). Since pathcopies() already deals with backwards renames and ping-pong renames, we get that for free. I've run tests with hard-coded debug logging for "fullcopy" and while I haven't looked at every difference it produces, all the ones I have looked at seemed reasonable to me. I'm a little surprised that no more tests fail when run with '--extra-config-opt experimental.copies.read-from=compatibility' compared to before this patch. This patch also fixes the broken cases in test-annotate.t and test-fastannotate.t. It also enables the part of test-copies.t that was previously disabled exactly because mergecopies() needed to get a changeset-centric version. One drawback of the rewritten code is that we may now make remotefilelog prefetch more files. We used to prefetch files that were unique to either side of the merge compared to the other. We now prefetch files that are unique to either side of the merge compared to the base. This means that if you added the same file to each side, we would not prefetch it before, but we would now. Such cases are probably quite rare, but one likely scenario where they happen is when moving from a commit to its successor (or the other way around). The user will probably already have the files in the cache in such cases, so it's probably not a big deal. Some timings for calculating mergecopies between two revisions (revisions shown on each line, all using the common ancestor as base): In the hg repo: 4.8 4.9: 0.21s -> 0.21s 4.0 4.8: 0.35s -> 0.63s In and old copy of the mozilla-unified repo: FIREFOX_BETA_60_BASE^ FIREFOX_BETA_60_BASE: 0.82s -> 0.82s FIREFOX_NIGHTLY_59_END FIREFOX_BETA_60_BASE: 2.5s -> 2.6s FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE: 3.9s -> 4.1s FIREFOX_AURORA_50_BASE FIREFOX_BETA_60_BASE: 31s -> 33s So it's measurably slower in most cases. The most significant difference is in the hg repo between revisions 4.0 and 4.8. In that case it seems to come from the fact that pathcopies() uses fctx.isintroducedafter() (in _tracefile), while the old mergecopies() used fctx.linkrev() (in _checkcopies()). That results in a single call to filectx._adjustlinkrev(), which is responsible for the entire difference in time (in my repo). So we pay a performance penalty but we get more correct code (see change in test-mv-cp-st-diff.t). Deleting the "== f.filenode()" in _tracefile() recovers the lost performance in the hg repo. There were are few other optimizations in _checkcopies() that I could not measure any impact from. One was from the "seen" set. Another was from a "continue" when the file was not in the destination manifest (corresponding to "am" in _tracefile). Also note that merge copies are not calculated when updating with a clean working copy, which is probably the most common case. I therefore think the much simpler code is worth the slowdown. Differential Revision: https://phab.mercurial-scm.org/D6255

tests: add test where copy source is deleted and added back This shows another difference between pathcopies() and mergecopies(): mergecopies() considers files that have been deleted and then added back as different files, but pathcopies() does not. Differential Revision: https://phab.mercurial-scm.org/D6330

merge with stable

Added signature for changeset 07e479ef7c96

Added tag 5.0 for changeset 07e479ef7c96

obsolete: drop the legacy `_enabled` variable Evolve 8.5.0 stopped setting this, and it would have been easier to figure out why TortoiseHg stopped allowing amends if it would have crashed on the missing variable.

discovery: only calculate closed branches if required The number of new closed branches is required for printing in error message. So let's only calculate them if we need to print error about new branches. Differential Revision: https://phab.mercurial-scm.org/D6314

hghave: deal with "rc" release Without this change, 5.0rc0 is not recognised as 5.0

branchcache: store the maximum tip in a variable inside for loop Instead of assigning self.tiprev multiple times in the for loop, and calling cl.node() on it, let's store that in a temporary variable and assign it in the end of loop. Differential Revision: https://phab.mercurial-scm.org/D6311

tests: demonstrate that rename is followed to wrong parent from merge This test case shows another way that copies are handled differently between `hg st` (pathcopies()) and `hg co -m` (mergecopies). The reason is that pathcopies() calls _tracefiles(), which checks that the file nodeid of an ancestor matches the file nodeid in the base commit. mergecopies() should probably be doing the same. Differential Revision: https://phab.mercurial-scm.org/D6323

test: demonstrate failure to follow rename with shadowed linkrev This shows a difference in handling of copies between `hg st` (pathcopies()) and `hg co -m`. The issue here is that mergecopies() uses the unadjusted linkrev() for determining when to stop walking ancestors. Differential Revision: https://phab.mercurial-scm.org/D6322

tests: slightly modify a linkrev test to prepare for expanding it The test case checks that the copy tracing code doesn't get confused by linkrevs when walking a file's ancestors. This patch chnages the test slightly so a second commit is grafted, thus producing a second "bad" linkrev. I'll use this in the next patch to demonstrate a bug. Differential Revision: https://phab.mercurial-scm.org/D6321

copies: process files in deterministic order for stable tests I also fixed a typo while at it. Differential Revision: https://phab.mercurial-scm.org/D6320

narrow: send specs as bundle2 data instead of param (issue5952) (issue6019) Before this patch, when ACL is involved, narrowspecs are send as bundle2 parameter for narrow:spec bundle2 part. The limitation of bundle2 parts are they cannot send data larger than 255 bytes. Includes and excludes in narrow are not limited by size and they can grow over 255 bytes. This patch introduces a new mandatory bundle2 part and send narrowspecs as data of that. The new bundle2 part is introduced to keep things cleaner and easy to distinguish related to backward compatibility. The part is mandatory because without server's narrowspec, the local ACL narrow repo won't work. This patch makes clients compatible with servers which have older versions. However I left a comment that we should drop the other bundle2 part soon as that's broken and people should not rely on that. I named the new bundle2 part 'Narrow:responsespec' because: 1) Capital 'N' to make it mandatory 2) 'Narrow:spec' cannot be used because bundle2 enforces that there should not be two different parts which resolve to same name when lowercased. 3) reponsespec clears that they are specs which are send as reponse by the server While I was here, I renamed `narrowhgacl` section to `narrowacl` as suggested by idlsoft@ and martinvonz@. Differential Revision: https://phab.mercurial-scm.org/D6310

py3: properly reject non-encoded strings given to hgweb

py3: handle meta-path finders that only use pre-python3.4 API

remotefilelog: add missing argument to hg.verify wrapper Differential Revision: https://phab.mercurial-scm.org/D6313

revsetbenchmark: track some simple use of "only" The only revset is quite useful and has various possible optimisation. tracking its timing seems useful.

inno: bump keyring to 18.0.1 to avoid AttributeError (issue6043) The error seems to be harmless, because it happens after closing the connection. For whatever reason, this isn't bundled with the Wix installer. https://github.com/jaraco/keyring/issues/386 https://bitbucket.org/Mekk/mercurial_keyring/issues/63/attributeerror-during-process-finish-with

push: added clear warning message when pushing closed branches(issue6080) Differential Revision: https://phab.mercurial-scm.org/D6038

branch: abort if closing branch from a non-branchhead cset This patch make sure that we abort if the user is trying to close a branch from a cset which is not a branch head. Changes in test file reflect the fixed behaviour. Differential Revision: https://phab.mercurial-scm.org/D6282

branch: add tests which shows branch can be closed from a non-branchhead cset This patch shows that we can close a branch even from a cset which is not a branch head. It was supposed to abort this operation. Next patch will be fixing the issue. Differential Revision: https://phab.mercurial-scm.org/D6281

phabricator: read more metadata from local:commits local:commits metadata can contain branch info, and 'rev' has been superseded by 'commit', see: https://github.com/phacility/arcanist/blob/83661809e532c3fe444a8bf7c7d6936e6377691b/src/repository/api/ArcanistMercurialAPI.php#L281 Differential Revision: https://phab.mercurial-scm.org/D6300

phabricator: don't assume the existence of properties of local:commits Not all the properties are guaranteed to be there, so if we don't check first we could die with a KeyError. Differential Revision: https://phab.mercurial-scm.org/D6299

phabricator: include branch in the diffproperty metadata This does not make Phabricator display the branch in web UI anywhere as that still need us to use creatediff API for that. However a future patch will make phabread use this to include the branch in its `hg import`-able output. Differential Revision: https://phab.mercurial-scm.org/D6297

tests: demonstrate `hg log -r . <file>` linkrev bug Differential Revision: https://phab.mercurial-scm.org/D6309

unionrepo: sync with repository API Differential Revision: https://phab.mercurial-scm.org/D6289

match: remove unused match.__iter__ implementation (API) Differential Revision: https://phab.mercurial-scm.org/D6305

fix: allow fixer tools to return metadata in addition to the file content With this change, fixer tools can be configured to output a JSON object that will be parsed and passed to hooks that can be used to print summaries of what code was formatted or perform other post-fixing work. The motivation for this change is to allow parallel executions of a "meta-formatter" tool to report back statistics, which are then aggregated and processed after all formatting has completed. Providing an extensible mechanism inside fix.py is far simpler, and more portable, than trying to make a tool like this communicate through some other channel. Differential Revision: https://phab.mercurial-scm.org/D6167

context: check file exists before getting data from _wrappedctx overlayworkingctx class is used to do in-memory merging. The data() function of that class has logic to look for data() in the wrappedctx if the file data in cache is empty and if the file is dirty. This assumes that if a file is dirty and cache has empty data for it, it will exists in the _wrappedctx. However this assumption can be False in case when we are merging a file which is empty in destination. In these cases, the backup file 'foo.orig' created by our internal merge algorithms will be empty, however it won't be present in _wrappedctx. This case will lead us to error like the one this patch is fixing. Let's only fallback to getting data from wrappedctx if cache has 'None' as data. Differential Revision: https://phab.mercurial-scm.org/D6308

tests: show IMM is broken when merging file empty in destination When we are doing in-memory merging, and we are merging a file which is empty in merge destination, it leads to error 'abort: xxx not found in manifest'. Next patch will fix this error. Differential Revision: https://phab.mercurial-scm.org/D6307

buildrpm: bump bundled Python version to 2.7.16 when building for centos{5,6} When building rpm packages for centos 5 and 6, we bundle a mercurial-specific version of Python 2.7 in /opt/python-hg. This change is analogous to 5e947367606c, and bumps the embedded Python version from 2.7.14 (released in 2017) to 2.7.16 (latest as of today).

merge with stable

phabricator: set local:commits time metadata as an int, not a string Same as arcanist does Differential Revision: https://phab.mercurial-scm.org/D6296

phabricator: use templatefilters.json in writediffproperties Instead of json.dumps, since it makes the code simpler and more readable. This would have been the better option for 8fd19a7b4ed6 but I wasn't aware of it at the time. Differential Revision: https://phab.mercurial-scm.org/D6295

commands: use byteskwargs() in verify() Otherwise Python 3 complains about the missing key. Differential Revision: https://phab.mercurial-scm.org/D6294

match: use raw strings to avoid illegal baskslash escape Python 3.8 was complaining about the invalid escape sequences. Let's use raw strings to avoid the warning and double baskslashes. Differential Revision: https://phab.mercurial-scm.org/D6293

revbranchcache: use context manager in _writerevs() to write to file The other _writenames() is a bit complicated to use context manager. Differential Revision: https://phab.mercurial-scm.org/D6292

revbranchcache: factor logic to write names and revs in separate functions Before this patch, the write function was so populated with upto 4 level of indentation, it was hard to understand what's going on. Differential Revision: https://phab.mercurial-scm.org/D6291

tests: make log style a little easier to read in test-copytrace-heuristics.t Revision numbers are much shorter and easier to read (especially compared to the full nodeids that were used here), so I switched to that. That's also what almost all the commands used (e.g. `hg rebase -s . -d 1`). I updated the two instances that used nodeids. I also made some other little cleanups to the log templates. Differential Revision: https://phab.mercurial-scm.org/D6279

tests: avoid cryptic nodeids in tests/test-rename-merge1.t These two nodeids had not been part of any output before, so one can't know which revision they refer to without adding something like `hg log` before them. It turned out that '.^' was equivalent for both of them, so that's what I replaced them with. Differential Revision: https://phab.mercurial-scm.org/D6280

tests: defines aliases for `hg log` calls in test-copytrace-heuristics.t This also makes the test cases more consistent since a few had missed the ":" in "changeset:" that the others used. Differential Revision: https://phab.mercurial-scm.org/D6278

rust-discovery: implementing and exposing stats() This time, it's simple enough that we can do it in all layers in one shot. Differential Revision: https://phab.mercurial-scm.org/D6233

rust-discovery: cpython bindings for the core logic As previously done with the ancestors submodule, testing for the bindings is provided from Python on a trivial case. Differential Revision: https://phab.mercurial-scm.org/D6232

rust-discovery: starting core implementation Once exposed to the Python side, this core object will avoid costly roundtrips with potentially big sets of revisions. This changeset implements the core logic of the object only, i.e., manipulation of the missing, common and undefined set-like revision attributes. Differential Revision: https://phab.mercurial-scm.org/D6231

rust-dagops: roots Unsuprisingly, the algorithm is much easier than for heads, provided we work on a set in the first place. To improve the signature, a trait for set-likes object would be useful, but that's not an immediate concern. Differential Revision: https://phab.mercurial-scm.org/D6230

rust-dagops: range of revisions This is a Rust implementation for what reachableroots2() does if includepath is True. The algorithmic details and performance notes are included in the documentation comment. Our main use case for now is a Rust counterpart of the partialdiscovery object, so we don't really need bindings yet. Differential Revision: https://phab.mercurial-scm.org/D6229

setup: tweak error message for Python 3 We now have beta support for Python 3. In my opinion, it isn't yet stable enough to allow `pip install Mercurial` to work with Python 3 out of the box: we don't want people accidentally using Mercurial with Python 3 just yet. But I do think we should be more friendly about informing people of their options. This commit tweaks the error message that users see when running setup.py with Python 3. We instruct them about the current level of Python 3 support, point them at the wiki for more info, and give them instructions on how to bypass the check. As part of this, I also changed which version value is printed, as we were printing a named tuple before.

setup: remove set and dict comprehensions Yuya observed in a recent review that it is worthwhile to keep setup.py parseable with Python 2.6 so a useful error message is seen when attempting to run with Python 2.6. This commit removes a set and dict comprehension so setup.py is parseable with Python 2.6.

branchcache: don't verify all nodes while writing nodes are verified either when they are added or used. In case of commits. we will load the whole branchmap, only verify nodes for the branch on which we are committing and then we write. However before this patch, writing the branchmap was validating all the nodes whereas it should not. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D6290

setup: properly package distutils in py2exe virtualenv builds Our in-repo py2exe packaging code uses virtualenvs for managing dependencies. An advantage of this is that packaging is more deterministic and reproducible. Without virtualenvs, we need to install packages in the system Python install. Packages installed by other consumers of the system Python could leak into the Mercurial package. A regression from this change was that py2exe packages contained the virtualenv's hacked distutils modules instead of the original distutils modules. (virtualenv installs a hacked distutils module because distutils uses relative path lookups that fail when running from a virtualenv.) This commit introduces a workaround so py2exe packaging uses the original distutils modules when running from a virtualenv. With this change, `import distutils` no longer fails from py2exe builds produced from a virtualenv. This fixes the regression. Furthermore, we now include all distutils modules. Before, py2exe's module finding would only find modules there were explicitly referenced in code. So, we now package a complete copy of distutils instead of a partial one. This is even better than before. # no-check-commit foo_bar function name

narrow: also warn when not deleting untracked or ignored files Differential Revision: https://phab.mercurial-scm.org/D6265

setdiscovery: fix a few typos Differential Revision: https://phab.mercurial-scm.org/D6263

copies: delete debug message about "unmatched files new in both" Same reasoning as previous patch. Differential Revision: https://phab.mercurial-scm.org/D6251

copies: delete debug message about changes since common ancestor Same reasoning as previous patch. Differential Revision: https://phab.mercurial-scm.org/D6250

copies: delete debug message about search limit I'm about to rewrite mergecopies() and this message will no longer be emitted then. Let's remove the message now to remove a distraction from that patch. Differential Revision: https://phab.mercurial-scm.org/D6249

copies: move early return for "no copies" case a little earlier We can return before the block that prints debug messages. That block will not be run anyway when there are no copies. Differential Revision: https://phab.mercurial-scm.org/D6248

copies: fix up "fullcopy" with missing entries from "diverge" Similar to the previous patch, but this doesn't even affect tests. It does affect tests if you change them to turn on debug logging. I'm fixing it here so reviewers of the later rewrite patch can hard-code debug logging to be on and more easily compare test results. Differential Revision: https://phab.mercurial-scm.org/D6247

copies: fix up "fullcopy" with missing entries from "copy" This is just a workaround similar to the previous one. It will make it easier to follow later patches. Differential Revision: https://phab.mercurial-scm.org/D6246

merge: remove workaround for issue5020 As I explained in the previous commit, I think the filtering added there is a better fix for the issue, so the workaround from 41f6af50c0d8 (merge: fix crash on criss cross merge with dir move and delete (issue5020), 2017-01-31) should no longer be needed. Differential Revision: https://phab.mercurial-scm.org/D6245

copies: don't include copies that are not in source in directory move I've been working on a rewrite of mergecopies(). I compared the output of the rewritten version with the current version. I noticed that between FIREFOX_NIGHTLY_59_END and FIREFOX_BETA_60_BASE in the mozilla-unified repo, there were many copies that the current version detected that the rewritten version did not. One example was js/src/gc/Iteration.h -> js/src/gc/PublicIterators.h. Then I realized that js/src/gc/Iteration.h doesn't even exist in FIREFOX_NIGHTLY_59_END. This patch adds a filtering step for the "fullcopy" dict. It turns out that that change also affects the test for issue5020 in test-merge-criss-cross.t. The 'dm' action no longer happens there. At first I thought that the test case change meant that this patch was broken, but I think it's actually correct tha the 'dm' action should not happen there. The result of the bid merge is still the same. I suspect this filtering is a better solution for the issue than 41f6af50c0d8 (merge: fix crash on criss cross merge with dir move and delete (issue5020), 2017-01-31). I also suspect that it was broken just a few months earlier by a005c33d0bd7 (mergecopies: add logic to process incomplete data, 2016-10-04). Note that bid merge had been enabled for a few years at that point, since 19903277f035 (merge: use bid merge by default (BC), 2014-10-01). This patch is still just a workaround. It will be cleaned up soon (with the rewrite of mergecopies()). But doing this in a separate patch makes later patches easier to understand and gives a place to explain why this is changing. Differential Revision: https://phab.mercurial-scm.org/D6244

tests: add test for issue5343 (grafting with copies) It seems that issue5353 resulted in a lot of tests in test-graft.t, but the bug actually reported in that issue didn't get a test case. This patch adds one for the "move" and one for the "copy" version of it. I also added a "copy+modify" case, to show what should be a merge conflict. I didn't add one for the "backwards" version of it since the comment says that that was already covered by previous work. The tests added by this patch show the broken behavior (the bug is still open). I suspect the results returned from mergecopies() are not expressive enough to fix this issue: it has a dict for copies to merge with, but that can only give one more filename, but here we need two (one for the path on the remote side and one for the path in the merge base). I want to have it tested anyway since I'm about to refactor mergecopies(). Differential Revision: https://phab.mercurial-scm.org/D6242

chistedit: use context manager to set verbose ui I'm still not exactly sure why this is necessary -- perhaps setting it unconditionally would leak this setting in chg invocations. Regardless, this would have looked very out of place as compared to how this setting is done everywhere else, so at least for the sake of style, let's be consistent with the rest of the codebase.

setdiscovery: stop limiting the number of local head we initially send In our testing this limitation provides now real gain and instead triggers pathological discovery timing for some repository with many heads. See inline documentation for details. Some timing below: Mozilla try repository, (~1M revs, ~35K heads), discovery between 2 clones with 100 head missing on each side before: ! wall 1.492111 comb 1.490000 user 1.450000 sys 0.040000 (best of 20) ! wall 1.813992 comb 1.820000 user 1.700000 sys 0.120000 (max of 20) ! wall 1.574326 comb 1.573500 user 1.522000 sys 0.051500 (avg of 20) ! wall 1.572583 comb 1.570000 user 1.520000 sys 0.050000 (median of 20) after: ! wall 1.147834 comb 1.150000 user 1.090000 sys 0.060000 (best of 20) ! wall 1.449144 comb 1.450000 user 1.330000 sys 0.120000 (max of 20) ! wall 1.204618 comb 1.202500 user 1.146500 sys 0.056000 (avg of 20) ! wall 1.194407 comb 1.190000 user 1.140000 sys 0.050000 (median of 20) pypy (~100 heads, 317 heads) discovery between clones with only 42 common heads before: ! wall 0.031653 comb 0.030000 user 0.030000 sys 0.000000 (best of 25) ! wall 0.055719 comb 0.050000 user 0.040000 sys 0.010000 (max of 25) ! wall 0.038939 comb 0.039600 user 0.038400 sys 0.001200 (avg of 25) ! wall 0.038660 comb 0.050000 user 0.040000 sys 0.010000 (median of 25) after: ! wall 0.018754 comb 0.020000 user 0.020000 sys 0.000000 (best of 49) ! wall 0.034505 comb 0.040000 user 0.030000 sys 0.010000 (max of 49) ! wall 0.019631 comb 0.019796 user 0.018367 sys 0.001429 (avg of 49) ! wall 0.019132 comb 0.020000 user 0.020000 sys 0.000000 (median of 49) Private repository (~1M revs, ~3K heads), discovery from a strip subset, about 100 changesets to be pulled. before: ! wall 1.837729 comb 1.840000 user 1.790000 sys 0.050000 (best of 20) ! wall 2.203468 comb 2.200000 user 2.100000 sys 0.100000 (max of 20) ! wall 2.049355 comb 2.048500 user 2.002500 sys 0.046000 (avg of 20) ! wall 2.035315 comb 2.040000 user 2.000000 sys 0.040000 (median of 20) after: ! wall 0.136598 comb 0.130000 user 0.110000 sys 0.020000 (best of 20) ! wall 0.330519 comb 0.330000 user 0.260000 sys 0.070000 (max of 20) ! wall 0.157254 comb 0.155500 user 0.123000 sys 0.032500 (avg of 20) ! wall 0.149870 comb 0.140000 user 0.110000 sys 0.030000 (median of 20) Same private repo, discovery between two clone with 500 different heads on each side: before: ! wall 2.372919 comb 2.370000 user 2.320000 sys 0.050000 (best of 20) ! wall 2.622422 comb 2.610000 user 2.510000 sys 0.100000 (max of 20) ! wall 2.450135 comb 2.450000 user 2.402000 sys 0.048000 (avg of 20) ! wall 2.443896 comb 2.450000 user 2.410000 sys 0.040000 (median of 20) after: ! wall 0.625497 comb 0.620000 user 0.570000 sys 0.050000 (best of 20) ! wall 0.834723 comb 0.820000 user 0.730000 sys 0.090000 (max of 20) ! wall 0.675725 comb 0.675500 user 0.628000 sys 0.047500 (avg of 20) ! wall 0.671614 comb 0.680000 user 0.640000 sys 0.040000 (median of 20)

peer: introduce a limitedarguments attributes When set to True, it signal that the peer cannot receive too larges arguments and that algorithm must adapt. This should only be True for http peer that does not support argument passed as "post". This will be useful to unlock better discovery performance in the next changesets. I am using a dedicated argument because this is not really a usual "capabilities" things. An alternative approach would be to adds a "large-arguments" to all peer, but the http peers. That seemed a bit too hacky to me.

verify: also check full manifest validity during verify runs Before this changes, `hg verify` only checked if a manifest revision existed and referenced the proper files. However it never checked the manifest revision content itself. Mercurial is expecting manifest entries to be sorted and will crash otherwise. Since `hg verify` did not attempted a full restoration of manifest entry, it could ignore this kind of corruption. This new check significantly increases the cost of a `hg verify` run. This especially affects large repository not using `sparse-revlog`. For now, this is hidden behind the `--full` experimental flag.

verify: introduce an experimental --full flag The flag currently has no effect, see next changeset for details. We introduce the flag as experimental to keep the freedom of changing our mind on the final UI. Note: this patch highlight a small but in `hg help`. An option section is generated even if no option are visible.

verify: introduce a notion of "level" Some checks are slower than others, to help the user to run the checks he needs, we are about to introduce new flag to select faster vs deeper runs. This put the scaffolding in place to do this.

tests: split out separate test for issue5020 The test was added to the existing setup in 41f6af50c0d8 (merge: fix crash on criss cross merge with dir move and delete (issue5020), 2017-01-31). I'm about to make a change that affects that test and it's much easier to follow if the test case for issue5020 is a separate test case. The separate test case is based on what mpm provided in comment 12 on the issue. `hg diff -r 41f6af50c0d8^ tests/test-merge-criss-cross.t` after this patch is pretty small (besides the added test). It's probably easier for reviewers to look at that than to try to understand the diff itself (I don't understand it). Differential Revision: https://phab.mercurial-scm.org/D6243

tests: avoid a rename/delete conflict when updating in test-narrow-update.t After the upcoming rewrite of mergecopies(), this test would otherwise (accurately) start warning about "inside/f1 was deleted and renamed". Differential Revision: https://phab.mercurial-scm.org/D6254

tests: delete unused function in test-rename-merge2.t Differential Revision: https://phab.mercurial-scm.org/D6253

tests: make merge conflicts explicit in `hg annotate` tests We were using `true` as merge tool. I think it makes the test easier to understand if we make the conflicts explcit. It also papered over a conflict that shouldn't have been a conflict (just a bug in copy tracing). I've marked that "BROKEN". Differential Revision: https://phab.mercurial-scm.org/D6252

narrow: make warning about possibly dirty files respect ui.relative-paths Differential Revision: https://phab.mercurial-scm.org/D6264

merge: forgot to pull before release These two changes should be part of 5.0, but we are fine leaving them out of the RC.

Added signature for changeset 4a8d9ed86475

Added tag 5.0rc0 for changeset 4a8d9ed86475

merge: default into stable for release candidate

debugdiscovery: include the number of heads in all sets We already displayed information about heads of the common set that are either local or remote heads. We now also do so for heads of the common set that are both local and remote heads too. This is useful because various step in the set discovery algorithm have head specific optimizations.

recover: add a --[no-]verify flag For trivial cases, the cost of the verify run after `hg recover` is getting in the way. In addition for very large repositories, the cost is simply too high to be paid, making `hg recover` an unusable commands. We introduce a --verify flag, set by default. If is automatically associated with a --no-verify flag that one can use to skip the verify step. We might consider changing the default behavior in the future. However this is out of scope for this series.

bundle2: handle compression in _forwardchunks _forwardchunks is used to compensate for getbundle protocol deficits. Since it transparently decodes the payload, it also needs to remove the corresponding compression parameter in case the server decides to send one. This the wire protocol part of issue 5990. Differential Revision: https://phab.mercurial-scm.org/D6182

changelog: parse copy metadata if available in extras This lets read back the copy metadata we just started writing. There are still many places left to teach about getting the copy information from the changeset, but we have enough ({file_copies}, specifically) that we can add it now and have some test coverage of it. Differential Revision: https://phab.mercurial-scm.org/D6186

copies: add config option for writing copy metadata to file and/or changset This introduces a config option that lets you choose to write copy metadata to the changeset extras instead of to filelog. There's also an option to write it to both places. I imagine that may possibly be useful when transitioning an existing repo. The copy metadata is stored as two fields in extras: one for copies since p1 and one for copies since p2. I may need to add more information later in order to make copy tracing faster. Specifically, I'm thinking out recording which files were added or removed so that copies._chaincopies() doesn't have to look at the manifest for that. But that would just be an optimization and that can be added once we know if it's necessary. I have also considered saving space by using replacing the destination file path by an index into the "files" list, but that can also be changed later (but before the feature is ready to release). Differential Revision: https://phab.mercurial-scm.org/D6183

revsetbenchmark: add some simpler revset for heads and roots This revset might leverage compiled code in the future so lets start to track them.

repoview: flag `server.view` as experimental Ideally, the non-experimental version of `experimental.extra-filter-revs` will cover the use case for `server.view=immutable` well enough than having to have this dedicated configuration. Since `server.view` is not part of any release, I would prefer to have it marked as experimental to avoid having it to support it for ever.

repoview: move subsettable in a dedicated module The dictionary got moved in `branchmap` to avoid import cycle. However, we are about to needs it in repoview too. So we introduce a now module to define that that mapping.

upgrade: support upgrade to/from zstd storage (issue6088) Now that we have an official config option for a shiny format improvement, we better make it simple to migrate to/from it.

compression: introduce an official `zstd-revlog` requirement This requirement supersedes `exp-compression-zstd`. However, we keep support for the old requirement. Strictly speaking, we do not need to add a new requirement, there are no logic change making "new" repo incompatible with mercurial client using a version that support `exp-compression-zstd`. The choice to introduce a new requirement is motivated by the following: * The previous requirement was explicitly "experimental". Using it by default could confuse users. * adding support for a hypothetical third compression engine will requires new code, and will comes with its own requirement tool. * We won't use it as the default for a while since I do not think we support zstd on all platform. I can imagine we'll gain another (unrelated but on my default) requirement by the time we turn this zstd by default.

compression: only declare revlog support for available engine Even if we know that an engine support revlog compression, the Mercurial process still won't support it if the compression engine is not available.

branchcache: lazily validate nodes in iteritems() This saves ~0.30 sec on creating a new branch on our internal repo. Differential Revision: https://phab.mercurial-scm.org/D6236

branchcache: only iterate over branches which needs to be verified Otherwise we loop over all the branches and call _verifybranch() even if not required. Differential Revision: https://phab.mercurial-scm.org/D6240

branchcache: fix the docstring of _verifybranch() Initially the function was designed to support verifying all branches but later I decided to have a separate function. I forget to remove the doc related to that. Thanks to Yuya for spotting this in review. Differential Revision: https://phab.mercurial-scm.org/D6239

branchcache: don't verify while creating a copy The copy will not be mark as verified, so there is no need to verify nodes. Thanks to Yuya who spotted this while reviewing. Differential Revision: https://phab.mercurial-scm.org/D6238

pull: improved message issued in case of failed update When running `hg pull --update`, the update step may fail. Nothing in the error message help to understand the abort is related to the secondary step (update) instead of the primary step (pull). We now add some information to the error message to clarify it comes from the update part. It is useful in various situation (uncommitted changes blocking the update, update to hidden destination, etc...) The pull output is updated from: $ hg pull ../repo-Bob --rev 956063ac4557 --update pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: filtered revision '6'! to: $ hg pull ../repo-Bob --rev 956063ac4557 --update pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: cannot update to target: filtered revision '6'! (I am not sure why the actual error, "filtered revision '6'", is not using the more modern format mentioning the obsolescence fate of '6')

pull: deal with locally filtered changeset passed into --rev Nowadays, it is possible to explicitly pull a remote revision that end up being hidden locally (eg: obsoleted locally). However before this patch, some internal processing where crashing trying to resolve a filtered revision. Without this patches, the pull output result a confusing output: $ hg pull ../repo-Bob --rev 956063ac4557 pulling from ../repo-Bob searching for changes adding changesets adding manifests adding file changes added 2 changesets with 0 changes to 2 files (+1 heads) (2 other changesets obsolete on arrival) abort: 00changelog.i@956063ac4557828781733b2d5677a351ce856f59: filtered node!

absorb: aborting if another operation is in progress This increases safety of using absorb by both aborting when another operation is in progress (since the absorption could confuse any other command a lot) and holding the locks throughout the reading of the working directory (for which changes to absorb) and the reading of the repo (for which changes to absorb into). Differential Revision: https://phab.mercurial-scm.org/D6237

repoview: keep the branchmap cache for the `served.hidden` view warm For the same reason we want to keep the cache for the `served` view up to date, we want to also keep the `served.hidden` view up to date. If some processes with a readonly access to the repo needs to access it, we better have the cache warm to avoid computing the same data over and over and over. In most case (no secret changesets), the "served" and "served.hidden" set will be identical and no cache will actually have to be updated.

repoview: introduce a filter for serving hidden changesets There are multiple usecase for being able to explicitly view or pull obsolete from a server. We need to be able to do so without exposing the secret changesets. We introduces a dedicated repository "view" to do so. Way to expose this "view" to the user will come later. To keep a behavior consistent with expected client/server behavior, the general idea is for the obsolete access to be explicitly requested by the code generating the request. In addition, the will be server side configuration to restrict the access to this feature.

repoview: fix conditional around unserved changesets The conditional could lead to wrong computation since we have more unserved changesets than just the "secret" phase. In addition, now that we have efficient caching of phased changesets, we don't need the conditional anymore.

branch: return early if we find an open named branch apart from default The current code builds a list of all the open named branches except default and then bool that. This is mostly fine until you get a repo which has thousands of named branches. Differential Revision: https://phab.mercurial-scm.org/D6211

branchcache: don't verify closed nodes in _branchtip() We only do membership testing there. Differential Revision: https://phab.mercurial-scm.org/D6210

branchcache: don't verify closed nodes in iteropen() We expect that the nodes passed to iteropen() will be verified. We are only testing for membership in closed nodes set, so we don't need to verify the whole closed nodes set. This will speed up calculating branchheads() when there are lot of closed nodes related to other branches. Differential Revision: https://phab.mercurial-scm.org/D6209

branchcache: lazily validate nodes from the branchmap On my personal hg-repository with 365 entries in .hg/cache/branch2, following are the numbers for perfbranchmapload. Before this patch: ! wall 0.000866 comb 0.000000 user 0.000000 sys 0.000000 (best of 2680) ! wall 0.001525 comb 0.000000 user 0.000000 sys 0.000000 (max of 2680) ! wall 0.001107 comb 0.001097 user 0.001086 sys 0.000011 (avg of 2680) ! wall 0.001104 comb 0.000000 user 0.000000 sys 0.000000 (median of 2680) With this patch: ! wall 0.000530 comb 0.000000 user 0.000000 sys 0.000000 (best of 4240) ! wall 0.001078 comb 0.000000 user 0.000000 sys 0.000000 (max of 4240) ! wall 0.000696 comb 0.000693 user 0.000677 sys 0.000017 (avg of 4240) ! wall 0.000690 comb 0.000000 user 0.000000 sys 0.000000 (median of 4240) On our internal repository with ~20k entries in branchcache, I see improvement from 0.125 sec to 0.066 sec which is 47% speed up. The above are the numbers of perfbranchmapload which shows how much time we saved by not validating the nodes. But we need to validate some nodes. Following are timings of some mercurial operations which have speed up because of this lazy validation of nodes: No-op `hg update` on our internal repository (Avg on 4 runs): Before: 0.540 secs After: 0.430 secs Setting a branch name which already exists without --force (Avg of 4 runs): Before: 0.510 secs After: 0.250 secs I ran the ASV performance suite and was unable to see any improvements except there was improvement of perfdirstatewrite() on netbeans which I think was not related. I looked into the commit code, the command which I am trying to speedup, it looks like it uses revbranchcache to update the branchcache. Differential Revision: https://phab.mercurial-scm.org/D6208

branchcache: add functions to validate changelog nodes This patch adds functions to validate closed nodes, validate nodes for a certain branch and for all the branches. These functions will be used in upcoming patches. Differential Revision: https://phab.mercurial-scm.org/D6207

readline: provide styled prompt to readline (issue6070) Differential Revision: https://phab.mercurial-scm.org/D6168

copies: move comment about implementation of mergecopies() to end When you start reading about mergecopies(), you want to know what it is, not that there are different implementations depending on config, so this patch moves that comment to the end. By the way, we don't seem to define what "copytracing" is. I'm just leaving it that way because I don't know what it is myself. It seems to be referred to only on mergecopies() (and not in pathcopies(), for example), so maybe "copytracing" is supposed to be exactly what mergecopies() does? Differential Revision: https://phab.mercurial-scm.org/D6235

remotefilelog: return expected type from copies overrides copies._computeforwardmissing() and copies._computenonoverlap() return sets, so the overrides should also do that. Differential Revision: https://phab.mercurial-scm.org/D6234

changelog: extract a _string_unescape() to mirror _string_escape() We use our own _string_escape() to encode the "extras" field. Then we use codecs.escape_decode() to escape it. But there's also a little workaround for dealing with escaped text that looks like octal numbers since the fix for https://bz.mercurial-scm.org/show_bug.cgi?id=3156. This patch extracts the call to codecs.escape_decode() along with the fix for octal numbers and puts it in a _string_unescape(). It also updates the test to check for the octal-number case from the aforementioned bug. As you may have suspected, I want to be able to reuse this new function later. Differential Revision: https://phab.mercurial-scm.org/D6184

copies: extract function for deciding whether to use changeset-centric algos We'll eventually have a "experimental.copies.read-from=changeset-only" option too and I don't want to spread the logic for determining if we should use changeset-centric of filelog-centric algorithms. Differential Revision: https://phab.mercurial-scm.org/D6163

getrenamedfn: get copy data from context object if configured The function returned from getrenamedfn() calls filelog.renamed(). That won't work when storing copy metadata in the changeset. I've just switched to a simple implementation here. We may or may not need to optimize it later, possibly by optimizing the callers. No more tests fail with "--extra-config-opt experimental.copies.read-from=compatibility)" than they did before this patch. Differential Revision: https://phab.mercurial-scm.org/D6162

branchmap: implement __contains__() We have good occurences of `if branch in branchmap()` in our code. If __contains__() is not implemented then it will use __iter__() to find whether the element exists or not which is not good. I am bit confused that whether I should move existing callers to hasbranch() or this patch is a good way. Differential Revision: https://phab.mercurial-scm.org/D6206

branchmap: prevent using __getitem__() in branchheads() branchheads() can directly use self._entries instead. Differential Revision: https://phab.mercurial-scm.org/D6205

branchmap: dynamically resolve type of branchcache class This is required to support subclassing. Thanks to Yuya for suggesting this in D6151. Differential Revision: https://phab.mercurial-scm.org/D6204

copies: print list of divergent renames in sorted order The current order is not obvious to the user (and I may change it in a later patch). Lexicographical order seems like the obvious choice here. Differential Revision: https://phab.mercurial-scm.org/D6227

import: suppress accept: header irccloud and others may perform content negotation and reject (406) if the accept header does not match the mime type of the object

rebase: fix bug that prevented dry-run rebases from printing failures As far as I can tell it should be fine to unconditionally skip _prepareabortorcontinue if we're in the process of raising an Abort here. Differential Revision: https://phab.mercurial-scm.org/D6226

rebase: demonstrate bug in dry-run mode which causes cycles to not be reported Differential Revision: https://phab.mercurial-scm.org/D6225

test: minor cleanup to test-server-view.t While looking into adding error output in this test, I did some cleanup.

repoview: improve documentation for `repo.filtered` method I am sitting next to Joerg Sonnenberger and we are discussion his experience with repoview. This first effect of this discussion is this documentation clarification.

revset: short docstring for checkstatus This is where all the action happens for the status-related revsets, and a little documentation doesn't hurt.

discovery: stop direct use of attribute of partialdiscovery Instead of accessing `undecided` directly for ui display purposes, we introduce a `stats()` method that could be extended in the future with more interesting information. This is in preparation for a forthcoming Rust version of this object. Indeed, attributes and furthermore properties are a bit complicated for classes in native code. We could go further and rename `undecided` to mark it private, but `_undecided` is already taken as support for `_undecided` lazyness.

overlayworkingctx: remove misleading trailing slash from directory pattern The paths passed into the matcher are normalized (this applies to include patterns and regular patterns, and to both glob kind and path kind), so the regex for input "foo/" ended up being "foo(?:/|$)". Once we have a (recursive) pattern kind only for directories, we could switch to that here and remove the "mfiles[0] == path" check. Until then, let's at least make it not misleading. Differential Revision: https://phab.mercurial-scm.org/D6224

overlayworkingctx: fix file/dir audit to be repo-relative Before this patch, test-rebase-inmemory.t would stop erroring out about the conflict if you added a "cd a" before line 252. That was because a glob matcher (which are relative) was unintentionally used. That happened because the matcher was given "include" patterns (not regular patterns), and "include" patterns are always glob by default (i.e. unless you write them including the kind prefix). IOW, the "default='path'" argument passed to ctx.match() was ignored. Differential Revision: https://phab.mercurial-scm.org/D6223

messages: replace some instances of "folder" by "directory" I'm pretty sure this is our preferred term. Differential Revision: https://phab.mercurial-scm.org/D6222

match: fix re2 compability broken in 2e2699af5649 When using re2, we call test_match() instead of match() on the compiled regex object. While match() returns a matcher object or None, test_match() returns True or False. So since 2e2699af5649 running test on a machine with a re2 install fails in many places. Instead we make the code a bit more general and everything goes back to normal.

py3: add b'' prefixes to new doctests in match.py # skip-blame as just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D6221

py3: add one new passing test found by buildbot Differential Revision: https://phab.mercurial-scm.org/D6220

cext: cast s# arguments of Py_BuildValue() to Py_ssize_t The doc doesn't state that "s#" of Py_BuildValue() is controlled by PY_SSIZE_T_CLEAN (unlike the one for PyArg_ParseTuple()), but actually it's switched to Py_ssize_t. https://docs.python.org/2/c-api/arg.html#c.Py_BuildValue https://github.com/python/cpython/blob/2.7/Python/modsupport.c#L432 Follow up for b01bbb8ff1f2 and 896b19d12c08.

remotefilelog: correctly reject wdir filenodes This fixes `hg grep -r 'wdir()'` when remotefilelog is enabled and the working directory contains uncommitted modifications. Differential Revision: https://phab.mercurial-scm.org/D6217

remotefilelog: add tests of `hg grep -r 'wdir()'` This demonstrates how remotefilelog breaks grepping dirtied working directories. A future change will introduce a fix. Differential Revision: https://phab.mercurial-scm.org/D6216

config: read configs from directories in lexicographical order Mercurial currently reads the .rc files specified in HGRCPATH (and the system-default paths) in directory order, which is unspecified. My team at work maintains a set of .rc files. So far there has been no overlap between them, so we had not noticed this behavior. However, we would now like to release some common .rc files and then have another one per plaform with platform-specific overrides. It would be nice if we can determine the load order by choosing names carefully. This patch enables that by loading the .rc files in lexicographical order. Before this patch, the added test case would consistently say "30" on my file system (whatever I have -- some Linux FS). Differential Revision: https://phab.mercurial-scm.org/D6193

remotefilelog: fix crash on `hg addremove` of added-but-deleted file If you `hg add` a file and then delete it from disk, and then run `hg addremove`, the file ends up in the "removed" set that gets passed to the findrenames() override. We then crash because the file is not in the working copy parent. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D6194

packaging: ensure that --python is an absolute path when building on Windows For whatever reason, even though only python2 is on PATH, passing `python.exe` causes the later check that it's not py3 to bail out.

packaging: don't crash building wix with python3.6 and earlier `capture_output` was added in 3.7. I was tempted to just check and abort in build.py, since Windows doesn't have the Linux problem where some distros only ship an older python. But this is in a library that could be used elsewhere in the future.

chistedit: add basic colours to diff view This isn't complete, and it would be nice to show the exact same colours that `hg diff` would show. That goal is too lofty, so this just shows some basic colours, on the premise that a little is better than nothing.

chistedit: use default curses colours Terminals will define default colours (for example, white text on black background), but curses doesn't obey those default colours unless told to do so. Calling `curses.use_default_colors` makes curses obey the default terminal colours. One of the most obvious effects is that this allows transparency on terminals that support it. This also brings chistedit closer in appearance to crecord, which also uses default colours. The call may error out if the terminal doesn't support colors, but as far as I can tell, everything still works. If we need a more careful handling of lack of colours, blame me for not doing it now.

match: let regex match function return a boolean Match function for regex pattern kind is built through _buildregexmatch() and _buildmatch() using _rematcher() that returns a re.match function, which either returns a match object or None. This does not conform to Mercurial's matcher interface for __call__() or exact(), which are expected to return a boolean value. We fix this by building a lambda around _rematcher() in _buildregexmatch(). Accordingly, we update doctest examples to remove bool() calls that are now useless.

match: make arguments of _expandsets() optional Arguments 'ctx', 'listsubrepos' and 'badfn' are optional in function body.

match: make _donormalize's auditor and warn arguments optional Argument 'warn' is actually non-required, since there's a 'if warn:' check before usage. Argument 'auditor' is passed to pathutil.canonpath(), in which it is optional.

match: add doctest examples in match() Make the docstring raw, as it now includes escape characters.

match: complete documentation of match() parameters

match: add doctest examples for patkind()

match: add a docstring with doctest examples to patternmatcher Doctest examples aim at illustrating how __call__() and exact() are different, depending on the pattern kind.

match: add doctest examples for exactmatcher Make the docstring raw, since it now includes escape characters.

localrepo: don't allow lookup of working directory revision It seems that repo.lookup(), which is what supports the "lookup" wire protocol command, should not allow the working copy revision input. This fixes both the pull test and the convert test I just added. Differential Revision: https://phab.mercurial-scm.org/D6215

tests: demonstrate broken pull of "ffffffffffff" revision Differential Revision: https://phab.mercurial-scm.org/D6214

tests: demonstrate broken `hg convert` if "ffffffffffff" is in description Differential Revision: https://phab.mercurial-scm.org/D6213

tests: add test of for hash reference translation by `hg convert` The convert extension translates commit references in the commit message. We didn't have any explicit testing of this before, so let's add a test. Differential Revision: https://phab.mercurial-scm.org/D6212

py3: write out hgextindex as bytes in setup.py I hit this trying to build the py2exe target using python3, just to see what would happen. After commenting out `py2exe.Distribution` in setup.py and pointing to a local copy of py2exe that supports python3[1], it complained that `out` was bytes, not str. [1] https://github.com/albertosottile/py2exe/releases/tag/v0.9.3.0

setup: fix a possible NameError on rust build File "setup.py", line 975, in rustbuild "command: %r, environment: %r" % (self.rustsrcdir, cmd, env)) NameError: global name 'cmd' is not defined

crecord: new keys g & G to navigate to the top and bottom respectively This patch introduces two new keys 'g' and 'G' that helps to navigate to the top and bottom of the file/hunk/line respectively. This is inline with the shortcuts used in man, less, more and such tools that makes it convenient to navigate swiftly. 'g' or HOME navigates to the top most file in the ncurses window. 'G' or END navigates to the bottom most file/hunk/line depending on the whether the fold is active or not. If the bottom most file is folded, it navigates to that file and stops there. If the bottom most file is unfolded, it navigates to the bottom most hunk in that file and stops there. If the bottom most hunk is unfolded, it navigates to the bottom most line in that hunk. Differential Revision: https://phab.mercurial-scm.org/D6178

chistedit: properly show verbose diffs I'm not sure if that ever worked and it's an internal API breakage, but `"verbose": True` is not correctly parsed, as most of these options are parsed by diffopts, whereas verbose is a global option. Setting the UI to verbose instead does work and does show a verbose patch, with full commit message. It also shows all files, which unfortunately are a bit hard to read on a single line in the default verbose template. Thus, we also change the default template to use the status template, which shows one file per line as well as its modification state.

interactive: do not prompt about files given in command line For commit and revert commands with --interactive and explicit files given in the command line, we now skip the invite to "examine changes to <file> ? [Ynesfdaq?]". The reason for this is that, if <file> is specified by the user, asking for confirmation is redundant. In patch.filterpatch(), we now use an optional "match" argument to conditionally call the prompt() function when entering a new "header" item. We use .exact() method to compare with files from the "header" in order to only consider (rel)path patterns. Add tests with glob patterns for commit and revert, to make sure we still ask to examine files in these cases.

zstandard: vendor python-zstandard 0.11 The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. The project contains a vendored copy of zstandard 1.3.8. The old version was 1.3.6. This should result in some minor performance wins. test-check-py3-compat.t was updated to reflect now-passing tests on Python 3.8. Some HTTP tests were updated to reflect new zstd compression output. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D6199

cext: make osutil.c PY_SSIZE_T_CLEAN This is needed to avoid a deprecation warning on Python 3.8. With this change, we no longer see deprecation warnings for this issue on Python 3.8. Differential Revision: https://phab.mercurial-scm.org/D6198

cext: make parsers.c PY_SSIZE_T_CLEAN This is needed to avoid a deprecation warning in Python 3.8. I believe the conversion of int to Py_ssize_t is harmless in the changed locations. But this being C code, it should be audited with care. Differential Revision: https://phab.mercurial-scm.org/D6197

cext: make revlog.c PY_SSIZE_T_CLEAN Without this, Python 3.8 emits a deprecation warning, as using int for # values is deprecated. Many existing modules use PY_SSIZE_T_CLEAN, so this shouldn't be contentious. I audited the file for all # formatters and verified we are using Py_ssize_t everywhere now. Differential Revision: https://phab.mercurial-scm.org/D6196

tests: add optional output for Python 2.7 deprecation We already had one of these a few lines above. We need it here as well. Differential Revision: https://phab.mercurial-scm.org/D6203

setup: use raw string for regular expression Otherwise Python 3.8 complains about the backslash. Differential Revision: https://phab.mercurial-scm.org/D6202

automation: use raw strings when there are backslashes Otherwise Python 3.8 complains. Differential Revision: https://phab.mercurial-scm.org/D6201

perf: make perf.run-limits code work with Python 3 We need b'' because perf.py isn't run through the source transformer. We need to cast the exception to bytes using pycompat.bytestr() because ValueError can't be %s formatted due to built-in exceptions lacking __bytes__. We need to pycompat.sysstr() before the float() and int() cast so the ValueError message doesn't have b'' in it. Even with that, it looks like the error message for the ValueError for float casts added quotes, so we need to account for that in test output. Differential Revision: https://phab.mercurial-scm.org/D6200

localrepo: rename crev in _filecommit() to cnode, since it's a node I know we often use "rev" generically, but here's it always a node, so it helps to be specific. Differential Revision: https://phab.mercurial-scm.org/D6181

tests: unset environment variable P in test-revset2.t (issue6109) The test tests/test-revset2.t fails the test case "Test repo.anyrevs with customized revset overrides" (line 1609) if the environment variable P is set. The test implicitly expects that the environment, in which it is started, does not export the variable 'P'. To solve this issue, unset 'P' right before the test commands are run. Differential Revision: https://phab.mercurial-scm.org/D6195

hgmanpage: use a py2 and py3 compatible iterable protocol

hgmanpage: use range instead of xrange

packaging: allow to run make with python3 Use "?=", otherwise the variable cannot be set from environment.

cleanup: use set literals where possible Differential Revision: https://phab.mercurial-scm.org/D6192

tests: rename "u" to more usual "ui" in test-context.py Differential Revision: https://phab.mercurial-scm.org/D6191

tests: better document the graft copy case Differential Revision: https://phab.mercurial-scm.org/D6190

py2exe: add workaround to allow bundling of hgext3rd.* extensions py2exe doesn't know how to handle namespace packages *at all*, so it treats them like normal packages. As a result, if we try and bundle hgext3rd.evolve in a py2exe build, it won't work if we install evolve into the virtualenv. In order to work around this, tortoisehg installs hgext3rd.evolve etc into its staged hg directory, since it doesn't use a virtualenv. As a workaround for us, we'll just allow any extra packages users want bundled are part of hg during the pseudo-install phase that py2exe uses. I'm not happy about this, but it *works*. As a sample of how you'd make an MSI with evolve bundled: import os import shutil import subprocess import tempfile def stage_evolve(version): """Stage evolve for inclusion in py2exe binary.""" with tempfile.TemporaryDirectory() as temp: evolve = os.path.join(temp, "evolve") subprocess.check_call([ "hg.exe", "clone", "https://www.mercurial-scm.org/repo/evolve/", "--update", version, evolve, ]) dest = os.path.join('..', 'hgext3rd', 'evolve') if os.path.exists(dest): shutil.rmtree(dest) shutil.copytree(os.path.join(evolve, "hgext3rd", "evolve"), dest) def main(): stage_evolve('tip') print("\0") print("hgext3rd") print("hgext3rd.evolve") print("hgext3rd.evolve.hack") print("hgext3rd.evolve.thirdparty") if __name__ == "__main__": main() is a script you can pass to the wix/build.py as --extra-packages-script, and the resulting .msi will have an hg binary with evolve baked in. users will still need to enable evolve in their hgrc, so you'd probably also want to bundle configs in your msi for an enterprise environment, but that's already easy to do with the support for extra features and wxs files in the wix build process. Differential Revision: https://phab.mercurial-scm.org/D6189

wix: fix the package build when not adding features Should have used ifdef, not if. Sigh. Differential Revision: https://phab.mercurial-scm.org/D6187

histedit: narrow the scope of discarded ui output In 34165875fa5df813bec3a0cd348932b304d44efb, a lot of the output from histedit was excluded. This slightly adjusts the scope of that exclusion, to both discard more uninsteresting messages, and ensure that pre-merge-tool output gets shown before the external merge tool is executed. Differential Revision: https://phab.mercurial-scm.org/D6177

uncommit: abort if an explicitly given file cannot be uncommitted (BC) I've gotten burned several times by this in the last few days. The former tests look simple enough, but if a good file and a bad file are given, the bad files are silently ignored. Some commands like `forget` will warn about bogus files, but that would likely get lost in the noise of an interactive uncommit. The commit command aborts if a bad file is given, so this seems more consistent for commands that alter the repository.

unshelve: disable unshelve during merge (issue5123) As stated in the issue5123, unshelve can destroy the second parent of the context when tried to unshelve with an uncommitted merge. This patch makes unshelve to abort when called with an uncommitted merge. See how shelve.mergefiles works. Commit structure looks like this: ``` ... -> pctx -> tmpwctx -> shelvectx / / second merge parent pctx = parent before merging working context(first merge parent) tmpwctx = commited working directory after merge(with two parents) shelvectx = shelved context ``` shelve.mergefiles first updates to pctx then it reverts shelvectx to pctx with: ``` cmdutil.revert(ui, repo, shelvectx, repo.dirstate.parents(), *pathtofiles(repo, files), **{'no_backup': True}) ``` Reverting tmpwctx files that were merged from second parent to pctx makes them added because they are not in pctx. Changing this revert operation is crucial to restore parents after unshelve. This is a complicated issue as this is not fixing a regression. Thus, for the time being, unshelve during an uncommitted merge can be aborted. (Details taken from http://mercurial.808500.n3.nabble.com/PATCH-V3-shelve-restore-parents-after-unshelve-issue5123-tt4036858.html#a4037408) Differential Revision: https://phab.mercurial-scm.org/D6169

wix: add functionality to inject additional Features into installer This is the last bit required to be able to glue extra configs etc into the installer. Differential Revision: https://phab.mercurial-scm.org/D6180

wix: add support for additional wxs files As with my previous change for an --extra-prebuiild-script, I'm assuming this is predominantly useful in an enterprise environment and am only adding this to wix and not also to inno install scripts. Differential Revision: https://phab.mercurial-scm.org/D6179

wix: add a hook for a prebuild script to inject extra libraries I need this to build packages for Google so we can bundle some extensions in the installed image. My assumption is that this is most interesting for the .msi images so I only wired it up there. I'm not thrilled with the interface this provides, but it was an easy way to retain debug messages on Windows while also having enough structure to know what lines are actually module names for py2exe. Still pending on my end: I need to bundle a couple of config files, and at least one data file. I'm open to advice on how to do those things, and how to do this better. Differential Revision: https://phab.mercurial-scm.org/D6164

compression: introduce an official `format.revlog-compression` option This option supersedes the `experiment.format.compression` option. The value currently supported are zlib (default) and zstd (if Mercurial was compiled with zstd support). The option gained an explicit reference to `revlog` since this is the target usage here. Different storage methods might require different compression strategies. In our tests, using zstd give a significant CPU usage improvement (both compression and decompressing) while keeping similar repository size. Zstd as other interresting mode (dictionnary, pre-text, etc…) that are probably worth exploring. However, just plain switching from zlib to zstd provide a large benefit.

compression: display compression level in debugformat Now that we have options to control the compression level, we teach `hg debugformat` about them. This is a useful information when comparing repositories. Note that we have no trace of the compression level used to store existing deltas. Actually, it would even varies from one delta to another. So we display the currently set value.

compression: introduce a `storage.revlog.zstd.level` configuration This option control the zstd compression level used when compressing revlog chunk. The usage of zstd for revlog compression has not graduated from experimental yet, but we intend to fix that soon. The option name for the compression level is more straight forward to pick, so this changesets comes first. Having a dedicated option for each compression engine is useful because they don't support the same range of values. I ran the same measurement as for the zlib compression level (in the parent changesets). The variation in repository size is stay mostly in the same (small) range. The "read/write" performance see smallish variation, but are overall much better than zlib. Write performance show the same tend of having better write performance for when reaching high-end compression. Again, we don't intend to change the default zstd compression level (currently: 3) in this series. However this is worth investigating in the future. The Performance comparison of zlib vs zstd is quite impressive. The repository size stay in the same range, but the performance are much better in all situations. Comparison summary ================== We are looking at: - performance range for zlib - performance range for zstd - comparison of default zstd (level-3) to default zlib (level 6) - comparison of the slowest zstd time to the fastest zlib time Read performance: ----------------- | zlib | zstd | cmp | f2s mercurial | 0.170159 - 0.189219 | 0.144127 - 0.149624 | 80% | 88% pypy | 2.679217 - 2.768691 | 1.532317 - 1.705044 | 60% | 63% netbeans | 122.477027 - 141.620281 | 72.996346 - 89.731560 | 58% | 73% mozilla | 147.867662 - 170.572118 | 91.700995 - 105.853099 | 56% | 71% Write performance: ------------------ | zlib | zstd | cmp | f2s mercurial | 53.250304 - 56.2936129 | 40.877025 - 45.677286 | 75% | 86% pypy | 460.721984 - 476.589918 | 270.545409 - 301.002219 | 63% | 65% netbeans | 520.560316 - 715.930400 | 370.356311 - 428.329652 | 55% | 82% mozilla | 739.803002 - 987.056093 | 505.152906 - 591.930683 | 57% | 80% Raw data -------- repo alg lvl .hg/store size 00manifest.d read write mercurial zlib 1 49,402,813 5,963,475 0.170159 53.250304 mercurial zlib 6 47,197,397 5,875,730 0.182820 56.264320 mercurial zlib 9 47,121,596 5,849,781 0.189219 56.293612 mercurial zstd 1 49,737,084 5,966,355 0.144127 40.877025 mercurial zstd 3 48,961,867 5,895,208 0.146376 42.268142 mercurial zstd 5 48,200,592 5,938,676 0.149624 43.162875 mercurial zstd 10 47,833,520 5,913,353 0.145185 44.012489 mercurial zstd 15 47,314,604 5,728,679 0.147686 45.677286 mercurial zstd 20 47,330,502 5,830,539 0.145789 45.025407 mercurial zstd 22 47,330,076 5,830,539 0.143996 44.690460 pypy zlib 1 370,830,572 28,462,425 2.679217 460.721984 pypy zlib 6 340,112,317 27,648,747 2.768691 467.537158 pypy zlib 9 338,360,736 27,639,003 2.763495 476.589918 pypy zstd 1 362,377,479 27,916,214 1.532317 270.545409 pypy zstd 3 354,137,693 27,905,988 1.686718 294.951509 pypy zstd 5 342,640,043 27,655,774 1.705044 301.002219 pypy zstd 10 334,224,327 27,164,493 1.567287 285.186239 pypy zstd 15 329,000,363 26,645,965 1.637729 299.561332 pypy zstd 20 324,534,039 26,199,547 1.526813 302.149827 pypy zstd 22 324,530,595 26,198,932 1.525718 307.821218 netbeans zlib 1 1,281,847,810 165,495,457 122.477027 520.560316 netbeans zlib 6 1,205,284,353 159,161,207 139.876147 715.930400 netbeans zlib 9 1,197,135,671 155,034,586 141.620281 678.297064 netbeans zstd 1 1,259,581,737 160,840,613 72.996346 370.356311 netbeans zstd 3 1,232,978,122 157,691,551 81.622317 396.733087 netbeans zstd 5 1,208,034,075 160,246,880 83.080549 364.342626 netbeans zstd 10 1,188,624,176 156,083,417 79.323935 403.594602 netbeans zstd 15 1,176,973,589 153,859,477 89.731560 428.329652 netbeans zstd 20 1,162,958,258 151,147,535 82.842667 392.335349 netbeans zstd 22 1,162,707,029 151,150,220 82.565695 402.840655 mozilla zlib 1 2,775,497,186 298,527,987 147.867662 751.263721 mozilla zlib 6 2,596,856,420 286,597,671 170.572118 987.056093 mozilla zlib 9 2,587,542,494 287,018,264 163.622338 739.803002 mozilla zstd 1 2,723,159,348 286,617,532 91.700995 570.042751 mozilla zstd 3 2,665,055,001 286,152,013 95.240155 561.412805 mozilla zstd 5 2,607,819,817 288,060,030 101.978048 505.152906 mozilla zstd 10 2,558,761,085 283,967,648 104.113481 497.771202 mozilla zstd 15 2,526,216,060 275,581,300 105.853099 591.930683 mozilla zstd 20 2,485,114,806 266,478,859 95.268795 576.515389 mozilla zstd 22 2,484,869,080 266,456,505 94.429282 572.785537

compression: introduce a `storage.revlog.zlib.level` configuration This option control the zlib compression level used when compression revlog chunk. This is also a good excuse to pave the way for a similar configuration option for the zstd compression engine. Having a dedicated option for each compression algorithm is useful because they don't support the same range of values. Using a higher zlib compression impact CPU consumption at compression time, but does not directly affected decompression time. However dealing with small compressed chunk can directly help decompression and indirectly help other revlog logic. I ran some basic test on repositories using different level. I am using the mercurial, pypy, netbeans and mozilla-central clone from our benchmark suite. All tested repository use sparse-revlog and got all their delta recomputed. The different compression level has a small effect on the repository size (about 10% variation in the total range). My quick analysis is that revlog mostly store small delta, that are not affected by the compression level much. So the variation probably mostly comes from better compression of the snapshots revisions, and snapshot revision only represent a small portion of the repository content. I also made some basic timings measurements. The "read" timings are gathered using simple run of `hg perfrevlogrevisions`, the "write" timings using `hg perfrevlogwrite` (restricted to the last 5000 revisions for netbeans and mozilla central). The timings are gathered on a generic machine, (not one of our performance locked machine), so small variation might not be meaningful. However large trend remains relevant. Keep in mind that these numbers are not pure compression/decompression time. They also involve the full revlog logic. In particular the difference in chunk size has an impact on the delta chain structure, affecting performance when writing or reading them. On read/write performance, the compression level has a bigger impact. Counter-intuitively, the higher compression levels improve "write" performance for the large repositories in our tested setting. Maybe because the last 5000 delta chain end up having a very different shape in this specific spot? Or maybe because of a more general trend of better delta chains thanks to the smaller chunk and snapshot. This series does not intend to change the default compression level. However, these result call for a deeper analysis of this performance difference in the future. Full data ========= repo level .hg/store size 00manifest.d read write ---------------------------------------------------------------- mercurial 1 49,402,813 5,963,475 0.170159 53.250304 mercurial 6 47,197,397 5,875,730 0.182820 56.264320 mercurial 9 47,121,596 5,849,781 0.189219 56.293612 pypy 1 370,830,572 28,462,425 2.679217 460.721984 pypy 6 340,112,317 27,648,747 2.768691 467.537158 pypy 9 338,360,736 27,639,003 2.763495 476.589918 netbeans 1 1,281,847,810 165,495,457 122.477027 520.560316 netbeans 6 1,205,284,353 159,161,207 139.876147 715.930400 netbeans 9 1,197,135,671 155,034,586 141.620281 678.297064 mozilla 1 2,775,497,186 298,527,987 147.867662 751.263721 mozilla 6 2,596,856,420 286,597,671 170.572118 987.056093 mozilla 9 2,587,542,494 287,018,264 163.622338 739.803002

compression: accept level management for zlib compression We update the zlib related class to be support setting the compression level. This changeset focus on updating the internal only. A way to configure this level will be introduced in the next changeset.

util: extract compression code in `mercurial.utils.compression` The code seems large enough to be worth extracting. This is similar to what was done for various module in `mercurial/utils/`. Since None of the compression logic takes a `ui` objet, issuing deprecation warning is tricky. Luckly the logic does not seems to have many external users.

merge: make "labels" argument to graft() optional, like it is for update() graft() just passes the argument on to update(), and update() doesn't require it, so graft() shouldn't either. Differential Revision: https://phab.mercurial-scm.org/D6175

revset: remove comment about linkrev workaround from user-facing docs I think the code is clear enough so we don't need to keep the comment at all (by now, most Mercurial developers are probably familiar with the linkrevs issues). Differential Revision: https://phab.mercurial-scm.org/D6176

shelve: let cmdutil.revert() take care of backing up untracked files cmdutil.revert() backs up untracked files, so I don't see a reason to do it shelve.mergefiles(). We have tests for this and they still pass. Differential Revision: https://phab.mercurial-scm.org/D6174

shelve: stop passing list of files to revert It seems to work just fine to not specify any files here. I suspect it looked the way it did for historical reasons. It apparently used to use merge instead of rebase until 1d7a36ff2615 (shelve: use rebase instead of merge (issue4068), 2013-10-23) and it makes sense to want to restrict the set of files then. I noticed this because of the files.extend(shelvectx.p1().files()). If the working copy was clean before, then shelvectx.p1() will be the working copy parent and that ended up adding all the files in that set. In our Google-internal Mercurial setup (including a FUSE) that was very noticeably slow when the working copy parent happened to have many files in large directories. This patch doesn't yet remove the call to shelvectx.p1().files(). We also use that set for deciding what to back up. I'm pretty sure it's safe to back up only the set of files we already back even if we no longer restrict the set of files to revert, so this patch should be safe on its own. Regardless, the next patch will delegate the backing-up to cmdutil.revert(). Incidentally, this also gets rid of a repo.pathto() that I had earlier wanted to get rid of. Differential Revision: https://phab.mercurial-scm.org/D6173

remotefilelog: prefetch files in deterministic order I have been troubleshooting some slowness in this area (it's unclear if it's the client or server that's to blame, but that's beside the point) and it's a lot easier to do troubleshoot if the files are prefetched in the same order each time. Differential Revision: https://phab.mercurial-scm.org/D6172

debugdiscovery: display time elapsed during the discovery step This is a useful information. Now that we perform more analysing after the discovery is done, it is worth have a more precise measurement. For serious timing analysis use `hg perfdiscovery`.

debugdiscovery: only list common heads on verbose The list of common heads is only part of the useful information. In addition on repository with many heads, the information is very not helpful (just fill a couple of screen with hash). As a result we hide it behind a --verbose flag.

debugdiscovery: drop duplicated information The old line informing about the local being a superset or subset of the remote is redundant with the newly introduced data. So we drop it.

debugdiscovery: display more statistic about the common set We display a lot more information now. Especially, we display the overlap between the common heads and the local/remote heads. There are various optimization geared toward heads, as a result, the less common the heads the more complex the discovery. Having this information easily accessible help when working on discovery.

debugdiscovery: small internal refactoring The part of the code displaying statistic is made independant from the one running the discovery. In the same do, the declaration of the discovery function is a bit simplified.

debugdiscovery: allow to select random seed during debugdiscovery run The randomness can lead to large timing difference, controling it is important.

discovery: move cl.hasnode outside of the for-loop IIUC, resolving attributes for changelog can lead to some overhead. So this patch moves that to outside of a for-loop. Differential Revision: https://phab.mercurial-scm.org/D6147

discovery: prevent deleting items from a dictionary Removing elements from Python dictionary is expensive. So let's prevent adding them instead. I added a newline to make code look a bit better. Differential Revision: https://phab.mercurial-scm.org/D6146

discovery: drop some unused sets Differential Revision: https://phab.mercurial-scm.org/D6145

discovery: prevent recomputing info about server and outgoing changesets We already iterate over the outgoing.missing above and lookup repo for them. So let's reuse info calculated at that time instead of recomputing that again. Also we calculate the set of remotebranches by doing set(remotemap), so let's reuse that again. Upcoming patches will clean things a bit more. Differential Revision: https://phab.mercurial-scm.org/D6144

crecord: draw on the whole screen When starting crecord, one can see that it has a small gap on the rightmost column which doesn't get used. This is in contrast to other interactive curses frontends such as chistedit. Disabling the displaying of the cursor allows drawing on the whole availabe area and thus some hacky code in align() could be removed. Differential Revision: https://phab.mercurial-scm.org/D6171

automation: perform tasks on remote machines Sometimes you don't have access to a machine in order to do something. For example, you may not have access to a Windows machine required to build Windows binaries or run tests on that platform. This commit introduces a pile of code intended to help "automate" common tasks, like building release artifacts. In its current form, the automation code provides functionality for performing tasks on Windows EC2 instances. The hgautomation.aws module provides functionality for integrating with AWS. It manages EC2 resources such as IAM roles, EC2 security groups, AMIs, and instances. The hgautomation.windows module provides a higher-level interface for performing tasks on remote Windows machines. The hgautomation.cli module provides a command-line interface to these higher-level primitives. I attempted to structure Windows remote machine interaction around Windows Remoting / PowerShell. This is kinda/sorta like SSH + shell, but for Windows. In theory, most of the functionality is cloud provider agnostic, as we should be able to use any established WinRM connection to interact with a remote. In reality, we're tightly coupled to AWS at the moment because I didn't want to prematurely add abstractions for a 2nd cloud provider. (1 was hard enough to implement.) In the aws module is code for creating an image with a fully functional Mercurial development environment. It contains VC9, VC2017, msys, and other dependencies. The image is fully capable of building all the existing Mercurial release artifacts and running tests. There are a few things that don't work. For example, running Windows tests with Python 3. But building the Windows release artifacts does work. And that was an impetus for this work. (Although we don't yet support code signing.) Getting this functionality to work was extremely time consuming. It took hours debugging permissions failures and other wonky behavior due to PowerShell Remoting. (The permissions model for PowerShell is crazy and you brush up against all kinds of issues because of the user/privileges of the user running the PowerShell and the permissions of the PowerShell session itself.) The functionality around AWS resource management could use some improving. In theory we support shared tenancy via resource name prefixing. In reality, we don't offer a way to configure this. Speaking of AWS resource management, I thought about using a tool like Terraform to manage resources. But at our scale, writing a few dozen lines of code to manage resources seemed acceptable. Maybe we should reconsider this if things grow out of control. Time will tell. Currently, emphasis is placed on Windows. But I only started there because it was likely to be the most difficult to implement. It should be relatively trivial to automate tasks on remote Linux machines. In fact, I have a ~1 year old script to run tests on a remote EC2 instance. I will likely be porting that to this new "framework" in the near future. # no-check-commit because foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D6142

contrib: PowerShell script to install development dependencies Configuring a Windows machine to hack on Mercurial is a bit of work and it isn't documented very well. This commit introduces a PowerShell script to automate going from a fresh Windows install to an environment suitable for building Mercurial, its installers, and running tests. Differential Revision: https://phab.mercurial-scm.org/D6141

chistedit: change in-progress message Saying "running histedit" is an artifact of when chistedit was a separate thing from histedit. I found the message a bit confusing, since wasn't I running histedit from the beginning, just from the curses interface? The whole thing is now histedit, both the curses interface and the underlying procedure to apply a plan, so let's use a message that doesn't make a distinction.

perf: copyedit a few documentation strings Differential Revision: https://phab.mercurial-scm.org/D6170

shelve: add --keep to list of allowables

perf: introduce a `perf.run-limits` options This options make it possible to configure the number of run that the extensions will perform. This is useful for automated benchmark or for performance measurement that need better accuracy.

perf: pass limits as a function argument The function applying the limit has no access to the configuration. Therefore, some higher layer will have to pass it as argument. We do this in an independent change to clarify the next change.

perf: more flexible implementation for checking stop conditions We want to make this logic simpler to configure. The first step is to stop hard-coding every values.

perf: document perfparents

perf: document config options We have configuration, so we better document it.

tests: use "perf" as a the extension name in test-contrib-perf.t This is simpler and seems more "inline" with the name people usually use for this extensions.

shelve: do not update when keeping changes, just move the dirstate This is to leave the working directory unchanged. We reuse the match object as an optimisation.

shelve: refactor _shelvecreatedcommit's match object into calling site We might need to use this match object again to move the dirstate in case the user requested to `--keep` the changes.

shelve: new keep option Does nothing yet. The docstring describes what it will soon be doing.

diff: support diffing explicit files in subrepos Most other commands support implied recursion based on file names already.

fix: make the order of the work queue deterministic This makes any output generated during the parallel phase of execution stable if parallelism is disabled. This helps write tests like that in the future. Differential Revision: https://phab.mercurial-scm.org/D6166

fix: allow fixing untracked files when given as arguments It's more useful to fix the file than to silently avoid it when the user does this: hg fix --working-dir untracked-file We will still do nothing to ignored files, even if they are specified. This may be safer, given that such files can unexpectedly slip into the arguments via a shell glob or fileset. Differential Revision: https://phab.mercurial-scm.org/D6165

branchcache: have a hasnode function to validate nodes Upcoming patches will delay node validation until it's required. So we need to a way in branchcache class to check whether a node exists or node. Other options were making repo or changelog an attribute of branchcache object. But the branchcache depends on filters so I decided to pass a function object. Differential Revision: https://phab.mercurial-scm.org/D6157

branchcache: add attributes to track which nodes are verified Half of the cost of loading branchcache comes from verifiying all the nodes it has. We don't need to verify all the nodes in all the cases. Sometimes we need to verify only a set of nodes for a set of branches. We can ignore nodes of other branches as we are not going to read them. This patch introduces two attributes to branchcache class. _verifiedbranches is a set which will tell the branches for which it's head nodes are verified. _closedverified is a boolean which will tell whether all closednodes are verified or not. Another idea which I had was to keep a set of nodes which are verified, but it will be more ugly and difficult to track things on node level. Differential Revision: https://phab.mercurial-scm.org/D6156

branchcache: make entries a private attribute Differential Revision: https://phab.mercurial-scm.org/D6155

branchcache: introduce hasbranch() This will be used to check whether a branch exists or not. This will optimized in future. Differential Revision: https://phab.mercurial-scm.org/D6154

branchmap: drop branchcache.setdefault() (API) All the callers are updated to call setdefault of branchcache.entries Differential Revision: https://phab.mercurial-scm.org/D6153

branchcache: rename itervalues() to iterheads() The itervalues() exists because branchcache() had a dict interface. Since it no longer has a dict interface, it makes sense to have better function names. If a person does not understand how branchcache stores info, it will be hard for them to guess what itervalues() does. Differential Revision: https://phab.mercurial-scm.org/D6152

branchmap: remove the dict interface from the branchcache class (API) The current branchmap computation involves reading the whole branchmap from disk, validating all the nodes even if they are not required. This leads to a lot of time on repos which have large branchmap or a lot of branches. On large repos, this can validate around 1000's of nodes. On some operations, like finding whether a branch exists or not, we don't need to validate all the nodes. Or updating heads for a single branch. Before this patch, branchcache class was having dict interface and it was hard to keep track of reads. This patch removes the dict interface. Upcoming patches will implement lazy loading and validation of data and implement better API's. Differential Revision: https://phab.mercurial-scm.org/D6151

test-template: fix stdio mode on Windows Otherwise, CBOR data would be corrupted. Spotted by Matt Harbison.

samplehgrcs: update the list of suggested extensions Back in the day, this was color and pager, both of which are now default. Churn isn't that popular, but the other four below (obviously?) are.

samplehgrcs: clarify which lines should be uncommented The original wording has confused at least one person. Hopefully it's clearer this way. https://stackoverflow.com/questions/55288177/adding-hg-strip-to-hgrc-config-file

templatefilters: add {x|cbor} filter for custom CBOR output

template: add CBOR output format The whole output is wrapped as an array just like the other serialization formats. It's an indefinite-length array since the size is unknown while encoding. Maybe we can add 'cbor-stream' (and 'pickle-stream') as needed.

memfilectx: override copysource() instead of using dummy nodeid The "_copied" property in basefilectx is used by renamed() and copysource(). committablefilectx (which memfilectx subclasses) overrides renamed() and writes it in terms of copysource() instead of _copied. That means that the nodeid part of "_copied" is memfilectx is unused. Let's instead override copysource() too so we don't need the "_copied". Differential Revision: https://phab.mercurial-scm.org/D6159

memctx: rename constructor argument "copied" to "copysource" (API) It's just the path, not the nodeid, so "copysource" seems more appropriate. Differential Revision: https://phab.mercurial-scm.org/D6158

crecord: redraw the screen when starting up chunkselector Failure to do this can cause screen corruption like: <headerline> [X] filename.cc <several blank lines> <output from previous iteration of split that happened to be here> I believe this might only happen in some terminals, and maybe only when using the "alternate screen". Regardless of the exact conditions to reproduce, it should be safe to always clear it when starting up and is probably the correct thing to do anyway :) Differential Revision: https://phab.mercurial-scm.org/D6131

crecord: redraw the screen on ctrl-L This is the normal use of Ctrl-L, so I think this is going to be what most people expect it to do. We're keeping the adjustment of what line we're scrolled to as well. I believe both to be necessary to handle otherwise inescapable situations when we've got screen corruption or edge-cases during window resizing. Differential Revision: https://phab.mercurial-scm.org/D6130

crecord: completely redraw screen when coming back from editor Differential Revision: https://phab.mercurial-scm.org/D6129

tests: glob seconds in test-upgrade-repo.t I had the test failing locally for me with diff showing `1.4s` instead of 0.0s Differential Revision: https://phab.mercurial-scm.org/D6161

store: recommend using `hg debugrebuildfncache` is fncache is corrupted Differential Revision: https://phab.mercurial-scm.org/D6160

debugsparse: abort if the repository is not sparse instead of ui.status() This is similar to what narrow extension does. Differential Revision: https://phab.mercurial-scm.org/D6149

revert: option to choose what to keep, not what to discard I know the you (the reader) are probably tired of discussing how `hg revert -i -r .` should behave and so am I. And I know I'm one of the people who argued that showing the diff from the working copy to the parent was confusing. I think it is less confusing now that we show the diff from the parent to the working copy, but I still find it confusing. I think showing the diff of hunks to keep might make it easier to understand. So that's what this patch provides an option for. One argument doing it this way is that most people seem to find `hg split` natural. I suspect that is because it shows the forward diff (from parent commit to the commit) and asks you what to put in the first commit. I think the new "keep" mode for revert (this patch) matches that. In "keep" mode, all the changes are still selected by default. That means that `hg revert -i` followed by 'A' (keep all) (or 'c' in curses) will be different from `hg revert -a`. That's mostly because that was simplest. It can also be argued that it's safest. But it can also be argued that it should be consistent with `hg revert -a`. Note that in this mode, you can edit the hunks and it will do what you expect (e.g. add new lines to your file if you added a new lines when editing). The test case shows that that works. Differential Revision: https://phab.mercurial-scm.org/D6125

patch: include newline at EOF in help text for interactive patch The lack of a newline means that some "editors" that are useful in tests, such as `echo "+new line" >> "$1"` don't work. It's obviously easy to work around it, but newline at EOF seems like a good practice anyway. Differential Revision: https://phab.mercurial-scm.org/D6124

merge with stable

Added signature for changeset 4ea21df312ec

Added tag 4.9.1 for changeset 4ea21df312ec

patch: include flag-only file changes in "special" when filtering (issue5864) This patch fix the issue5864 (or maybe issue5865 too) which occurs during split (or I should say at the time of filtering the hunks in interactive mode) where user hits a not ending loop of "no changes to record". And it's not only the case for split it will happen in every interactive case for e.g. `hg commit -i` or `hg uncommit -i` After looking into code I found that when filtering we have some notation called "special" for the file headers which doesn't contain any hunk and just contain the header (for e.g. newly added empty file or deleted file) where the user cannot change the content of operation. And I think we can put this "flag-only" file change in that same bucket of "special". But I doubt a bit about the case when a file have flag change and atleast one hunk then user won't be able to separate the flag change from hunks. Changed test file reflect the fixed behaviour. Differential Revision: https://phab.mercurial-scm.org/D6058

store: error out if fncache does not ends with a newline If fncache does not ends with a newline, chunk will not be fully consumed. It should be a bug somewhere or the fncache is corrupted if that happens. Let's error out in such cases. Differential Revision: https://phab.mercurial-scm.org/D6148

tracked: add documentation about `--import-rules` flag The documentation is inspired from the `--import-rules` flag of hg debugsparse command. Differential Revision: https://phab.mercurial-scm.org/D6150

discovery: fix embarrassing typo in slice definition The code introduced in e514799e4e07 ended up having a silly bug. The indexing selected a single item slice picking only p1. The discovery result was still correct, but the sampling was hampered, sometime leading to much more round trips being performed. Fixing this issue restore the previous sampling behavior. This fix has a negative performance impact on the pathological case the previous test has been built. # parent of this changesets ! wall 5.313884 comb 5.310000 user 5.260000 sys 0.050000 (best of 5) ! wall 6.711860 comb 6.710000 user 6.670000 sys 0.040000 (max of 5) ! wall 5.844016 comb 5.842000 user 5.784000 sys 0.058000 (avg of 5) ! wall 5.778635 comb 5.780000 user 5.740000 sys 0.040000 (median of 5) # With this changesets. ! wall 6.350879 comb 6.350000 user 6.300000 sys 0.050000 (best of 5) ! wall 6.653647 comb 6.660000 user 6.480000 sys 0.180000 (max of 5) ! wall 6.492762 comb 6.494000 user 6.414000 sys 0.080000 (avg of 5) ! wall 6.547577 comb 6.550000 user 6.490000 sys 0.060000 (median of 5) Changeset e514799e4e07 raised the question of using the "_uncheckedparentrevs" instead of the current code. So I ran comparative timing: # old code: 55919b96c02a (e514799e4e07 parent) ! wall 64.078708 comb 64.080000 user 63.160000 sys 0.920000 (best of 5) ! wall 68.296300 comb 68.290000 user 67.410000 sys 0.880000 (max of 5) ! wall 65.899075 comb 65.894000 user 65.082000 sys 0.812000 (avg of 5) ! wall 66.140286 comb 66.130000 user 65.330000 sys 0.800000 (median of 5) # buggy code: e514799e4e07 ! wall 46.605362 comb 46.610000 user 45.880000 sys 0.730000 (best of 5) ! wall 48.619659 comb 48.620000 user 47.890000 sys 0.730000 (max of 5) ! wall 47.350247 comb 47.350000 user 46.672000 sys 0.678000 (avg of 5) ! wall 46.983224 comb 46.980000 user 46.350000 sys 0.630000 (median of 5) # fixed code: e514799e4e07 with this fix ! wall 55.858460 comb 55.850000 user 55.090000 sys 0.760000 (best of 5) ! wall 59.048805 comb 59.060000 user 58.110000 sys 0.950000 (max of 5) ! wall 57.192639 comb 57.192000 user 56.350000 sys 0.842000 (avg of 5) ! wall 57.056373 comb 57.060000 user 56.160000 sys 0.900000 (median of 5) # version using uncheckedparents ! wall 56.471916 comb 56.470000 user 55.630000 sys 0.840000 (best of 5) ! wall 58.228793 comb 58.230000 user 57.600000 sys 0.630000 (max of 5) ! wall 57.377583 comb 57.378000 user 56.674000 sys 0.704000 (avg of 5) ! wall 57.008843 comb 57.010000 user 56.330000 sys 0.680000 (median of 5) So it looks like the overhead from `_uncheckedparentrevs` is not that impactful. I'll investigate this shortly. I'm almost done updating our benchmark suite with more meaningful discovery cases.

store: don't read the whole fncache in memory In large repositories with lot of files, the fncache grows more than 100 MB and reading that whole thing into memory slows things down. Let's not read the whole thing into memory. This patch changes fncache loading code to read 1 MB at once. Loading 1 MB at once saves ~1 sec on perffncacheload for our internal repository. I tried various values such as 0.5 MB, 5 MB, 10 MB but best results were produced using 1 MB as the chunksize. On a narrow clone with fncache around 40 MB, this patch saves ~0.04 seconds on average on perffncacheload. To test the code, I have coded an extension in test-fncache.t which set chunksize to 1 byte, and the test passes with that. Differential Revision: https://phab.mercurial-scm.org/D5296

record: prevent commits that don't pick up dirty subrepo changes (issue6102) This path covers interactive mode for commit, amend, and shelve, as well as the deprecated record extension. Since shelf creation uses commit without -S in the non-interactive case, aborting here should be OK. (I didn't check what happens to non interactive shelve creation if `ui.commitsubrepos=True` is set.) subrepoutil.precommit() will abort on a dirty subrepo if the config option isn't set, but the hint recommends using --subrepos to commit. Since only the commit command currently supports that option, the error has to be raised here to omit the hint. Doing the check before asking about all of the hunks in the MQ test seems like an improvement on its own. There's probably an additional check on this path that can be removed.

wix: restore COPYING.rtf 8427fea04017 accidentally blew away the content of this file. As part of restoring the content, I updated the copyright year to 2019. Differential Revision: https://phab.mercurial-scm.org/D6098

test-https: add some more known failure messages of client certs (issue6030) I don't think the exact error message is important here. On Debian sid, ECONNRESET is raised, and "[SSL] tlsv13 alert certificate required" on NetBSD.

test-https: turn off system OpenSSL configuration This mostly fixes the test failure on Debian sid where TLS 1.0 and 1.1 are disabled by default. https://sources.debian.org/patches/openssl/1.1.1a-1/Set-systemwide-default-settings-for-libssl-users.patch/ $OPENSSL_CONF could be set by run-tests.py, but the other tests should work without a "legacy" TLS, so I decided to not.

store: move logic to check for invalid entry in fncache to own function This helps separate the original reading logic from the one which finds for an invalid entry. Differential Revision: https://phab.mercurial-scm.org/D6030

py3: add test-phabricator.py to python3-whitelist Differential Revision: https://phab.mercurial-scm.org/D6114

py3: convert to/from bytes/unicode for json.(dump|load)s in debugcallconduit Differential Revision: https://phab.mercurial-scm.org/D6113

py3: use pycompat.byteskwargs on opts in phabricator.py Differential Revision: https://phab.mercurial-scm.org/D6107

watchman: ignore some of watchman errors Don't display 'illegal_fstypes' errors. In environments with network filesystems, the error messages are quickly pilling up and polluting outputs. Differential Revision: https://phab.mercurial-scm.org/D5955

watchman: add the possibility to set the exact watchman binary location This is necessary to make rolling releases of new watchman versions across users. Differential Revision: https://phab.mercurial-scm.org/D5954

context: use wdirhex constant instead of calculating it Differential Revision: https://phab.mercurial-scm.org/D6143

split: use the new movedirstate() we now have in scmutil This avoids unnecessarily touching the working copy when splitting the parent of the working copy. That also makes the test-removeemptydirs.t case invalid, so we can just delete it. Differential Revision: https://phab.mercurial-scm.org/D6127

manifestcache: use `wcache` directory for manifest cache The manifest full text cache is tightly related to the working copy. We should use the `wcache` directory for it, instead of the `cache`. Otherwise, multiple shares would keep overwriting each other cache entry and we loose its benefit. This is also more consistent with the fact this cache file is protected by `wlock`.

manifestcache: protect write with `wlock` instead of `lock` The `wlock` is taken by both `update` and `commit` type operation. This would help persisting the cache more aggressively. An explicit test is introduced. However, we can already see the effect of this change on earlier test output.

manifestcache: clear the cache before testing the debug command Right now the cache is empty before this section of the test. However, we are about to improve the persistence of the cache (putting it under `wllock`, intead of `lock`). So we install the cleanup before.

manifestcache: abstract the filename in a class attribute This make the code clearer and simpler to update.

manifestcache: skip setup earlier if we don't have the lock There a no point preparing a closure if we are not going to use it.

manifestcache: test the cache is warm after a commit

manifestcache: stop altering the lru cache order while displaying it Accessing value with `.get` alter the iteration order and make the output of the debug command misbehave, showing multiple entry twice. We need more than 2 entry to see the bug, so there are not test change. Later test will introduce a third entry and would fail without this fix.

manifestcache: support multiple cache addition in one debug command run This is more practical.

wix: autogenerate wxs file for library files Currently, dist.wxs contains an enumeration of .pyd and .dll files staged to dist/lib by py2exe. Having a manual list of files is error prone, as things can easily get out of sync (as the previous commit demonstrates). This is especially an issue for TortoiseHG, which ships a number of custom modules, which may pull in additional dependencies. Let's prevent this problem from manifesting by dynamically generating a .wxs file containing the .pyd and .dll files staged by py2exe. Differential Revision: https://phab.mercurial-scm.org/D6139

wix: introduce variable to hold path to wix packaging directory For convenience. Differential Revision: https://phab.mercurial-scm.org/D6138

wix: package missing .dll and .pyd files dist.wxs is currently missing some .pyd and .dll files which are picked up and staged by py2exe. This means that the WiX installer is missing some Python extension modules and their dependencies which are referenced by Mercurial or a Python package distributed with it. This commit adds the missing files to the WiX installer. Differential Revision: https://phab.mercurial-scm.org/D6137

setup: exclude crypt32.dll in py2exe builds py2exe is picking up crypt32.dll as a dependency and is including the DLL in the dist/lib directory, where it can get picked up by an installer and distributed. crypt32.dll is a core Windows DLL since Windows XP. We don't need to distribute it. Differential Revision: https://phab.mercurial-scm.org/D6136

packaging: don't bundle DLLs in py2exe library.zip for x86 builds I had ported the x86/x64 behavior difference from the Inno Setup installer files. Why things were this way, I'm not sure. The WiX configuration files are expecting to have standalone DLL files for both configurations. And the 32-bit WiX installers were broken due to missing DLLs. Let's standardize on standalone DLL files on all configurations for consistency. I /think/ this will be faster, as I /think/ py2exe binaries would have to extract the DLL to a temporary file in order to load it. But I'm not 100% sure about that. Differential Revision: https://phab.mercurial-scm.org/D6135

packaging: convert files to LF My editor accidentally wrote CRLF line endings because I authored this code on Windows. Derp. I'm kinda surprised no linters caught this... # no-check-commit to avoid false positive for foo_bar names Differential Revision: https://phab.mercurial-scm.org/D6134

dirstate: remove obsolete reference to dirstate.beginparentchange The only valid API since 265e91da56fd (dirstate: drop deprecated methods (API), 2018-02-02) is the context manager returned from dirstate.parentchange(). Differential Revision: https://phab.mercurial-scm.org/D6126

py3: use pycompat.iterbytestr to convert memoryview slice to bytestring Otherwise ch is the int value of the byte in py3 rather than the actual character. Differential Revision: https://phab.mercurial-scm.org/D6103

rebase: fix crash with in-memory rebase and copies When using regular on-disk rebase, filectx.markcopies() calls to dirstate.copy(), which happily records the copy. Then it's simply ignored if it doesn't matter for the commit (as in the test case I added in the previous patch). Let's do the same for overlayworkingctx. Differential Revision: https://phab.mercurial-scm.org/D6133

test: demonstrate crash with in-memory rebase and copies In the added test case, there is a merge commit that has one obsolete parent with a rename. Since the rename is not in the other parent, pathcopies() from that other parent will include the copy. Then when we try to rebase this merge commit onto another commit that has the same content changes, but no tracking of the rename (because it was done with "hg remove; hg add" instead of "hg mv"), we try to propagate the copy information. That fails because overlayworkingctx expects a file to be modified if it's going to have copy information. Differential Revision: https://phab.mercurial-scm.org/D6132

manifestcache: actually honor --clear Before this change, the --clear flag was not clearing the on disk cache. (We also remove the extra verbosity when using --clear. Same as what we did for --add)

manifestcache: make sure the entry are ordered by access time This is an LRU cache, let us make sure of that.

manifestcache: adding a second distinct entry Let makes sure the cache can hold multiple value.

manifestcache: test that adding the same entry twice do not duplicates it Simple sanity check.

manifestcache: do not display data when using --add If the command invocation is about adding a new entry, we should remain terse (the same as we do for many commands).

manifestcache: only lock the repository if the debug command touch the cache Not doing so had two consequences: 1) the command cannot be run on read only repositories, 2) when using --add on an empty cache, the command crash prematurely trying to read the cache file on disk.

manifestcache: further fix to debug command output Removing more capital letters. The output will get a test once other issues get fixed.

manifestcache: test and fix some output of the debug command The message was lacking an end of line. In addition we do not capitalize output in Mercurial.

chunkselector: fix typos in instructions when user reviews patch Differential Revision: https://phab.mercurial-scm.org/D6121

scmutil: document matcher argument of movedirstate() Differential Revision: https://phab.mercurial-scm.org/D6120

uncommit: move _movedirstate() to scmutil for reuse The function should be applicable generically when moving from one commit to another. I'll try to add more callers when I find time. I'm not convinced it's handling all the cases correctly, but we should have a generic function for this kind of operation, so I think it belongs somewhere in core (not in the uncommit extension). Differential Revision: https://phab.mercurial-scm.org/D6119

copies: remove dependency on scmutil by directly using match.exact() I want to add a dependency from scmutil.copies(), so I need to remove this dependency first. Differential Revision: https://phab.mercurial-scm.org/D6118

uncommit: convert _fixdirstate() into _movedirstate() _fixdirstate() already also updates to the given commit, so let's rename it to _movedirstate(). Also update the documentation and drop the unnecessary "curctx" argument, since that should always be repo['.']. Differential Revision: https://phab.mercurial-scm.org/D6117

updatecaches: also warm the tags caches Resolving any name requires the tags cache to be warm. We make sure that `hg debugupdatecache` warm the tag cache entry too.

updatecaches: also warm revbranchcache for filtered revisions We are in the "full" case, so we better warm everything we can.

copies: handle a case when both merging csets are not descendant of merge base This patch fix the behaviour of fullcopytracing algorithm in the case when both the merging csets are not the descendant of merge base. Although it seems to be the rare case when both the csets are not descendant of merge base. But it can be seen in most of cases of content-divergence in evolve extension, where merge base is the common predecessor. Previous patch added a test where this algorithm can fail to continue because of an assumption that only one of the two csets can be dirty. This patch fix that error. For refrence I suggest you to look into the previous discussion held on a patch sent by Pulkit: https://phab.mercurial-scm.org/D3896 Differential Revision: https://phab.mercurial-scm.org/D5963

copies: add test that makes both the merging csets dirty and fails This patch is a part of series which is about the case when both the merging csets are not descendant of merge base. The existing code assumes if c1 is dirty there shouldn't be any partial copies from c2 i.e both2['incomplete'] and same for c2, if c2 is dirty both1['incomplete'] should be empty, but this is not the right assumption. Now as we know we can have both c1 and c2 dirty at the same time, it is possible that c1 is dirty and both2['incomplete'] has some value. Or if c2 is dirty and both1['incomplete'] has some value. Added test shows that because of this assumption it could fail. Differential Revision: https://phab.mercurial-scm.org/D5962

copies: add test that makes both the merging csets dirty and run w/o error This series of patches is to cover a case in fullcopytracing algorithms where both the merging csets are not descendant of merge base. In this algorithm we call a merging cset "dirty" if that cset is not the descendant of merge base. That said, added test in this patch cover case when both the merging csets are "dirty". Actually this case of "both dirty" was encountered by Pulkit when he was working on content-divergence where it is possible that both the csets are not descendant of merging base. For reference you can look into: https://phab.mercurial-scm.org/D3896 As this test run fine without any error and correctly traced the copies, I added this test to make sure that it doesn't break even after I will modify some code in next patches to fix an error. Next patch adds the tests where this algorithm throws an error for the same case of "both dirty". Differential Revision: https://phab.mercurial-scm.org/D5961

tests: stabilize test-bundle.t on Windows Similar to 92055d539e49.

discovery-helper: use reflink copy if available A reflink copy will copy the files "as usual" but keep using the same data block underneath. This is only supported by "copy on write" file system like btrfs or zfs. This will achieve similar performance that the existing hardlink clone that Mercurial performs with the same initial space saving. However, it will behave better on revlogs start being touch by strip. Instead of duplicating all data in the touched revlogs, only the block actually affected by the strip will be duplicated. This save a lot of space when building many variants of large repositories. The --reflink=always flag make sure the `cp` call fails if reflink copies are not supported. Falling back to local clone.

discovery-helper: bail out if destination already exists

discovery-helper: move repository creation in a function This makes it easier to update this duplicated code. (we do a small output fix as we go)

discovery-helper: add an extra argument to generate only one repo This is useful to generate left and right in parallel when dealing with very large repositories.

wix: remove enum and future packages These were cargo culted from the THG installer code. I'm not sure what needs them in THG land. But the official MSIs certainly do not - at least not as direct dependencies. .. bc:: The Windows MSI installers no longer include the enum and future Python packages. Differential Revision: https://phab.mercurial-scm.org/D6101

wix: remove pywin32 This dependency was for ancient Mercurial versions. We recently removed it from the Inno Setup installers. So let's remove it from the WiX installers as well. .. bc:: The Windows MSI installers no longer include the pywin32 Python package. Differential Revision: https://phab.mercurial-scm.org/D6100

wix: remove sphinx and dependencies Sphinx was cargo culted into our install environment as part of emulating TortoiseHG's behavior. THG seems to install Sphinx in order to generate THG specific documentation. We don't appear to need Sphinx or any of its dependencies in the official WiX installers. So remove it. This shaves ~1MB off the size of the MSI installers. .. bc:: The Windows MSI installers no longer include the Python sphinx package and its various dependencies. Differential Revision: https://phab.mercurial-scm.org/D6099

wix: functionality to automate building WiX installers Like we did for Inno Setup, we want to make it easier to produce WiX installers. This commit does that. We introduce a new hgpackaging.wix module for performing all the high-level tasks required to produce WiX installers. This required miscellaneous enhancements to existing code in hgpackaging, including support for signing binaries. A new build.py script for calling into the module APIs has been created. It behaves very similarly to the Inno Setup build.py script. Unlike Inno Setup, we didn't have code in the repo previously to generate WiX installers. It appears that all existing automation for building WiX installers lives in the https://bitbucket.org/tortoisehg/thg-winbuild repository - most notably in its setup.py file. My strategy for inventing the code in this commit was to step through the code in that repo's setup.py and observe what it was doing. Despite the length of setup.py in that repository, the actual amount of steps required to produce a WiX installer is actually quite low. It consists of a basic py2exe build plus invocations of candle.exe and light.exe to produce the MSI. One rabbit hole that gave me fits was locating the Visual Studio 9 C Runtime merge modules. These merge modules are only present on your system if you have a full Visual Studio 2008 installation. Fortunately, I have a copy of Visual Studio 2008 and was able to install all the required updates. I then uploaded these merge modules to a personal repository on GitHub. That is where the added code references them from. We probably don't need to ship the merge modules. But that is for another day. The installs from the MSIs produced with the new automation differ from the last official MSI in the following ways: * Our HTML manual pages have UNIX line endings instead of Windows. * We ship modules in the mercurial.pure package. It appears the upstream packaging code is not including this package due to omission (they supply an explicit list of packages that has drifted out of sync with our setup.py). * We do not ship various distutils.* modules. This is because virtualenvs have a custom distutils/__init__.py that automagically imports distutils from its original location and py2exe gets confused by this. We don't use distutils in core Mercurial and don't provide a usable python.exe, so this omission should be acceptable. * The version of the enum package is different and we ship an enum.pyc instead of an enum/__init__.py. * The version of the docutils package is different and we ship a different set of files. * The version of Sphinx is drastically newer and we ship a number of files the old version did not. (I'm not sure why we ship Sphinx - I think it is a side-effect of the way the THG code was installing dependencies.) * We ship the idna package (dependent of requests which is a dependency of newer versions of Sphinx). * The version of imagesize is different and we ship an imagesize.pyc instead of an imagesize/__init__.pyc. * The version of the jinja2 package is different and the sets of files differs. * We ship the packaging package, which is a dependency for Sphinx. * The version of the pygments package is different and the sets of files differs. * We ship the requests package, which is a dependency for Sphinx. * We ship the snowballstemmer package, which is a dependency for Sphinx. * We ship the urllib3 package, which is a dependency for requests, which is a dependency for Sphinx. * We ship a newer version of the futures package, which includes a handful of extra modules that match Python 3 module names. # no-check-commit because foo_bar naming Differential Revision: https://phab.mercurial-scm.org/D6097

wix: move contrib/wix to contrib/packaging/wix We're trying to consolidate all our packaging code into contrib/packaging. Let's move the WiX files there. Differential Revision: https://phab.mercurial-scm.org/D6096

wix: remove hg.cmd This file is not referenced anywhere AFAICT. Differential Revision: https://phab.mercurial-scm.org/D6095

setup: include hgext3rd package in py2exe builds This is a core Mercurial package and we should always ship it. Differential Revision: https://phab.mercurial-scm.org/D6094

setup: properly install build_hgextindex for py2exe builds Because the hgbuild class has a private copy of build.sub_commands, modifying build.sub_commands from this code effectively resulted in a no-op. Registering the sub-command on hgbuild actually results in the sub-command running when building Mercurial. Differential Revision: https://phab.mercurial-scm.org/D6093

setup: configure py2exe config via environment variables The Inno Setup and WiX installers ship a different set of packages with py2exe builds. And there are multiple WiX installer variants (e.g. TortoiseHG). Since there are multiple variants of py2exe configs and they can be defined by entities not in our repository, let's provide a mechanism for setup.py to supplement behavior via environment variables. This is slighly less hacky than a setup.cfg file IMO since the caller doesn't need to worry about mutating global state of the source directory. Differential Revision: https://phab.mercurial-scm.org/D6092

packaging: extract py2exe functionality to own module py2exe builds are shared between Inno Setup and WIX. We'll want the logic for performing py2exe builds to be reusable across the code for both installers. This commit extracts the py2exe-specific functionality into its own module. There's definitely room to customize things further. This will be done in future commits, as necessary. (I'm not even sure what customizations WIX will require yet. Presumably a lot.) Differential Revision: https://phab.mercurial-scm.org/D6091

packaging: extract python exe info to own function This is generic functionality. We'll need it for WIX. As part of the port, we expose the full version and return the data as a dict. Differential Revision: https://phab.mercurial-scm.org/D6090

packaging: don't use temporary directory We were no longer doing anything with it after extracting virtualenv and py2exe to the build directory. Differential Revision: https://phab.mercurial-scm.org/D6089

packaging: extract virtualenv and py2exe to build directory The build directory is essentially a cache. We can extract the virtualenv and py2exe package sources to this directory. Differential Revision: https://phab.mercurial-scm.org/D6088

packaging: move Inno Setup core logic into a module Aspects of building the Inno Setup and WIX installers are shared. It will make sense for them to share code. Plus, having code in a reusable library (as opposed to a standalone script) is just a better approach. This commit moves the core logic to build the Inno Setup installer into the hgpackaging package. inno/build.py is now a simple frontend script that calls into a module to do the bulk of the work. As part of this change, I also found a typo in build() where it was referencing "iscc" instead of "iscc_exe." Because "iscc" was in the global scope via the only caller, things just happened to work before. Another benefit of always using functions and not putting global code for __main__ in the same file as library code. Differential Revision: https://phab.mercurial-scm.org/D6087

packaging: move find_vc_runtime_files() into hgpackaging.util In preparation for moving the bulk of the Inno Setup code into hgpackaging. Differential Revision: https://phab.mercurial-scm.org/D6086

packaging: move DOWNLOADS dict to hgpackaging.downloads We'll want to keep state in sync between multiple packaging tools. It makes sense to share a central data structure defining downloads. We also change the function to return the downloads entry so callers don't have to access the global DOWNLOADS in the new location. Differential Revision: https://phab.mercurial-scm.org/D6085

packaging: split downloading code into own module As we will introduce more code to support packaging, it will be useful to have download code in its own module. Differential Revision: https://phab.mercurial-scm.org/D6084

packaging: establish hgpackaging package Previously, contrib/packaging behaved as a root to a package directory and we had a "packagingutil" module. As I work more on packaging code, we'll want to have more code shared between different packaging tools. I think it makes sense to have a single package containing multiple modules than multiple top-level modules. This commit establishes an "hgpackaging" package by moving the existing packagingutil code to it. Differential Revision: https://phab.mercurial-scm.org/D6083

py3: use % instead of .format() on a bytestring Differential Revision: https://phab.mercurial-scm.org/D6112

py3: use r'' for group name arguments to MatchObjects in phabricator.py MatchObject group names are strings, not bytes. Using bytes in py3 leads to an IndexError. Differential Revision: https://phab.mercurial-scm.org/D6111

py3: use %d instead of %s when formatting an int into a byte string Differential Revision: https://phab.mercurial-scm.org/D6110

py3: only pass unicode to json.dumps in writediffproperties Differential Revision: https://phab.mercurial-scm.org/D6109

py3: fix a few "dict keys as str instead of bytes" issues in phabricator.py Differential Revision: https://phab.mercurial-scm.org/D6108

py3: convert URL to str before passing it to request Differential Revision: https://phab.mercurial-scm.org/D6106

py3: convert indexes into bytes when enumerating lists in urlencodenested Otherwise it'll try to insert them them into a %s slot in a b'' later and fail. Differential Revision: https://phab.mercurial-scm.org/D6105

py3: don't try and format a bare dict into a byte string in callconduit Differential Revision: https://phab.mercurial-scm.org/D6104

py3: use fsencode for vcr recording paths and strings for custom_patches args This fixes phabricator.py's vcrcommand under py3 Differential Revision: https://phab.mercurial-scm.org/D6102

phabricator: convert conduit response JSON unicode to bytes inside callconduit Previously the byte conversion was happening piecemeal in callers, and in the case of createdifferentialrevision not at all, leading to UnicodeEncodeErrors when trying to phabsend a commit with a description containing characters not representable in ascii. (issue6040) Remove all the scattered encoding.unitolocal calls and perform it once, inside callconduit, on the entire response dict recursively before returning it, in keeping with the strategy of converting at the earliest opportunity. Convert all keys used on returned object to bytes. Modify the phabsend tests to test this by adding a € to the commit message of alpha. Differential Revision: https://phab.mercurial-scm.org/D6044

transaction: include txnname in the hookargs dictionary There is no reason to not include the txnname alongside the txnid in all case. The python hooks already have them, so aligning the the shell hooks seems it could be useful in the future. (I don't have a strong opinion about this, we can also decide to never align the python and shell hooks and this and I'll drop this patch).

discovery-helper: reflect argument value in the name of the results It is common to create multiple pairs of repositories using different argument values. Recording the argument value in the results names has two main advantages: * It is easy to remember the value used to create a pair, * It get simpler to create multiple pair at the same time from the same source. Previously, running: `./discovery-helper.sh pypy 50 10` would create a `pypy-left` and `pypy-right` repository. Now it will create `pypy-50h-10d-left` and `pypy-50h-10d-right`.

discovery-helper: echo the stripped revsets early Having them printed early make it easy for a user to just grab the generated revset and directly uses them

contrib: move the `discovery-helper.sh` script in`perf-utils` directory The script appeared before the directory. However the directory exists for this kind of script. We move it there.

tests: stabilize test-split.t for Windows It looks like there will be additional problems beyond this trivial fix, but this should make the bot green again.

tests: stabilize test-share.t on Windows PYTHON was not getting mangled for MSYS style paths, and remote was spitting out remote: 'C' is not recognized as an internal or external command, remote: operable program or batch file. (once -q was removed). Additionally, this should fix a failure with py3 because of spaces in the path.

split: add tests which demonstrate the issue5864 Differential Revision: https://phab.mercurial-scm.org/D6057

discovery: clarify why the caching of children is valid Yuya Nishihara pointed out that the code looks wrong without this clarification. (And, unsurprisingly, Yuya is right)

tests: clarify test setup test in test-uncommit.t I assume the "hg uncommit b" is there to prove that the working copy is dirty before we try "hg uncommit --allow-dirty-working-copy b". It seems clearer to put that check just before we run the actual test. Differential Revision: https://phab.mercurial-scm.org/D6078

tests: fix a stale reference to experimental.uncommitondirtywdir These tests no longer test the config option, they test the command line flag. Differential Revision: https://phab.mercurial-scm.org/D6077

discovery: explicitly use `undecided` for the children mapping Recent performance achievements makes the assumption that the `undecided` set is used for sampling. That assumption is always true in practice. We stop pretending that taking anything else would make sense and we directly use the `undecided` set from the object. This provides a more honest API.

discovery: cache the children mapping used during each discovery During discovery, the `undecided` set keep shrinking. Therefore, the map computed for an iteration N will be valid for iteration N+1. Instead of computing the same data over and over we cache it the first time. Our private pathological case speed up from about 7.5 seconds to about 6.3 seconds. (starting from over 70s at the start of the full series)

discovery: move children computation in its own method This clarifies the main logic and starts to pave the way to some caching.

discovery: simplify the building of the children mapping Since we only care about the revisions inside the set we are sampling, we can use simpler code (and probably sightly faster).

discovery: simply walk the undecided revs when building the children mapping The sampling only care about revisions in the undecided set, so building children relationship within this set is sufficient. The set of undecided changesets can be much smaller than the full span from its smallest item to the tip of the repository. This restriction can significantly speed up operations in some cases. For example, on our private pathological case, this speeds things up from about 53 seconds to about 7.5 seconds.

discovery: use a lower level but faster way to retrieve parents We already know that no revision in the undecided set are filtered, so we can skip multiple checks and directly access lower level data. In a private pathological case, this improves the timing from about 70 seconds to about 50 seconds. There are other actions to be taken to improve that case, however this gives an idea of the general overhead.

discovery: avoid computing identical sets of heads twice The very same set of heads is computed in the previous statement, it seems more efficient to just copy that result.

discovery: moved sampling functions inside discovery object In this patch, we transform the sampling functions into methods of the `partialdiscovery` class in the Python case. This will provide multiple benefit. For example we can keep some cache from one sampling to another. In addition this will help the Oxidation work as all graph traversal related logic will be contained in a single object.

discovery: rename `srvheads` to `knownsrvheads` The `srvheads` variable only contains the known set of remove heads. Renaming the variable make it clearer.

verify: small refactoring and documentation in `_verifymanifest` Small changes to make this area of code clearer.

verify: document the `_verifymanifest` method

verify: document `_verifychangelog` We document the method input, output and checks.

verify: rename the `checklog` to `_checkrevlog` The method is for internal use only. In addition we make the method name explicitly contains `revlog` to make it clearer it is checking higher level revlog properties.

verify: document the `checklog` method Let us add details about what the function is expected to do.

revlog: add some documentation to the `checksize` method I had to look at it, so I figured I would leave some documentation for the next person seeking information.

verify: make `checkentry` a private method This method is for internal use only.

verify: document the `checkentry` method This method checks various core propertes of a revision. We document inputs, outputs and the checks performed.

verify: add some inline documentation to the top level `verify` method The goal is to clarify each section goal.

verify: explicitly return 0 if no error are encountered Relying on the fact None is treated as 0 by other logics seems smarter than we should be.

verify: minimal documentation for `verifier.verify` I expect the docstring to grow over time, so lets start small.

verify: make the `exc` method private The method is for internal use only.

verify: document the `exc` method Simple method, get simple documentation.

verify: make `err` a private method The method is for internal use only.

verify: document the `err` method Simple method get simple documentation.

verify: make the `warn` method private The method is for internal use only. Let us make that fact clearer.

verify: document the `warn` method This is the first bit of an effort to document and augment the verify code.

uncommit: add flag --allow-dirty-working-copy This adds a flag `--allow-dirty-working-copy` as an alias for the experimental config option `experimental.uncommitondirtydir`. Differential Revision: https://phab.mercurial-scm.org/D6069

uncommit: don't allow dirty working copy with PATH (issue5977) On a dirty PATH, uncommit was working without even setting the config `experimental.uncommitondirtydir` to `True`. Ideally, it should abort as it does for a dirty dir. This patch makes uncommit to require the config option `experimental.uncommitondirtydir` on a dirty PATH. Original patch to evolve extension authored by Dan Villiom Podlaski Christiansen: https://bitbucket.org/octobus/evolve-devel/pull-requests/8/bug-5977-uncommit-dirtiness/diff Differential Revision: https://phab.mercurial-scm.org/D5940

cleanup: remove two bogus test names from python3 list I suspect one of these was a typo from the start, the other appears to have become a .t test at some point. Differential Revision: https://phab.mercurial-scm.org/D6076

revlog: preserve `_lazydelta` attribute in `revlog.clone` The attribute was introduce in 688fc33e105d, Yuya Nishihara pointed out that this preservation was missing. This changeset fixes the preservation and make sure we set the attribute according the modes.

localrepo: explicit `_lock` arguments in `lock` The argument for `_lock` are non-trivial, having them passed explicitly makes thing clearer in my opinion. This is a Gratuitous change, I expect it will save me (and others) time in the future.

inno: remove w9xpopen.exe w9xpopen.exe is a utility program shipped with Python <3.4 (https://bugs.python.org/issue14470 tracked its removal). The program was used by subprocess to wrap invoked processes on Windows 95 and 98 or when command.com was used in order to work around a redirect bug. The workaround is only used on ancient Windows versions - versions that we shouldn't see in 2019. While Python 2.7's subprocess module still references w9xpopen.exe, not shipping it shouldn't matter unless we're running an ancient version of Windows. Python will raise an exception if w9xpopen.exe can't be found. It's highly unlikely anyone is using current Mercurial releases on these ancient Windows versions. So remove w9xpopen.exe from the Inno installer. .. bc:: The 32-bit Windows Inno installers no longer distribute w9xpopen.exe. This should only impact people running Mercurial on Windows 95, 98, or ME. Differential Revision: https://phab.mercurial-scm.org/D6068

inno: stop shipping pywin32 Ancient versions of Mercurial relied on pywin32 and I suspect that's why we have this dependency. We also ship the "keyring" package, which has a dependency on "pywin32-ctypes" (providing the "win32ctypes" package). This is a stripped down version of pywin32 that doesn't have as many dependencies. Since we don't have a dependency on pywin32 and since pywin32 is a bit annoying to package, let's get rid of it. With this change, py2exe no longers picks up DLL dependencies on various UCRT DLLs (because we no longer have a .pyd file beloning to pywin32 which was pulling them in). So, we were able to remove code in support of the UCRT DLLs. .. bc:: The Windows Inno installers no longer ship the pywin32 package. This package was being bundled for historical reasons. Mercurial stopped using pywin32 several years ago and the disappearance of this package should not have any meaningful impact. Differential Revision: https://phab.mercurial-scm.org/D6067

inno: script to automate building Inno installer The official Inno installer build process is poorly documented. And attempting to reproduce behavior of the installer uploaded to www.mercurial-scm.org has revealed a number of unexpected behaviors. This commit attempts to improve the state of reproducibility of the Inno installer by introducing a Python script to largely automate the building of the installer. The new script (which must be run from an environment with the Visual C++ environment configured) takes care of producing an Inno installer. When run from a fresh Mercurial source checkout with all the proper system dependencies (the VC++ toolchain, Windows 10 SDK, and Inno tools) installed, it "just works." The script takes care of downloading all the Python dependencies in a secure manner and manages the build environment for you. You don't need any additional config files: just launch the script, pointing it at an existing Python and ISCC binary and it takes care of the rest. The produced installer creates a Mercurial installation with a handful of differences from the existing 4.9 installers (produced by someone else): * add_path.exe is missing (this was removed a few changesets ago) * The set of api-ms-win-core-* DLLs is different (I suspect this is due to me using a different UCRT / Windows version). * kernelbase.dll and msasn1.dll are missing. * There are a different set of .pyc files for dulwich, keyring, and pygments due to us using the latest versions of each. * We include Tcl/Tk DLLs and .pyc files (I'm not sure why these are missing from the existing installers). * We include the urllib3 and win32ctypes packages (which are dependencies of dulwich and pywin32, respectively). I'm not sure why these aren't present in the existing installers. * We include a different set of files for the distutils package. I'm not sure why. But it should be harmless. * We include the docutils package (it is getting picked up as a dependency somehow). I think this is fine. * We include a copy of argparse.pyc. I'm not sure why this was missing from existing installers. * We don't have a copy of sqlite3/dump.pyc. I'm not sure why. The SQLite C extension code only imports this module when conn.iterdump() is called. It should be safe to omit. * We include files in the email.test and test packages. The set of files is small and their presence should be harmless. The new script and support code is written in Python 3 because it is brand new and independent code and I don't believe new Python projects should be using Python 2 in 2019 if they have a choice about it. The readme.txt file has been renamed to readme.rst and overhauled to reflect the existence of build.py. Differential Revision: https://phab.mercurial-scm.org/D6066

setup: exclude some internal UCRT files When attempting to build the Inno installer locally, I was getting several file not found errors when py2exe was crawling DLL dependencies. The missing DLLs appear to be "internal" DLLs used by the Universal C Runtime (UCRT). In many cases, the missing DLLs don't appear to exist on my system at all! Some of the DLLs have version numbers that appear to be N+1 of what the existing version number is. Maybe the "public" UCRT DLLs are probing for version N+1 at load time and py2exe is picking these up? Who knows. This commit adds the non-public UCRT DLLs as found by py2exe on my system to the excluded DLLs set. After this change, I'm able to produce an Inno installer with an appropriate set of DLLs. Differential Revision: https://phab.mercurial-scm.org/D6065