convert: don't include file in "files" list if it's added in p2 If the file is from p2, we should clearly compare the flags to what they were in p2. Also note that manifest.flags('non-existent') unfortunately returns '' instead of erroring out. Differential Revision: https://phab.mercurial-scm.org/D6409

convert: demonstrate broken {files} list in merge commits with file flags When there is a merge in which the flags for a file from p2 is non-empty, `hg convert` will incorrectly include that in the changeset's files list. Differential Revision: https://phab.mercurial-scm.org/D6408

templater: drop support for old style keywords (API) These changes originated from several commits over a period of time, so I'm slightly unsure if this is correct. But the tests pass.

commands: drop support for legacy ^cmd registration (API)

extensions: drop support for extsetup() without `ui` argument (API)

relnotes: mention removed support for mixed log graph lines This adds release notes for 264a2cbb25d0 (graphmod: remove support for graph lines mixing parent/grandparent styles (BC), 2018-10-16). Differential Revision: https://phab.mercurial-scm.org/D6407

tests: fix test-clonebundles on recent openbsd I guess openbsd feels like it needs to stringify this errno in lowercase and omit the "host" part of "hostname. Okay. Reported in a big test diff talking about libressl, see 6122. I'm not flagging this because most of that issue is about a libressl string change, so this doesn't really make a big difference there. Differential Revision: https://phab.mercurial-scm.org/D6399

rust-python3: compatibility fix for integer conversion On python3, `to_py_object()` on the usize gives us a PyLong, whereas it is the generic `PyObject` already on python2, which fits the `py.None()` default value. Upcasting to `PyObject` explicitely in all cases solves the issue. Differential Revision: https://phab.mercurial-scm.org/D6396

rust: sort dependencies entries in Cargo.toml I should probably write a test to enforce this... Differential Revision: https://phab.mercurial-scm.org/D6398

py3: make contrib/testparseutil.py to work on str(unicodes) contrib/check-code work on unicodes and call functions from testparseutil.py which before this patch used to work on bytes. This path removes that inconsistency and make testparseutil.py work on unicodes. This makes test-check-code.t and test-contrib-check-code.t work on Python 3 again. Differential Revision: https://phab.mercurial-scm.org/D6391

rust-filepatterns: call new Rust implementations from Python This change adds the import to the `rust-cpython` bindings and uses them when appropriate. A wrapper function has been defined in the case of `_regex` to keep this patch simple. Differential Revision: https://phab.mercurial-scm.org/D6273

rust-filepatterns: add `rust-cpython` bindings for `filepatterns` This change adds the `rust-cpython` interface for top-level functions and exceptions in the filepatterns module. Contrary to the Python implementation, this tries to have finer-grained exceptions to allow for better readability and flow control down the line. Differential Revision: https://phab.mercurial-scm.org/D6272

rust-filepatterns: add a Rust implementation of pattern-related utils This change introduces Rust implementations of two functions related to pattern handling, all located in `match.py`: - `_regex` - `readpatternfile` These utils are useful in the long-term effort to improve `hg status`'s performance using Rust. Experimental work done by Valentin Gatien-Baron shows very promising improvements, but is too different from the current Mercurial core code structure to be used "as-is". This is the first - albeit very small - step towards the code revamp needed down the line. Two dependencies were added: `regex` and `lazy_static`. Both of them will be useful for a majority of the Rust code that will be written, are well known and maintained either by the Rust core team, or by very frequent contributors. Differential Revision: https://phab.mercurial-scm.org/D6271

exchange: don't take wlock if bookmarks are stored in .hg/store/ If bookmarks are stored in .hg/store/, there is no need for the wlock(). Differential Revision: https://phab.mercurial-scm.org/D6388

bookmarks: keep bookmarks in .hg/store if new config set Bookmarks storage consists of two parts: (1) the set of bookmarks and their positions, and (2) the current bookmark. The former can get updated by exchange, while the latter cannot. However, they are both stored in directly .hg/ and protected by repo.wlock(). As a result, ugly workarounds were needed. This patch introduces a new config option to store the set of bookmarks and their positions in .hg/store/ but still storing the current bookmark directory in .hg/. The config option only takes effect at repo creation time. It results in a new requirement being set. Differential Revision: https://phab.mercurial-scm.org/D6387

bookmark: also make bookmark cache depends of the changelog Since the changelog is also used during the parsing of bookmark data, it should be listed as a file cache dependency. This fix the race condition we just introduced a test for. This is a simple fix that might lead bookmark data to be invalidated more often than necessary. We could have more complicated code to deal with this race in a more "optimal" way. I feel it would be unsuitable for stable. In addition, the performance impact of this is probably minimal and I don't foresee the more advanced fix to actually be necessary.

localrepo: grab mixedrepostorecache class from 526750cdd02d On default, Martin von Zweigbergk <martinvonz@google.com> introduced a more advance filecache decorator. I need this decorator to fix a bug on stable. So I am grafting the relevant part of 526750cdd02d.

bookmark: add a test for a race condition on push Bookmark pointing to unknown nodes are ignored. Later these ignored bookmarks are dropped when writing the file back on disk. On paper, this behavior should be fine, but with the current implementation, it can lead to unexpected bookmark deletions. In theory, to make sure writer as a consistent view, taking the lock also invalidate bookmark data we already loaded into memory. However this invalidation is incomplete. The data are stored in a `filecache` that preserve them if the bookmark related file are untouched. In practice, the bookmark data in memory also depends of the changelog content, because of the step checking if the bookmarks refers to a node known to the changelog. So if the bookmark data were loaded from an up to date bookmark file but filtered with an outdated changelog file this go undetected. This condition is fairly specific, but can occurs very often in practice. We introduce a test recreating the situation. The test comes in an independant changeset to show it actually reproduce the situation. The fix will come soon after. A large share of the initial investigation of this race condition was made by Valentin Gatien-Baron <valentin.gatienbaron@gmail.com>.

test: properly gate a zstd section This part of the test can't run if zstd is not available. This was caught by --pure test (who don't support zstd).

test: update test for expected test output In 1fac9b931d46 as new test session was introduced. It did not take in account some part that only ran for pure. The test is now fixed.

log: flag topo-sorted set as such This isn't required right now, but revs.istopo() should return True.

copies: fix duplicatecopies() with overlay context The reasoning for this check is in 78d760aa3607 (duplicatecopies: do not mark items not in the dirstate as copies, 2013-03-28). The check was then moved to workingfilectx in 754b5117622f (context: add workingfilectx.markcopied, 2017-10-15) and no corresponding check was added later when overlayworkingfilectx was added. Rather than adding the check there, this patch adds a more generic check on the callers side and removes the check in workingfilectx.markcopied(). Differential Revision: https://phab.mercurial-scm.org/D6380

tests: demonstrate crash when rebasing across copy with --collapse As reported by timeless. Differential Revision: https://phab.mercurial-scm.org/D6379

exthelper: add some semi-useful trace logs It'd be nice to make the trace functions a little better-named in the output, but I'm not sure how much better we can do without overhead. This at least lets you see if a single reposetup function is eating all the time or if it's spread over all of them. I needed this because Google's uber-extension has a long load time and I wasn't sure where the problem was. Differential Revision: https://phab.mercurial-scm.org/D6381

help: add missing blank line, making "revlog-compression" show up Differential Revision: https://phab.mercurial-scm.org/D6386

tests: fix share test to actually share the repo "repo2" is clearly meant to be a share from "repo1" but without sharing bookmarks. However, `hg unshare` was called in the repo, so it had become completely unrelated and thus not testing what it was supposed to test. Differential Revision: https://phab.mercurial-scm.org/D6385

tests: separate out bookmarks tests from test-share.t Differential Revision: https://phab.mercurial-scm.org/D6384

bookmarks: use vfs.tryread() instead of reimplementing it Differential Revision: https://phab.mercurial-scm.org/D6383

bookmarks: use context manager when writing files Differential Revision: https://phab.mercurial-scm.org/D6382

bisect: do not crash with rewritten commits

log: add config for making `hg log -G` always topo-sorted I (and everyone else at Google) have an log alias that adds graph mode and templating. I have another one that builds on the first and also restricts the set of revisions to only show those I'm most likely to care about. This second alias also adds topological sorting. I still sometimes use the first one. When I do, it very often bothers me that it's not topologically sorted (branches are interleaved). This patch adds a config option for always using topological sorting with graph log. The revision set is sorted eagerly, which seems like a bad idea, but it doesn't seem to make a big difference in the hg repo (150ms). I initially tried to instead wrap the user's revset in sort(...,topo), but that seemed much harder. Differential Revision: https://phab.mercurial-scm.org/D6331

log: remove an unnecessary "and opts.get('rev')" condition As Yuya pointed out, the condition is unnecessary since revs.isdescending() would be true if --follow without --rev. Differential Revision: https://phab.mercurial-scm.org/D6372

graphmod: remove support for graph lines mixing parent/grandparent styles (BC) Currently, if the configuration for a graph edge draw style has multiple bytes (at least on python2), it is interpreted as "this is a request to draw the line partially in the style of the parent, partially in the style of the grandparent". This precludes the configuration handling unicode characters (which trigger the `len > 1` check, at least on python2), and I believe was part of the reason that beautifygraph was written the way it was. Talking with the person who implemented this, it appears to have been to achieve feature parity with the rendering of the smartlog extension. I suspect that this isn't actually used outside of that situation, so I think that we can remove it without much issue. This will make it so that multi-character edges are possible, and render any existing configuration that uses this feature with these multiple characters. This is *not* going to adjust the width of everything to make it line up correctly, please see the test that's being modified in this changeset for an example of how the previous configuration now renders. Note also that the previous configuration seems to have been broken, or at least it was behaving in a really non-obvious way - it was avoiding the grandparent character(s) when it should have been displaying them! This is why so many "!" characters changed to "3."; I don't know if this was intentional. Differential Revision: https://phab.mercurial-scm.org/D5112

py3: add 5 new passing tests Differential Revision: https://phab.mercurial-scm.org/D6378

py3: add a r'' to prevent transformer adding b'' # skip-blame because just r'' prefix Differential Revision: https://phab.mercurial-scm.org/D6377

rust-dirstate: call parse/pack bindings from Python A future patch will need to address the issue of Rust module policy, to avoid having ugly duplicate imports and conditionals all over the place. As the rewrite of dirstate in Rust progresses, we will need fewer of those "contact points". Differential Revision: https://phab.mercurial-scm.org/D6350

rust-dirstate: add rust-cpython bindings to the new parse/pack functions This allows for Python code to call `parse/pack_dirstate` transparently. These bindings are heavy given the relatively simple task, as they are bound to implementation details of both the C and Python code. They will be slimmed down in future patches and eventually completely removed once more of the dirstate code has been refactored/rewritten in Rust. Both functions emulate the mutate-on-loop style of the Python and C implementations by looping over changed items in the compatibility layer, instead of at the core functions. Differential Revision: https://phab.mercurial-scm.org/D6349

rust-dirstate: add rust implementation of `parse_dirstate` and `pack_dirstate` Working towards the goal of having a complete Rust implementation of `hg status`, these two utils are a first step of many to be taken to improve performance and code maintainability. Two dependencies have been added: `memchr` and `byteorder`. Both of them have been written by reputable community members and are very mature crates. The Rust code will often need to use their byte-oriented functions. A few unit tests have been added and may help future development and debugging. In a future patch that uses `parse_dirstate` to stat the working tree in parallel - which neither the Python nor the C implementations do - actual performance improvements will be seen for larger repositories. Differential Revision: https://phab.mercurial-scm.org/D6348

changelog: define changelogrevision.p[12]copies for null revision Looks like I missed these in 5382d8f8530b (changelog: parse copy metadata if available in extras, 2017-12-27). `hg debugp[12]copies -r null` fails before this patch. Differential Revision: https://phab.mercurial-scm.org/D6376

copies: write empty entries in changeset when also writing to filelog When writing to both changeset and filelog (during transition), we don't want the reader to waste time by falling back to reading from the filelog when there is no copy metadata. Let's write out empty copy metadata instead (the read path is already prepared for this case). Thanks to Greg for pointing this out. Differential Revision: https://phab.mercurial-scm.org/D6306

rebase: hide help for revisions.Predicates._destautoorphanrebase

unshelve: add space to help

context: default to using branch from dirstate only in workingctx Same reasoning as previous commits: only the workingctx should know about the dirstate. committablectx now seems free of dirstate references. Differential Revision: https://phab.mercurial-scm.org/D6374

context: let caller pass in branch to committablectx.__init__() committablectx.__init__() currently looks up the branch from the dirstate unless it's passed in the extras. memctx.__init__() has a branch argument, but since committablectx.__init__() doesn't accept it, it lets that constructor look up the branch from the dirstate before it overwrites it, which seems awkward. Differential Revision: https://phab.mercurial-scm.org/D6366

context: move contents of committablectx.markcommitted() to workingctx Same reasoning as previous commits: this function updates the dirstate. By not updating the dirstate here, we also fix the close-head test. Differential Revision: https://phab.mercurial-scm.org/D6365

tests: demonstrate that close-head command updates working copy The help text for the command says "...it doesn't change the working directory", so I don't think this is intentional. Differential Revision: https://phab.mercurial-scm.org/D6364

context: move walk() and match() overrides from committablectx to workingctx Same reasoning as previous commit: these functions update the dirstate. Differential Revision: https://phab.mercurial-scm.org/D6363

context: move flags overrides from committablectx to workingctx These read from the dirstate, so they shouldn't be used in other subclasses. Differential Revision: https://phab.mercurial-scm.org/D6362

context: reuse changectx._copies() in all but workingctx This moves the dirstate-specific _copies() implementation from committablectx into workingctx where it should be (I think all dirstate-specific stuff should be moved into workingctx). The part of changectx._copies() that is for producing changeset-wide copy dicts from the filectxs is moved into basectx so it's reused by the other subclasses. The part of changectx._copies() that's about reading copy information from the changeset remains there. This fixes in-memory rebase (and makes `hg convert` able to write copies to changesets). Differential Revision: https://phab.mercurial-scm.org/D6219

overlayworkingctx: don't include added-then-deleted files in memctx If a file (such as a .orig file) is temporarily added to the overlayworkingctx and then deleted, it's still going to be in the _cache dict. In tomemctx(), we created the list of files from _cache.keys(), so the memctx.files() would include the temporary file. That was fine because the list of files was only used in localrepo.commitctx() (I think), where there's an extra filtering of incorrectly removed files (annotated with an inaccurate "update manifest" comment). I'd like to call memctx.files() in another case, but first we need to make it accurate. Differential Revision: https://phab.mercurial-scm.org/D6361

tests: demonstrate loss of changeset copy metadata on rebase Differential Revision: https://phab.mercurial-scm.org/D6360

overlaycontext: allow calling copydata() on clean context We should just report no copy if the context is clean. Differential Revision: https://phab.mercurial-scm.org/D6358

tests: demonstrate another failure with in-memory rebase and copies This is a similar to dd1ab72be983 (test: demonstrate crash with in-memory rebase and copies, 2019-03-14). The new failure started with 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). It happens in the call to mergemod.update() on rebase.py:1268 where we call mergemod.update() to graft a node. Since the mergecopies() rewrite, that calls _related() with the filectx from the overlaywctx instead of a filectx from the changectx where the file was last modified. Either should be fine, so I don't think that's a bug. Differential Revision: https://phab.mercurial-scm.org/D6357

commit: fix a typo ("form p1" -> "from p1") Differential Revision: https://phab.mercurial-scm.org/D6375

automation: initial support for running Linux tests Building on top of our Windows automation support, this commit implements support for performing automated tasks on remote Linux machines. Specifically, we implement support for running tests on ephemeral EC2 instances. This seems to be a worthwhile place to start, as building packages on Linux is more or less a solved problem because we already have facilities for building in Docker containers, which provide "good enough" reproducibility guarantees. The new `run-tests-linux` command works similarly to `run-tests-windows`: it ensures an AMI with hg dependencies is available, provisions a temporary EC2 instance with this AMI, pushes local changes to that instance via SSH, then invokes `run-tests.py`. Using this new command, I am able to run the entire test harness substantially faster then I am on my local machine courtesy of access to massive core EC2 instances: wall: 16:20 ./run-tests.py -l (i7-6700K) wall: 14:00 automation.py run-tests-linux --ec2-instance c5.2xlarge wall: 8:30 automation.py run-tests-linux --ec2-instance m5.4xlarge wall: 8:04 automation.py run-tests-linux --ec2-instance c5.4xlarge wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 3:57 automation.py run-tests-linux --ec2-instance m5.12xlarge wall: 3:05 automation.py run-tests-linux --ec2-instance m5.24xlarge wall: 3:02 automation.py run-tests-linux --ec2-instance c5.18xlarge ~3 minute wall time to run pretty much the entire test harness is not too bad! The AMIs install multiple versions of Python. And the run-tests-linux command specifies which one to use: automation.py run-tests-linux --python system3 automation.py run-tests-linux --python 3.5 automation.py run-tests-linux --python pypy2.7 By default, the system Python 2.7 is used. Using this functionality, I was able to identity some unexpected test failures on PyPy! Included in the feature is support for running with alternate filesystems. You can simply pass --filesystem to the command to specify the type of filesystem to run tests on. When the ephemeral instance is started, a new filesystem will be created and tests will run from it: wall: 4:30 automation.py run-tests-linux --ec2-instance c5.9xlarge wall: 4:20 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem xfs wall: 4:24 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem tmpfs wall: 4:26 automation.py run-tests-linux --ec2-instance c5d.9xlarge --filesystem ext4 We also support multiple Linux distributions: $ automation.py run-tests-linux --distro debian9 total time: 298.1s; setup: 60.7s; tests: 237.5s; setup overhead: 20.4% $ automation.py run-tests-linux --distro ubuntu18.04 total time: 286.1s; setup: 61.3s; tests: 224.7s; setup overhead: 21.4% $ automation.py run-tests-linux --distro ubuntu18.10 total time: 278.5s; setup: 58.2s; tests: 220.3s; setup overhead: 20.9% $ automation.py run-tests-linux --distro ubuntu19.04 total time: 265.8s; setup: 42.5s; tests: 223.3s; setup overhead: 16.0% Debian and Ubuntu are supported because those are what I use and am most familiar with. It should be easy enough to add support for other distros. Unlike the Windows AMIs, Linux EC2 instances bill per second. So the cost to instantiating an ephemeral instance isn't as severe. That being said, there is some overhead, as it takes several dozen seconds for the instance to boot, push local changes, and build Mercurial. During this time, the instance is largely CPU idle and wasting money. Even with this inefficiency, running tests is relatively cheap: $0.15-$0.25 per full test run. A machine running tests as efficiently as these EC2 instances would cost say $6,000, so you can run the test harness a >20,000 times for the cost of an equivalent machine. Running tests in EC2 is almost certainly cheaper than buying a beefy machine for developers to use :) # no-check-commit because foo_bar function names Differential Revision: https://phab.mercurial-scm.org/D6319

automation: move image operations to own functions An upcoming commit will need this functionality with slightly different values and it is enough code to not want to duplicate. Let's refactor into standalone functions so it can be reused. Differential Revision: https://phab.mercurial-scm.org/D6318

automation: add --version argument to build-all-windows-packages This lets us pass a version string through when building all Windows packages, just like we can do with the individual commands which produce installers. Differential Revision: https://phab.mercurial-scm.org/D6317

automation: do a force push to synchronize We don't know what the state of the remote is. Force pushing will be more resilient. Differential Revision: https://phab.mercurial-scm.org/D6316

automation: add check that hg source directory is a repo Synchronizing from e.g. source distributions is not yet supported. Let's add a check so we fail with an error message indicating such. Differential Revision: https://phab.mercurial-scm.org/D6315

automation: shore up rebooting behavior There was a race condition in the old code. Use instance.stop()/instance.start() to eliminate it. As part of debugging this, I also found another race condition related to PowerShell permissions after the reboot. Unfortunately, I'm not sure the best way to work around it. I've added a comment for now. Differential Revision: https://phab.mercurial-scm.org/D6288

automation: wait longer for WinRM connection I got a few timeouts waiting for only 120s for the WinRM connection to become available. Increasing to 180s seems to fix. I guess AWS isn't as consistent as I would like :( Differential Revision: https://phab.mercurial-scm.org/D6287

automation: wait for instance profiles and roles Otherwise there is a race condition between creating the resources and us attempting to use them / them becoming available. The role waiter API was recently introduced, so we had to upgrade the boto3 package to get it. Other packages were also updated to latest versions just because. Even with this change, I still run into issues with the IAM instance profile not being available when we attempt to create an EC2 instance using a just-created profile. I'm not sure what's going on. Possibly a bug on Amazon's end. But the new behavior is "more correct." Differential Revision: https://phab.mercurial-scm.org/D6286

automation: don't create resources when deleting things Otherwise running these commands can result in resources being created. In the case of `purge-ec2-resources`, we will create resources only to delete them immediately afterwards! With this change, `purge-ec2-resources` now no-ops if no resources exist. # no-check-commit because foo_bar function name Differential Revision: https://phab.mercurial-scm.org/D6285

automation: detach policies before deleting role You can't delete an IAM role that has attached policies. With this change, the purge-ec2-resources command now works. Differential Revision: https://phab.mercurial-scm.org/D6284

automation: only iterate over our AMIs We can't delete AMIs that we don't own. Iterating over other AMIs won't work and slows down execution. Differential Revision: https://phab.mercurial-scm.org/D6283

remotefilelog: move most setup from onetimesetup() to uisetup() All the wrappers moved in this patch check if remotefilelog is enabled before they change behavior, so it's safe to always wrap. Differential Revision: https://phab.mercurial-scm.org/D6334

remotefilelog: move most functions in onetimeclientsetup() to top level This is how most extensions seem to do it. It makes sure we don't accidentally depend on the captured ui instance. Differential Revision: https://phab.mercurial-scm.org/D6333

tests: avoid the word "dirty" to mean "not a descendant of merge base" The term "dirty" is no longer used in the code since 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). Differential Revision: https://phab.mercurial-scm.org/D6373

releasenotes: add a file in which to record release notes I've just spent a few very boring hours going through the changelog for the 5.0 release (829 commits). We only had 5 commits that used the syntax that the release notes extension expects. This commit adds a file in which we can record important changes. The file should preferably be edited in the patch that makes the important change, but it can also be edited after (I think this is an important benefit compared to the release notes extension). I'm thinking that we can rename the file from "next" to "5.1" or something when it's time, and then we'd create a new "next" file on the default branch. I've used the syntax that we use on the our wiki in the template, but I don't care much that we use any valid syntax at all. The idea is mostly to record important changes when they happen. I expect that some copy editing will be needed at release time anyway. Differential Revision: https://phab.mercurial-scm.org/D6332

record: avoid modifying the matcher passed as a method parameter No problem observed, but I remember the previous pattern causing problems with largefiles and/or subrepos. This special matcher was added in 419ac63fe29c, so directly modifying the `fail` callback was probably an oversight in 44611ad4fbd9. Differential Revision: https://phab.mercurial-scm.org/D6371

sslutil: add support for SSLKEYLOGFILE to wrapsocket I recently learned of a Firefox/Chrome feature that allows wiresharking otherwise-TLS'd network connections. Gloriously, there's a pypi module that enables this same feature on Python, so let's add support for it to Mercurial in case we need to wireshark some HTTPs connections. Differential Revision: https://phab.mercurial-scm.org/D6343

phabricator: add custom vcr matcher to match request bodies Currently when the phabricator extension's conduit output changes the tests don't notice since the default vcr matcher only matches on 'method' and 'uri', not the body. Add a custom matcher that checks the same params are in the body (ignoring ordering). vcr's in-built body matcher can't be used since it fails under py3 with a "UnicodeEncodeError" on the "€ in commit message" tests. The DREV ids have decreased since the recordings were generated against a different phabricator instance to avoid spamming mercurial-devel. Differential Revision: https://phab.mercurial-scm.org/D6347

merge with stable

absorb: be more specific when erroring out on merge commit When you have a merge commit checked out and run `hg absorb`, it would tell you abort: no mutable changeset to change That makes it sound like the problem is public commits when isn't really. Let's be more specific in this case. There was already a test case that test this, so that now prints the new message. I added a new test case that shows the old message (when a public commit is checked out). Differential Revision: https://phab.mercurial-scm.org/D6354

remotefilelog: log when we're about to fetch files I'm debugging a slow client situation and knowing how many files are in the batch request would be a nice thing. Differential Revision: https://phab.mercurial-scm.org/D6353

revset: populate wdir() by its hash or revision number It belongs to the same category as the null hash/revision, and we do handle these virtual identifiers in id()/rev() predicates. Let's do that more consistently.

sslutil: fsencode path returned by certifi (issue6132) By inspection, this is the only codepath that could be returning a string instead of a bytes on Python 3.

revset: extract private constant of {nullrev, wdirrev} set I'll add a few more users of this constant to get around wdir identifiers.

help: suggest merge() revset instead of -m/--only-merges Suggested by Dr Rainer Woitok.

tests: update annotate tests to work around simplemerge bug test-annotate.t and test-fastannotate.hg were failing with --pure since 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11). It turned out to be because the pure file merge code behaved differently. I'm guessing it's the mdiff.get_matching_blocks() that behaves differently, but I haven't confirmed that. With this content in the base: a a a And this on the local side: a z a And this on the other side: a a a b4 c b6 It produced this conflict: a z a <<<<<<< working copy: b80e3e32f75a - test: c ||||||| base a ======= a b4 c b5 >>>>>>> merge rev: 64afcdf8e29e - test: mergeb I don't care enough about the pure Python code to fix it, so this patch just updates the tests to manually resolve the conflict. Differential Revision: https://phab.mercurial-scm.org/D6351

copies: delete misplaced comment The comment was added in 78d760aa3607 (duplicatecopies: do not mark items not in the dirstate as copies, 2013-03-28). It became misplaced in 3666331164bb (cmdutil: add copy-filtering support to duplicatecopies, 2014-06-07). Then the relevant code was moved far away in 754b5117622f (context: add workingfilectx.markcopied, 2017-10-15). Differential Revision: https://phab.mercurial-scm.org/D6352

phabricator: include branch in the phabread output Depends on D6301 Differential Revision: https://phab.mercurial-scm.org/D6302

phabricator: fallback to reading metadata from diff for phabread Differential Revision: https://phab.mercurial-scm.org/D6301

phabricator: include commit (node) and parent in the local:commits metadata Differential Revision: https://phab.mercurial-scm.org/D6298

copies: remove redundant filtering of ping-pong renames in _chain() We already handle the ping-pong rename case in the filtering step, so there's very little point in doing it in the chaining loop (ping-pong renames are very rare, so I'm not worried about the cost of adding it and then removing it again). Differential Revision: https://phab.mercurial-scm.org/D6344

repair: reword comments that I noticed while working on source formatting I think this is clearer, and one will also keep us from upsetting check-code when other formatting cleanups happen. Differential Revision: https://phab.mercurial-scm.org/D6339

commit: allow --interactive to work again when naming a directory (issue6131)

gendoc: nest command headers under category headers Differential Revision: https://phab.mercurial-scm.org/D6329

minirst: support subsubsubsubsections (header level 5) with marker '''' Differential Revision: https://phab.mercurial-scm.org/D6328

gendoc: guarantee that all commands were processed The new logic renders the commands belonging to each category in turn. Commands with an unregistered category are at risk of getting skipped because their category is not in the list. By comparing the list of all commands to a log of processed commands, we can detect commands with unregistered categories and fail with an error message. Differential Revision: https://phab.mercurial-scm.org/D6327

gendoc: group commands by category in man page and HTML help Make Mercurial's man page and HTML help group commands by category, and present the categories in a helpful order. `hg help` already does this; this patch uses the same metadata. This patch uses the same header level for command categories and for commands. A subsequent patch will push the command headers down one level. Differential Revision: https://phab.mercurial-scm.org/D6326

gendoc: indent loop to make next patch more legible Differential Revision: https://phab.mercurial-scm.org/D6325

contrib: have byteify-strings explode if run in Python 2 Differential Revision: https://phab.mercurial-scm.org/D6341

repair: reword comment about bookmarks logic Again, this will help auto-formatting shortly. Differential Revision: https://phab.mercurial-scm.org/D6340

monotone: fix a bogus _() wrapper that was caught when formatting code There was a spurious space after `debug`, which hid the _() inside ui.debug() from check-code. Sigh. While here, wrap things more concisely. Differential Revision: https://phab.mercurial-scm.org/D6338

commit: add ability to print file status after each successful invocation When commands.commit.post-status is enabled, `hg commit` will effectively run `hg status -mardu` after committing. It can help catch mistakes like not committing all needed files or not adding unknown files that should've been part of the just created commit.

tests: flatten repo structure in test-commit.t Let's move to parent directory before `hg init` repos, since they don't need to be nested. It makes amend/strip messages that include full path to the backup bundle shorter, for instance.

lfs: add a TODO file This is a cleaned up and reorganized list of items I sent out about a year ago. But tracking this in the repo (like the narrow extension) gives more visibility in case anyone wants to help out.

copies: make "limit" argument to _tracefile() mandatory We always pass a limit. I think the fact that it was optional was also the reason we checked ">=limit" before we used it. So now we can remove that condition too. Differential Revision: https://phab.mercurial-scm.org/D6335

localrepo: don't use defaults arguments that will never be overridden The commithook() callback will be called when the lock is released. lock.release() calls the callback without arguments, so it was quite confusing to me that this function declared extra arguments. We can just close on the variables in the outer scope instead. Differential Revision: https://phab.mercurial-scm.org/D6336

tags: avoid double-reversing a list Differential Revision: https://phab.mercurial-scm.org/D6337

updatecaches: also warm hgtagsfnodescache Now that a full update of this cache run in a reasonable amount of time, we can warm everything when during a full update.

hgtagsfnodescache: inherit fnode from parent when possible If a changeset does not update the content of `.hgtags`, it means it will use the same file-node (for `.hgtags`) as its parents. In this case we can directly reuse the parent's file-node. We use this property when updating the `hgtagsfnodescache` taking a faster path if we already have a cached value for the parents of the node we are looking at. Doing so provides a large performance boost when looking at a lot of fnodes, especially on repository with very large manifest: timing for `tagsmod.fnoderevs(ui, repo, repo.changelog.revs())` mercurial: (41907 revisions, 1923 files) before: 6.9 seconds after: 2.7 seconds (-54%) pypy: (96266 revisions, 5198 files) before: 80 seconds after: 20 seconds (-75%) mozilla-central: (463411 revisions, 272080 files) before: 7166.4 seconds after: 47.8 seconds (-99%, x150 speedup) On a copy of mozilla-try with about 35K heads ans 1.7M changesets, this moves the computation from many hours to a couple of minutes, making it more interesting to do a full warm up of this cache before computing tags (from a cold cache). There seems to be other performance low hanging fruits, like avoiding the use of changectx or a more revision centric logic. However, the new code is fast enough for my needs right now.

hgtagsfnodescache: handle nullid lookup The null revision is empty, so it `.hgtags` content is `nullid` in regards with the `hgtagsfnodescache`. Dealing with `nullid` will help with the next changeset. Before this change, feeding `nullid` to `hgtagsfnodescache.getfnode` would return a wrong result (fnode for tip).

help: register the 'gpg' command category and give it a description help.py expects extensions to register their command category in the CATEGORY_ORDER and CATEGORY_NAMES variables. Once gendoc.py orders commands by category, in the next patch, it'll assume this registration (and raise an exception on encountering any unregistered categories). Luckily, gpg is the only bundled extension with an unregistered custom category, so let's fix it. Differential Revision: https://phab.mercurial-scm.org/D6324

histedit: Speed up scrolling in patch view mode Store patchcontents into the mode state, avoiding the expensive call to ui for computing the patchcontents. Before this change in large repos histedit patch view mode can be very irresponsive.

histedit: Show file names in multiple line format

parser: fix crash by parsing "()" in keyword argument position A tree node can be either None or a tuple because x=("group", None) is reduced to x[1].

repoview: introduce a `experimental.extra-filter-revs` config The option define revisions to additionally filter out of all repository "view". The end goal is to provide and easy to way to serve multiple subset of the same repository using multiple "shares". The simplest use case of this feature is to have one view serving the public changesets and one view also serving the draft. This is currently achievable using the new `server.view` option introduced recently by Joerg Sonnenberger. However, more advanced use cases need more advanced definitions. For example some needs a view dedicated to some release branches, or view that hides security fixes to be released. Joerg Sonnenberger and I discussed this topic at the recent mini-sprint and the both of us have seen real life use cases for this. (This series got written during the same mini-sprint). The feature is fully functional, and use similar cache-fallback mechanism to ensure decent performance. However,there remaining room to ensure each share caches and hooks collaborate with each others. This will come at a later time once users start to actually test this feature on real usecase.

copies: filter out copies from non-existent source later in _chain() _changesetforwardcopies() repeatedly calls _chain(). That is very expensive because _chain() does lookups in the manifest. I hope to split up the function in two parts: 1) simple chaining, not considering end points, and 2) filter out files that don't exist in the end points (and ping-pong copies/renames). This patches gets us closer to that by moving the check for non-existent source later in the function. Now there are no more checks for "src" and "dst" in the first loop; all the filtering of invalid copies is done in the second loop. The code also looks much more consistent now. No measureable impact on `hg debugpathcopies 4.0 4.8`. That shouldn't be surprising since the only case we're doing more checks now is in case of chained copies/renames, which are quire rare in practice. Differential Revision: https://phab.mercurial-scm.org/D6277

copies: clarify mutually exclusive cases in _chain() with a s/if/elif/ If the 'b' dict has a rename from 'x' to 'y', it shouldn't be possible for 'x' to be both (a key) in 'a' and in 'src'. That would mean that 'x' is a file in the source commit and also a rename destination in the intermediate commit. But we currently don't allow renaming files onto existing files, so that shouldn't happen. So let's clarify that by using an "elif" instead of an "if". And if we did allow renaming files onto existing files, we should prefer to use the rename destination in the intermediate commit as source anyway. Differential Revision: https://phab.mercurial-scm.org/D6276

copies: delete a redundant cleanup step in _chain() The check is redundant since d5edb5d3a337 (copies: filter out copies when target is not in destination manifest, 2019-02-14). To test that hypothesis, I made this change in the commit that commit, but all tests still passed. I think the case was necessary before then, we just didn't have tests for it. Differential Revision: https://phab.mercurial-scm.org/D6275

copies: document cases in _chain() Differential Revision: https://phab.mercurial-scm.org/D6274

copies: ignore heuristics copytracing when using changeset-centric algos Differential Revision: https://phab.mercurial-scm.org/D6269

copies: move check for experimental.copytrace==<falsy> earlier I'm going to ignore experimental.copytrace when changeset-centric algorithms are required. This little refactoring makes that easier to add. Differential Revision: https://phab.mercurial-scm.org/D6268

copies: replace .items() by .values() where appropriate As pointed out by Pierre-Yves. Differential Revision: https://phab.mercurial-scm.org/D6266

copies: inline _computenonoverlap() in mergecopies() We now call pathcopies() from the base to each of the commits, and that calls _computeforwardmissing(), which does file prefetching (in the remotefilelog override). So the call to _computenonoverlap() is now pointless (the sets of files from _computenonoverlap() are subsets of the sets of files from _computeforwardmissing()). This somehow also fixes a broken remotefilelog test. Differential Revision: https://phab.mercurial-scm.org/D6256

copies: calculate mergecopies() based on pathcopies() When copies are stored in changesets, we need a changeset-centric version of mergecopies() just like we have a changeset-centric version of pathcopies(). I think the natural way of thinking about mergecopies() is in terms of pathcopies() from the base to each of the commits. So if we can rewrite mergecopies() based on two such pathcopies() calls, we'll get the changeset-centric version for free. That's what this patch does. A nice bonus is that it ends up being a lot simpler. mergecopies() has accumulated a lot of technical debt over time. One good example is the code for dealing with grafts (the "partial/incomplete/dirty" stuff). Since pathcopies() already deals with backwards renames and ping-pong renames, we get that for free. I've run tests with hard-coded debug logging for "fullcopy" and while I haven't looked at every difference it produces, all the ones I have looked at seemed reasonable to me. I'm a little surprised that no more tests fail when run with '--extra-config-opt experimental.copies.read-from=compatibility' compared to before this patch. This patch also fixes the broken cases in test-annotate.t and test-fastannotate.t. It also enables the part of test-copies.t that was previously disabled exactly because mergecopies() needed to get a changeset-centric version. One drawback of the rewritten code is that we may now make remotefilelog prefetch more files. We used to prefetch files that were unique to either side of the merge compared to the other. We now prefetch files that are unique to either side of the merge compared to the base. This means that if you added the same file to each side, we would not prefetch it before, but we would now. Such cases are probably quite rare, but one likely scenario where they happen is when moving from a commit to its successor (or the other way around). The user will probably already have the files in the cache in such cases, so it's probably not a big deal. Some timings for calculating mergecopies between two revisions (revisions shown on each line, all using the common ancestor as base): In the hg repo: 4.8 4.9: 0.21s -> 0.21s 4.0 4.8: 0.35s -> 0.63s In and old copy of the mozilla-unified repo: FIREFOX_BETA_60_BASE^ FIREFOX_BETA_60_BASE: 0.82s -> 0.82s FIREFOX_NIGHTLY_59_END FIREFOX_BETA_60_BASE: 2.5s -> 2.6s FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE: 3.9s -> 4.1s FIREFOX_AURORA_50_BASE FIREFOX_BETA_60_BASE: 31s -> 33s So it's measurably slower in most cases. The most significant difference is in the hg repo between revisions 4.0 and 4.8. In that case it seems to come from the fact that pathcopies() uses fctx.isintroducedafter() (in _tracefile), while the old mergecopies() used fctx.linkrev() (in _checkcopies()). That results in a single call to filectx._adjustlinkrev(), which is responsible for the entire difference in time (in my repo). So we pay a performance penalty but we get more correct code (see change in test-mv-cp-st-diff.t). Deleting the "== f.filenode()" in _tracefile() recovers the lost performance in the hg repo. There were are few other optimizations in _checkcopies() that I could not measure any impact from. One was from the "seen" set. Another was from a "continue" when the file was not in the destination manifest (corresponding to "am" in _tracefile). Also note that merge copies are not calculated when updating with a clean working copy, which is probably the most common case. I therefore think the much simpler code is worth the slowdown. Differential Revision: https://phab.mercurial-scm.org/D6255

tests: add test where copy source is deleted and added back This shows another difference between pathcopies() and mergecopies(): mergecopies() considers files that have been deleted and then added back as different files, but pathcopies() does not. Differential Revision: https://phab.mercurial-scm.org/D6330

merge with stable