ancestor: remove extra generator from lazyancestors.__iter__()

localrepo: fix a mixmatched arg name in createrepository() docstring Differential Revision: https://phab.mercurial-scm.org/D4548

error: ensure ProgrammingError message is always a str Since this error is internal-only and a runtime error, let's give it a treatment that makes it behave identically when repr()d on both Python 2 and Python 3. Differential Revision: https://phab.mercurial-scm.org/D4545

py3: whitelist a test caught by the ratchet Differential Revision: https://phab.mercurial-scm.org/D4547

tests: handle Python 3 not quoting non-empty-directory error I assume this happens on Windows too, so I did the same regex on both versions of the output. The whole message printed by these aborts comes from Python, so if we want to exert control over the quoting here it'll be a bit of a pain. Differential Revision: https://phab.mercurial-scm.org/D4546

context: don't count deleted files as candidates for path conflicts in IMM This patch makes sure we don't consider the deleted files in our IMM wctx as potential conflicts while calculating paths conflicts. This fixes the bug demonstrated in previous patch. Differential Revision: https://phab.mercurial-scm.org/D4543

rebase: add tests showing patch conflict detection needs to be smarter in IMM This patch adds test which shows that you can't rebase a cset which removes a dir and adds a file of the same as that of dir as there are False positives path conflicts reported. I fixed the case when there is a file and we adds a dir of same name while removing the file, but missed testing the current case. Next patch will fix this. Differential Revision: https://phab.mercurial-scm.org/D4544

zsh_completion: add new and remove deprecated flags Differential Revision: https://phab.mercurial-scm.org/D4519

zsh_completion: update various arguments, descriptions, metavariables Addition of "=" means the flag must have an argument after it. Differential Revision: https://phab.mercurial-scm.org/D4518

setup: don't support py 3.5.0, 3.5.1, 3.5.2 because of bug in codecs codecs.escape_encode() raises SystemError if an empty bytestring is passed. We do that at some places in our code and because of this bug, things break. Therefore we can't support the mentioned version. The bug was fixed in 3.5.3, 3.6.0 beta 2. We can't support 3.6.0 anyway because of bug in formatting bytestrings. Link to the python bug: https://bugs.python.org/issue25270 Differential Revision: https://phab.mercurial-scm.org/D4475

util: update lrucachedict order during get() get() should have the same semantics as __getitem__ for item retrieval. Differential Revision: https://phab.mercurial-scm.org/D4506

util: lower water mark when removing nodes after cost limit reached See the inline comment for the reasoning here. This is a pretty common strategy for garbage collectors, other cache-like primtives. The performance impact is substantial: $ hg perflrucachedict --size 4 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 100 ! inserts w/ cost limit ! wall 1.659181 comb 1.650000 user 1.650000 sys 0.000000 (best of 7) ! wall 1.722122 comb 1.720000 user 1.720000 sys 0.000000 (best of 6) ! mixed w/ cost limit ! wall 1.139955 comb 1.140000 user 1.140000 sys 0.000000 (best of 9) ! wall 1.182513 comb 1.180000 user 1.180000 sys 0.000000 (best of 9) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000 ! inserts ! wall 0.679546 comb 0.680000 user 0.680000 sys 0.000000 (best of 15) ! sets ! wall 0.825147 comb 0.830000 user 0.830000 sys 0.000000 (best of 13) ! inserts w/ cost limit ! wall 25.105273 comb 25.080000 user 25.080000 sys 0.000000 (best of 3) ! wall 1.724397 comb 1.720000 user 1.720000 sys 0.000000 (best of 6) ! mixed ! wall 0.807096 comb 0.810000 user 0.810000 sys 0.000000 (best of 13) ! mixed w/ cost limit ! wall 12.104470 comb 12.070000 user 12.070000 sys 0.000000 (best of 3) ! wall 1.190563 comb 1.190000 user 1.190000 sys 0.000000 (best of 9) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000 --mixedgetfreq 90 ! inserts ! wall 0.711177 comb 0.710000 user 0.710000 sys 0.000000 (best of 14) ! sets ! wall 0.846992 comb 0.850000 user 0.850000 sys 0.000000 (best of 12) ! inserts w/ cost limit ! wall 25.963028 comb 25.960000 user 25.960000 sys 0.000000 (best of 3) ! wall 2.184311 comb 2.180000 user 2.180000 sys 0.000000 (best of 5) ! mixed ! wall 0.728256 comb 0.730000 user 0.730000 sys 0.000000 (best of 14) ! mixed w/ cost limit ! wall 3.174256 comb 3.170000 user 3.170000 sys 0.000000 (best of 4) ! wall 0.773186 comb 0.770000 user 0.770000 sys 0.000000 (best of 13) $ hg perflrucachedict --size 100000 --gets 1000000 --sets 1000000 --mixed 1000000 --mixedgetfreq 90 --costlimit 5000000 ! gets ! wall 1.191368 comb 1.190000 user 1.190000 sys 0.000000 (best of 9) ! wall 1.195304 comb 1.190000 user 1.190000 sys 0.000000 (best of 9) ! inserts ! wall 0.950995 comb 0.950000 user 0.950000 sys 0.000000 (best of 11) ! inserts w/ cost limit ! wall 1.589732 comb 1.590000 user 1.590000 sys 0.000000 (best of 7) ! sets ! wall 1.094941 comb 1.100000 user 1.090000 sys 0.010000 (best of 9) ! mixed ! wall 0.936420 comb 0.940000 user 0.930000 sys 0.010000 (best of 10) ! mixed w/ cost limit ! wall 0.882780 comb 0.870000 user 0.870000 sys 0.000000 (best of 11) This puts us ~2x slower than caches without cost accounting. And for read-heavy workloads (the prime use cases for caches), performance is nearly identical. In the worst case (pure write workloads with cost accounting enabled), we're looking at ~1.5us per insert on large caches. That seems "fast enough." Differential Revision: https://phab.mercurial-scm.org/D4505

util: optimize cost auditing on insert Calling popoldest() on insert with cost auditing enabled introduces significant overhead. The primary reason for this overhead is that popoldest() needs to walk the linked list to find the first non-empty node. When we call popoldest() within a loop, this can become quadratic. The performance impact is more pronounced on caches with large capacities. This commit effectively inlines the popoldest() call into _enforcecostlimit(). By doing so, we only do the backwards walk to find the first empty node once. However, we still may still perform this work on insert when the cache is near cost capacity. So this is only a partial performance win. $ hg perflrucachedict --size 4 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 100 ! gets w/ cost limit ! wall 0.598737 comb 0.590000 user 0.590000 sys 0.000000 (best of 17) ! inserts w/ cost limit ! wall 1.694282 comb 1.700000 user 1.700000 sys 0.000000 (best of 6) ! wall 1.659181 comb 1.650000 user 1.650000 sys 0.000000 (best of 7) ! mixed w/ cost limit ! wall 1.157655 comb 1.150000 user 1.150000 sys 0.000000 (best of 9) ! wall 1.139955 comb 1.140000 user 1.140000 sys 0.000000 (best of 9) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000 ! gets w/ cost limit ! wall 0.598526 comb 0.600000 user 0.600000 sys 0.000000 (best of 17) ! wall 0.601993 comb 0.600000 user 0.600000 sys 0.000000 (best of 17) ! inserts w/ cost limit ! wall 37.838315 comb 37.840000 user 37.840000 sys 0.000000 (best of 3) ! wall 25.105273 comb 25.080000 user 25.080000 sys 0.000000 (best of 3) ! mixed w/ cost limit ! wall 18.060198 comb 18.060000 user 18.060000 sys 0.000000 (best of 3) ! wall 12.104470 comb 12.070000 user 12.070000 sys 0.000000 (best of 3) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000 --mixedgetfreq 90 ! gets w/ cost limit ! wall 0.600024 comb 0.600000 user 0.600000 sys 0.000000 (best of 17) ! wall 0.614439 comb 0.620000 user 0.620000 sys 0.000000 (best of 17) ! inserts w/ cost limit ! wall 37.154547 comb 37.120000 user 37.120000 sys 0.000000 (best of 3) ! wall 25.963028 comb 25.960000 user 25.960000 sys 0.000000 (best of 3) ! mixed w/ cost limit ! wall 4.381602 comb 4.380000 user 4.370000 sys 0.010000 (best of 3) ! wall 3.174256 comb 3.170000 user 3.170000 sys 0.000000 (best of 4) Differential Revision: https://phab.mercurial-scm.org/D4504

util: teach lrucachedict to enforce a max total cost Now that lrucachedict entries can have a numeric cost associated with them and we can easily pop the oldest item in the cache, it now becomes relatively trivial to implement support for enforcing a high water mark on the total cost of items in the cache. This commit teaches lrucachedict instances to have a max cost associated with them. When items are inserted, we pop old items until enough "cost" frees up to make room for the new item. This feature is close to zero cost when not used (modulo the insertion regressed introduced by the previous commit): $ ./hg perflrucachedict --size 4 --gets 1000000 --sets 1000000 --mixed 1000000 ! gets ! wall 0.607444 comb 0.610000 user 0.610000 sys 0.000000 (best of 17) ! wall 0.601653 comb 0.600000 user 0.600000 sys 0.000000 (best of 17) ! inserts ! wall 0.678261 comb 0.680000 user 0.680000 sys 0.000000 (best of 14) ! wall 0.685042 comb 0.680000 user 0.680000 sys 0.000000 (best of 15) ! sets ! wall 0.808770 comb 0.800000 user 0.800000 sys 0.000000 (best of 13) ! wall 0.834241 comb 0.830000 user 0.830000 sys 0.000000 (best of 12) ! mixed ! wall 0.782441 comb 0.780000 user 0.780000 sys 0.000000 (best of 13) ! wall 0.803804 comb 0.800000 user 0.800000 sys 0.000000 (best of 13) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 ! init ! wall 0.006952 comb 0.010000 user 0.010000 sys 0.000000 (best of 418) ! gets ! wall 0.613350 comb 0.610000 user 0.610000 sys 0.000000 (best of 17) ! wall 0.617415 comb 0.620000 user 0.620000 sys 0.000000 (best of 17) ! inserts ! wall 0.701270 comb 0.700000 user 0.700000 sys 0.000000 (best of 15) ! wall 0.700516 comb 0.700000 user 0.700000 sys 0.000000 (best of 15) ! sets ! wall 0.825720 comb 0.830000 user 0.830000 sys 0.000000 (best of 13) ! wall 0.837946 comb 0.840000 user 0.830000 sys 0.010000 (best of 12) ! mixed ! wall 0.821644 comb 0.820000 user 0.820000 sys 0.000000 (best of 13) ! wall 0.850559 comb 0.850000 user 0.850000 sys 0.000000 (best of 12) I reckon the slight slowdown on insert is due to added if checks. For caches with total cost limiting enabled: $ hg perflrucachedict --size 4 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 100 ! gets w/ cost limit ! wall 0.598737 comb 0.590000 user 0.590000 sys 0.000000 (best of 17) ! inserts w/ cost limit ! wall 1.694282 comb 1.700000 user 1.700000 sys 0.000000 (best of 6) ! mixed w/ cost limit ! wall 1.157655 comb 1.150000 user 1.150000 sys 0.000000 (best of 9) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000 ! gets w/ cost limit ! wall 0.598526 comb 0.600000 user 0.600000 sys 0.000000 (best of 17) ! inserts w/ cost limit ! wall 37.838315 comb 37.840000 user 37.840000 sys 0.000000 (best of 3) ! mixed w/ cost limit ! wall 18.060198 comb 18.060000 user 18.060000 sys 0.000000 (best of 3) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000 --mixedgetfreq 90 ! gets w/ cost limit ! wall 0.600024 comb 0.600000 user 0.600000 sys 0.000000 (best of 17) ! inserts w/ cost limit ! wall 37.154547 comb 37.120000 user 37.120000 sys 0.000000 (best of 3) ! mixed w/ cost limit ! wall 4.381602 comb 4.380000 user 4.370000 sys 0.010000 (best of 3) The functions we're benchmarking are slightly different, which could move numbers by a few milliseconds. But the slowdown on insert is too great to be explained by that. The slowness is due to insert heavy operations needing to call popoldest() repeatedly when the cache is at capacity. The next commit will address this. Differential Revision: https://phab.mercurial-scm.org/D4503

util: allow lrucachedict to track cost of entries Currently, lrucachedict allows tracking of arbitrary items with the only limit being the total number of items in the cache. Caches can be a lot more useful when they are bound by the size of the items in them rather than the number of elements in the cache. In preparation for teaching lrucachedict to enforce a max size of cached items, we teach lrucachedict to optionally associate a numeric cost value with each node. We purposefully let the caller define their own cost for nodes. This does introduce some overhead. Most of it comes from __setitem__, since that function now calls into insert(), thus introducing Python function call overhead. $ hg perflrucachedict --size 4 --gets 1000000 --sets 1000000 --mixed 1000000 ! gets ! wall 0.599552 comb 0.600000 user 0.600000 sys 0.000000 (best of 17) ! wall 0.614643 comb 0.610000 user 0.610000 sys 0.000000 (best of 17) ! inserts ! <not available> ! wall 0.655817 comb 0.650000 user 0.650000 sys 0.000000 (best of 16) ! sets ! wall 0.540448 comb 0.540000 user 0.540000 sys 0.000000 (best of 18) ! wall 0.805644 comb 0.810000 user 0.810000 sys 0.000000 (best of 13) ! mixed ! wall 0.651556 comb 0.660000 user 0.660000 sys 0.000000 (best of 15) ! wall 0.781357 comb 0.780000 user 0.780000 sys 0.000000 (best of 13) $ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 ! gets ! wall 0.621014 comb 0.620000 user 0.620000 sys 0.000000 (best of 16) ! wall 0.615146 comb 0.620000 user 0.620000 sys 0.000000 (best of 17) ! inserts ! <not available> ! wall 0.698115 comb 0.700000 user 0.700000 sys 0.000000 (best of 15) ! sets ! wall 0.560247 comb 0.560000 user 0.560000 sys 0.000000 (best of 18) ! wall 0.832495 comb 0.830000 user 0.830000 sys 0.000000 (best of 12) ! mixed ! wall 0.686172 comb 0.680000 user 0.680000 sys 0.000000 (best of 15) ! wall 0.841359 comb 0.840000 user 0.840000 sys 0.000000 (best of 12) We're still under 1us per insert, which seems like reasonable performance for a cache. If we comment out updating of self.totalcost during insert(), performance of insert() is identical to __setitem__ before. However, I don't want to make total cost evaluation lazy because it has significant performance implications for when we need to evaluate the total cost at mutation time (it requires a cache traversal, which could be expensive for large caches). Differential Revision: https://phab.mercurial-scm.org/D4502

util: add a popoldest() method to lrucachedict This allows consumers to prune the oldest item from the cache. This could be useful for e.g. a consumer that wishes for the size of items tracked by the cache to remain under a high water mark. Differential Revision: https://phab.mercurial-scm.org/D4501

util: ability to change capacity when copying lrucachedict This will allow us to easily replace an lrucachedict with one with a higher or lower capacity as consumers deem necessary. IMO it is easier to just create a new cache instance than to muck with the capacity of an existing cache. Mutating an existing cache's capacity feels more prone to bugs. Differential Revision: https://phab.mercurial-scm.org/D4500

util: make capacity a public attribute on lrucachedict So others can query it. Useful for operations that may want to verify the cache has capacity for N items before it performs an operation that may cause cache eviction. Differential Revision: https://phab.mercurial-scm.org/D4499

util: properly copy lrucachedict instances Previously, copy() only worked if the cache was full. We teach copy() to only copy defined nodes. Differential Revision: https://phab.mercurial-scm.org/D4498

tests: rewrite test-lrucachedict.py to use unittest This makes the code so much easier to test and debug. Along the way, I discovered a bug in copy(), which I kind of added test coverage for. Differential Revision: https://phab.mercurial-scm.org/D4497

wireprotov2peer: stream decoded responses Previously, wire protocol version 2 would buffer all response data. Only once all data was received did we CBOR decode it and resolve the future associated with the command. This was obviously not desirable. In future commits that introduce large response payloads, this caused significant memory bloat and slowed down client operations due to waiting on the server. This commit refactors the response handling code so that response data can be streamed. Command response objects now contain a buffered CBOR decoder. As new data arrives, it is fed into the decoder. Decoded objects are made available to the generator as they are decoded. Because there is a separate thread processing incoming frames and feeding data into the response object, there is the potential for race conditions when mutating response objects. So a lock has been added to guard access to critical state variables. Because the generator emitting decoded objects needs to wait on those objects to become available, we've added an Event for the generator to wait on so it doesn't busy loop. This does mean there is the potential for deadlocks. And I'm pretty sure they can occur in some scenarios. We already have a handful of TODOs around this. But I've added some more. Fixing this will likely require moving the background thread receiving frames into clienthandler. We likely would have done this anyway when implementing the client bits for the SSH transport. Test output changes because the initial CBOR map holding the overall response state is now always handled internally by the response object. Differential Revision: https://phab.mercurial-scm.org/D4474

wireprotoframing: buffer emitted data to reduce frame count An upcoming commit introduces a wire protocol command that can emit hundreds of thousands of small objects. Without a buffering layer, we would emit a single, small frame for every object. Performance profiling revealed this to be a source of significant overhead for both client and server. This commit introduces a very crude buffering layer so that we emit fewer, bigger frames in such a scenario. This code will likely get rewritten in the future to be part of the streams API, as we'll need a similar strategy for compressing data. I don't want to think about it too much at the moment though. server before: user 32.500+0.000 sys 1.160+0.000 after: user 20.230+0.010 sys 0.180+0.000 client before: user 133.400+0.000 sys 93.120+0.000 after: user 68.370+0.000 sys 32.950+0.000 This appears to indicate we have significant overhead in the frame processing code on both client and server. It might be worth profiling that at some point... Differential Revision: https://phab.mercurial-scm.org/D4473

wireprotov2: implement commands as a generator of objects Previously, wire protocol version 2 inherited version 1's model of having separate types to represent the results of different wire protocol commands. As I implemented more powerful commands in future commits, I found I was using a common pattern of returning a special type to hold a generator. This meant the command function required a closure to do most of the work. That made logic flow more difficult to follow. I also noticed that many commands were effectively a sequence of objects to be CBOR encoded. I think it makes sense to define version 2 commands as generators. This way, commands can simply emit the data structures they wish to send to the client. This eliminates the need for a closure in command functions and removes encoding from the bodies of commands. As part of this commit, the handling of response objects has been moved into the serverreactor class. This puts the reactor in the driver's seat with regards to CBOR encoding and error handling. Having error handling in the function that emits frames is particularly important because exceptions in that function can lead to things getting in a bad state: I'm fairly certain that uncaught exceptions in the frame generator were causing deadlocks. I also introduced a dedicated error type for explicit error reporting in command handlers. This will be used in subsequent commits. There's still a bit of work to be done here, especially around formalizing the error handling "protocol." I've added yet another TODO to track this so we don't forget. Test output changed because we're using generators and no longer know we are at the end of the data until we hit the end of the generator. This means we can't emit the end-of-stream flag until we've exhausted the generator. Hence the introduction of 0-sized end-of-stream frames. Differential Revision: https://phab.mercurial-scm.org/D4472

internals: extract frame-based protocol docs to own document wireprotocol.txt is quite long and difficult to digest. The frame-based protocol is effectively a standalone concept (and could even be used outside of Mercurial). So this commit extracts its docs to a standalone file. The first few paragraphs were rewritten as part of the extraction. Sections headers were adjusted accordingly. Existing referalls in wireprotocol.txt were updated to refer to the new doc / concept, which I've started referring to as `hgrpc`. I'm on the fence as to whether to move the HTTP and SSH transport details to the new doc as well. For now, I'm leaving them in wireprotocol.txt. Differential Revision: https://phab.mercurial-scm.org/D4443

narrow: remove hack to write narrowspec to shared .hg directory AFAIK, we no longer need it since the narrowspec file was move to the store directory in 576eef1ab43d, "narrow: move .hg/narrowspec to .hg/store/narrowspec."

narrowspec: remove parseserverpatterns() which isn't used anymore Follows up 10a8472f6662, "narrow: drop support for remote expansion."

hg: write narrow patterns after repo creation Now that hg.clone() knows when a narrow clone is requested, it makes sense to have it update the narrow patterns for the repo soon after the repo is created, before any exchange occurs. Previously, the narrow extension was monkeypatching an exchange function to do this. The old code is redundant and has been removed. Differential Revision: https://phab.mercurial-scm.org/D4541

narrow: don't wrap exchange.pull() during clone The wrapped version was setting up the narrow repo requirement when a narrow clone was requested. Previous commits taught hg.clone() and repo creation to add the narrow requirement when a narrow clone was requested. So this requirement should already be set up for us and this code is no longer necessary. Differential Revision: https://phab.mercurial-scm.org/D4540

exchange: support defining narrow file patterns for pull This commit teaches exchange.pull() about the desire to perform a narrow file pull. We simply pass include and exclude patterns to the function. The values are validated and stored on the pulloperation instance. hg.clone() has been taught to pass these arguments to exchange.pull(). If the arguments are not passed to exchange.pull(), the active narrow patterns from the repository will automatically be used. We /could/ always use the narrow patterns from the repo. However, allowing explicit values to be passed in allows us to perform data fetching that doesn't necessarily align with the repo configuration. This provides more flexibility. Differential Revision: https://phab.mercurial-scm.org/D4539

commands: pass include and exclude options to hg.clone() These arguments are defined by the narrow extension. Let's teach core to recognize them so we can delete some code from the narrow extension and start to exercise the in-core code for performing a narrow clone. We have no way of easily testing it, but this change should result in .hg/requires having the narrow requirement from the time the file is written rather than added as part of pull. We'll confirm this when we delete some monkeypatched functions from the narrow extension in later commits. Test output changed because hg.clone() is now receiving patterns and validation of those values is occurring sooner, before the exchange code runs and prints the message that was deleted. Differential Revision: https://phab.mercurial-scm.org/D4538

localrepo: add requirement when narrow files creation option present The previous commit taught hg.clone() to define a creation option when file include or exclude patterns are passed. This commit teaches the new repo creation code to convert that creation option into a repository requirement. While not yet used by the narrow extension, the eventual side-effect of this change is that newly-created repositories will have the narrow requirement from their creation onset. Currently, the requirement is added to the repo at exchange.pull() time via a wrapped function in the narrow extension. Differential Revision: https://phab.mercurial-scm.org/D4537

hg: recognize include and exclude patterns when cloning This commit teaches clone() to accept arguments defining file patterns to clone. This is the first step in teaching core code about the existence of a narrow clone. Right now, we only perform validation of the arguments and pass additional options into createopts to influence repository creation. Nothing of consequence happens with that creation option yet, however. For now, arbitrary restrictions exist, such as not allowing patterns for shared repos and disabling local copies when patterns are defined. We can potentially lift these restrictions in the future once partial clone/storage support is more flushed out. I figure it is best to reduce the surface area for bugs for the time being. It may seem weird to prefix these arguments with "store." However, clone is effectively pull + update and file patterns could apply to both the store and the working directory. The prefix is there to disambiguate in the future when this function may want to use different sets of patterns for the store and working directory. Differential Revision: https://phab.mercurial-scm.org/D4536

hg: allow extra arguments to be passed to repo creation (API) Currently, repository creation is influenced by consulting the ui instance and turning config options into requirements. This means that in order to influence repository creation, you need to define and set a config option and that the option must translate to a requirement stored in the .hg/requires file. This commit introduces a new mechanism to influence repository creation. hg.repository() and hg.peer() have been taught to receive a new optional argument defining extra options to apply to repository creation. This value is passed along to the various instance() functions and can be used to influence repository creation. This will allow us to pass rich data directly to repository creation without having to go through the config layer. It also allows us to be more explicit about the features requested during repository creation and provides a natural point to detect unhandled options influencing repository creation. The new code detects when unknown creation options are present and aborts in that case. .. api:: options can now be passed to influence repository creation The various instance() functions to spawn new peers or repository instances now receive a ``createopts`` argument that can be a dict defining additional options to influence repository creation. localrepo.newreporequirements() also receives this argument. Differential Revision: https://phab.mercurial-scm.org/D4535

localrepo: move repo creation logic out of localrepository.__init__ (API) It has long bothered me that local repository creation is handled as part of localrepository.__init__. Upcoming changes I want to make around how repositories are initialized and instantiated will make the continued existence of repository creation code in localrepository.__init__ even more awkward. localrepository instances are almost never constructed directly: instead, callers are supposed to go through hg.repository() to obtain a handle on a repository. And hg.repository() calls localrepo.instance() to return a new repo instance. This commit teaches localrepo.instance() to handle the create=True logic. Most of the code for repo construction has been moved to a standalone function. This allows extensions to monkeypatch the function to further customize freshly-created repositories. A few calls to localrepo.localrepository.__init__ that were passing create=True were converted to call localrepo.instance(). .. api:: local repo creation moved out of constructor ``localrepo.localrepository.__init__`` no longer accepts a ``create`` argument to create a new repository. New repository creation is now performed as part of ``localrepo.instance()`` and the bulk of the work is performed by ``localrepo.createrepository()``. Differential Revision: https://phab.mercurial-scm.org/D4534

subrepo: mask out passwords embedded in the messages displaying a URL I noticed the password in maintenance logs for the "no changes since last push" and "pushing to" messages when pushing with an explicit path. But the test case here with :pushurl was also affected. I didn't see that cloning or pulling subrepos on demand had this problem, but it seems safer to just mask that too. There's a bit of a disconnect here, because it looks like clone is slicing off the password (makes sense not to store it in the hgrc in cleartext). But not shearing it off of an explicit path causes the subrepo not to realize that it already pushed the latest stuff. This is the easiest fix, however.

localrepo: pass ui to newreporequirements() (API) newreporequirements() is called as part of creating a new repository. It doesn't make much sense for it to receive a repo instance as part of determining what requirements for new repos should be. .. api:: localrepo.newreporequirements() receives a ui instead of a repo Differential Revision: https://phab.mercurial-scm.org/D4533

narrow: set opts['narrow'] instead of local variable This will allow the command function in core to infer the presence of the option without duplicating logic. Differential Revision: https://phab.mercurial-scm.org/D4532

narrow: drop support for remote expansion (BC) Previous patches to validate narrow patterns accidentically dropped support for the include: syntax that allows patterns to be expanded from a remote. This feature was never implemented in core and is only implemented on Google's custom server. Per @martinvonz's review comment in D4522, it is OK to drop this feature since it isn't used. The concept of this feature does seem useful. I anticipate it making a comeback some day in some shape or form. But for now, let's jettison the dead code. Differential Revision: https://phab.mercurial-scm.org/D4530

fastannotate: use repo.local() This is the proper way to check whether we're dealing with a local repository, since extensions should be coding to an interface and not testing for exact types. Differential Revision: https://phab.mercurial-scm.org/D4542

tests: drop extra "file:" prefix from paths in narrow test It looks like these were added by mistake in f4d4bd8c8911 (narrow: add a --narrowspec flag to clone command, 2018-08-08). Differential Revision: https://phab.mercurial-scm.org/D4531

narrow: validate spec files are well-formed during clone (BC) Previously, specfiles would get read then normalized. We want specfiles to be normalized on read so there is no confusion about what the format of specfiles should be. This commit validates the parsed result of --specfile. If entries aren't prefixed, an error is raised. Previously, validation would occur at exchange time, hence why we dropped a line of test output related to server iteraction. Differential Revision: https://phab.mercurial-scm.org/D4526

narrow: validate patterns on incoming bundle2 part The remote data is untrusted and needs to be validated for pattern conformance. Differential Revision: https://phab.mercurial-scm.org/D4525

narrowspec: validate patterns when loading and saving spec file Patterns should be normalized and validated before being passed into narrowspec.save(). Let's assert that by checking immediately before writing the narrow spec file. And let's assert that patterns loaded from the spec file also conform. Differential Revision: https://phab.mercurial-scm.org/D4524

ancestor: use heapreplace() in place of heappop/heappush() This should be slightly faster. Overall perfancestors result:: cpython nginx mercurial ------------- ---------------- ---------------- ---------------- b6db2e80a9ce^ 0.103461 0.006303 0.035716 8eb2145ff0fb 0.192307 (x1.86) 0.012115 (x1.92) 0.052135 (x1.46) this patch 0.139986 (x1.35) 0.006389 (x1.01) 0.037176 (x1.04)

ancestor: rename local aliases of heapq functions in _lazyancestorsiter() The original names no longer look pretty. Just call them as heap*() instead.

ancestor: optimize _lazyancestorsiter() for contiguous chains If there's no revision between p1 and current, p1 must be the next revision to visit. In this case, we can get around the overhead of heappop/push operations. Note that this is faster than using heapreplace(). 'current - p1 == 1' could be generalized as 'all(r not in seen for r in xrange(p1, current)', but Python is too slow to do such thing.

ancestor: unroll loop of parents in _lazyancestorsiter() This change itself isn't major performance win, but it helps optimizing the visit loop for contiguous chains. See the next patch.

ancestor: return early from _lazyancestorsiter() when reached to stoprev There's no need to empty the heap.

ancestor: remove alias of initrevs from _lazyancestorsiter() It's just redundant and less comprehensible.

narrow: validate patterns returned by expandnarrow Remotes could supply malicious or invalid patterns. We should validate them as soon as possible. Differential Revision: https://phab.mercurial-scm.org/D4523

narrowspec: limit patterns to path: and rootfilesin: (BC) Some matcher patterns are computationally expensive and may even have security issues (e.g. evaluating some file sets). For these reasons, we want to limit the types of matcher patterns that can be used in narrow specs and by command line arguments used for defining narrow specs. This commit teaches ``narrowspec.parsepatterns()`` to validate the pattern types against "safe" patterns. Surprisingly, no existing tests broke. So tests for the feature have been added. We also added a function to validate a patterns data structure. This will be used in future commits. Differential Revision: https://phab.mercurial-scm.org/D4522

narrow: mark wire proto capability names experimental and versioned We already plan to add a "widen" wire protocol command to the "narrow" capability, so let's version the capabilities as "exp-narrow-1" and "exp-ellipses-1". When we add the "widen" command, we will then add a "exp-narrow-2" capability to indicate support for that command. Differential Revision: https://phab.mercurial-scm.org/D4529

narrow: move wire proto capabilities to narrowwirepeer These are not bundle2 capabilities (they just happened to share the name "narrow"), so they seem to belong with the wirepeer overrides. Differential Revision: https://phab.mercurial-scm.org/D4528

narrow: check "narrow" wire protocol capability, not bundle2 capability It seems like the new "narrow" wire protocol capability should be what determines if the server supports the "narrow" and "{,old}{in,ex}cludepats" arguments to the getbundle request. Differential Revision: https://phab.mercurial-scm.org/D4527

sparse: add local files to temporaryfiles if they exist out of sparse We get the f1 from args if it's merge and check that whether that exists in sparse checkout or not. If that does not, we add that for merging. The error comes from very low-level where we try to read data of a working-filectx which does not exists in the working directory. It will be extremely ugly to plug in logic to update sparse copy with new file at such a low level. We already have logic related to updating the checkout with required files in calculateupdates() and let's handle this case there only. calculateupdates() call sparse.filterupdatesactions() and the logic is added into the latter function. To get the exact traceback, this patch can be backed out and test-sparse-merges.t can be run with ui.traceback=True. Differential Revision: https://phab.mercurial-scm.org/D4341

tests: show that merging with sparse is broken when rename is involved This patch adds test to show that merging with sparse is broken when you have a rename on one side and just modification on another side. Differential Revision: https://phab.mercurial-scm.org/D4340

narrowspec: fix a typoed 'supported' Spotted by martinvonz. Differential Revision: https://phab.mercurial-scm.org/D4374

narrow: build the known set of nodes only when ellipses is enabled We don't need to build the known set in non-ellipses case because we don't have a shallow repo. In this patch, this checks whether the server has ellipses enabled or not using the server capability and then build the known set of nodes. Building the known set of nodes can take ~3-4 minutes on repositories with millions of csets so this patch speeds up extending a non-shallow narrow clone on large repositories. In future, we should first check whether local repository is an ellipses repo using a new ellipses repo requirement and then control all the combinations between local repo requirement and server capability. Differential Revision: https://phab.mercurial-scm.org/D4520

narrow: add narrow and ellipses as server capabilities Right now we don't have a way to differentiate between whether a server can serve ellipsis or not. The way we check whether a server is narrow enabled is by checking bundle2 capability which does not scale outside of bundle2 world. The goal is to use have wireprotocol commands just like remotefilelog for widening the narrow clone, atleast in non-ellipses cases. Having a server capability will help there as we can't rely on bundle2 capability there. Also having a server capability is neat than having a bundle2 capability. There are lot of things we can optimize locally on the client side by knowing before hand that whether the server supports ellipses or not. This will also help us in making sure that a client ellipses repo does not communicate with a server repo without ellipses. Differential Revision: https://phab.mercurial-scm.org/D4521

treemanifest: use visitchildrenset when doing a walk For this series, starting at 'introduce lazy loading of subdirs' and ending with this commit, we get the following timing numbers, using roughly the same methodology and setup that we did in an earlier commit. "before" is 6268fed3. diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.585 s +- 0.034 s | 1.574 s +- 0.045 s | 99.3% m-u | | x | 1.591 s +- 0.024 s | 1.601 s +- 0.034 s | 100.6% m-u | x | | 1.579 s +- 0.032 s | 1.603 s +- 0.029 s | 101.5% m-u | x | x | 109.8 ms +- 2.1 ms | 108.8 ms +- 2.2 ms | 99.1% l-d-r | | | 234.6 ms +- 5.1 ms | 240.1 ms +- 7.9 ms | 102.3% l-d-r | | x | 238.3 ms +- 8.1 ms | 232.2 ms +- 6.3 ms | 97.4% l-d-r | x | | 120.0 ms +- 2.7 ms | 119.6 ms +- 2.1 ms | 99.7% l-d-r | x | x | 120.2 ms +- 1.6 ms | 119.1 ms +- 1.6 ms | 99.1% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 356.9 ms +- 5.6 ms | 355.9 ms +- 5.6 ms | 99.7% m-u | | x | 209.1 ms +- 3.3 ms | 205.8 ms +- 2.6 ms | 98.4% m-u | x | | 418.3 ms +- 14.0 ms | 350.8 ms +- 7.7 ms | 83.9% <-- m-u | x | x | 168.3 ms +- 1.5 ms | 168.1 ms +- 2.7 ms | 99.9% l-d-r | | | 100.5 ms +- 2.1 ms | 99.1 ms +- 1.6 ms | 98.6% l-d-r | | x | 5.672 s +- 0.133 s | 4.335 s +- 0.051 s | 76.4% <-- l-d-r | x | | 102.8 ms +- 2.8 ms | 100.4 ms +- 1.7 ms | 97.7% l-d-r | x | x | 1.025 s +- 0.028 s | 875.9 ms +- 12.1 ms | 85.5% <-- rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 6.654 s +- 0.137 s | 6.718 s +- 0.046 s | 101.0% m-u | | x | 6.672 s +- 0.104 s | 6.716 s +- 0.143 s | 100.7% m-u | x | | 6.661 s +- 0.072 s | 6.658 s +- 0.066 s | 100.0% m-u | x | x | 696.0 ms +- 49.0 ms | 696.2 ms +- 54.8 ms | 100.0% l-d-r | | | 789.1 ms +- 12.5 ms | 793.5 ms +- 17.3 ms | 100.6% l-d-r | | x | 7.621 s +- 0.194 s | 7.144 s +- 0.116 s | 93.7% <-- l-d-r | x | | 335.4 ms +- 9.7 ms | 339.6 ms +- 10.8 ms | 101.3% l-d-r | x | x | 6.315 s +- 0.085 s | 5.633 s +- 0.105 s | 89.2% <-- status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 331.2 ms +- 7.0 ms | 335.1 ms +- 6.5 ms | 101.2% m-u | | x | 184.0 ms +- 3.8 ms | 183.1 ms +- 3.4 ms | 99.5% m-u | x | | 330.5 ms +- 7.2 ms | 332.7 ms +- 6.9 ms | 100.7% m-u | x | x | 148.7 ms +- 3.5 ms | 147.5 ms +- 2.2 ms | 99.2% l-d-r | | | 97.3 ms +- 1.3 ms | 98.2 ms +- 2.5 ms | 100.9% l-d-r | | x | 5.196 s +- 0.030 s | 4.281 s +- 0.025 s | 82.4% <-- l-d-r | x | | 101.4 ms +- 3.1 ms | 100.9 ms +- 1.8 ms | 99.5% l-d-r | x | x | 1.015 s +- 0.018 s | 861.2 ms +- 11.2 ms | 84.8% <-- status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 2.356 s +- 0.032 s | 2.369 s +- 0.027 s | 100.6% m-u | | x | 2.371 s +- 0.041 s | 2.385 s +- 0.028 s | 100.6% m-u | x | | 2.366 s +- 0.036 s | 2.379 s +- 0.030 s | 100.5% m-u | x | x | 121.8 ms +- 3.8 ms | 121.1 ms +- 2.7 ms | 99.4% l-d-r | | | 726.2 ms +- 12.9 ms | 723.1 ms +- 12.7 ms | 99.6% l-d-r | | x | 738.7 ms +- 11.2 ms | 741.3 ms +- 11.1 ms | 100.4% l-d-r | x | | 215.9 ms +- 3.6 ms | 214.7 ms +- 5.9 ms | 99.4% l-d-r | x | x | 213.1 ms +- 2.5 ms | 212.8 ms +- 4.5 ms | 99.9% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.922 s +- 0.045 s | 3.937 s +- 0.034 s | 100.4% m-u | | x | 3.673 s +- 0.028 s | 3.660 s +- 0.035 s | 99.6% m-u | x | | 3.959 s +- 0.073 s | 3.904 s +- 0.051 s | 98.6% m-u | x | x | 417.0 ms +- 13.0 ms | 406.2 ms +- 6.5 ms | 97.4% l-d-r | | | 534.3 ms +- 12.2 ms | 537.9 ms +- 8.8 ms | 100.7% l-d-r | | x | 10.671 s +- 0.275 s | 10.111 s +- 0.114 s | 94.8% <-- l-d-r | x | | 309.7 ms +- 3.5 ms | 307.9 ms +- 4.0 ms | 99.4% l-d-r | x | x | 1.837 s +- 0.014 s | 1.887 s +- 0.035 s | 102.7% Differential Revision: https://phab.mercurial-scm.org/D4371

treemanifest: use visitchildrenset when filtering a manifest to a matcher Differential Revision: https://phab.mercurial-scm.org/D4370

tests: stabilize test-no-symlink This goes with 89630d0b3e23.

shelve: use the internal phase when possible If the repository support it, use the internal phase for all changesets created by shelve.

treemanifest: avoid loading everything just to get their nodeid Differential Revision: https://phab.mercurial-scm.org/D4369

treemanifest: avoid unnecessary copies/processing when using alwaysmatcher Differential Revision: https://phab.mercurial-scm.org/D4368

treemanifest: attempt to avoid loading all lazily-loaded subdirs in _isempty Differential Revision: https://phab.mercurial-scm.org/D4367

treemanifest: introduce lazy loading of subdirs An earlier patch series made it so that what to load was up to the calling code, which works fine until manifests are copied - when they're copied, they're loaded completely and thus we lose the entire benefit. By lazy loading everything, we can avoid having to pass in the matcher to ~every manifest function, and handle copies correctly as well. This changeset doesn't go as far as it could with loading only the necessary subsets, that will happen in later changes in this series; at the moment, except in a few situations, we just load everything the moment we want to interact with treemanifest._dirs. This is thus most likely to be a small slowdown if treemanifests is in use regardless of whether narrow is in use, but hopefully easier to verify correctness and review. This is part of a series of speedups, it is not expected to produce any real speed improvements itself, but the numbers show that it doesn't produce a large speed penalty in any common case, and for the cases it does provide a penalty in, it is not a large absolute amount (even if it is a large percentage amount). Timing numbers according to command: hyperfine --prepare <preparation_script> 'hg status' HGRCPATH points to a file with the following contents: [extensions] narrow = strip = rebase = mozilla-unified (called m-u below) was at revision #468856. regular hash: eb39298e432d treemanifests hash: 0553b7f29eaf large-dir-repo (called l-d-r below) was generated with the following script: #!/bin/bash hg init large-dir-repo mkdir -p large-dir-repo/third_party/rust/log touch large-dir-repo/third_party/rust/log/foo.txt for i in $(seq 1 30000); do d=$(mktemp -d large-dir-repo/third_party/XXXXXXXXX) touch $d/file.txt done hg -R large-dir-repo ci -Am 'rev0' --user test --date '0 0' echo hi > large-dir-repo/third_party/rust/log/bar.txt hg -R large-dir-repo ci -Am 'rev1' --user test --date '0 0' echo hi > large-dir-repo/third_party/rust/log/baz.txt hg -R large-dir-repo ci -Am 'rev2' --user test --date '0 0' for the repos that use narrow, the narrowspec was this: [include] rootfilesin:accessible/jsat rootfilesin:accessible/tests/mochitest/jsat rootfilesin:mobile/android/chrome/content rootfilesin:mobile/android/modules/geckoview rootfilesin:third_party/rust/log [exclude] This narrowspec was chosen due to the size of the third_party/rust directory (this directory was *not* modified in revision #468856 in mozilla-unified), plus all the directories that *were* modified in revision #468856 of mozilla-unified. Importantly, when using narrow, these repos had everything checked out (in the case of large-dir-repo, that means all 30,001 directories), *before* adding the narrowspec. This is to simulate the behavior when using a virtual filesystem that shows everything for the user even if they haven't added it to the narrowspec yet. This is not a supported configuration, and `hg update` and `hg rebase` will not really do the "correct" thing if there are mutations outside of the narrowspec (which is not the case in these tests, due to a carefully crafted narrowspec), but non-mutating commands should behave correctly. I'm not claiming anything less than a 5% speed win as improvements due to this change; these are probably eiter measurement artifacts or constant time improvements. The numbers that aren't changing are shown primarily to prove that this doesn't make anything worse in any case I plan on testing during this series. 'before' is hg from commit 6268fed3 'N' indicates narrow in use 'T' indicates treemanifest in use Please note that these commands and the narrowspec are a little different than the ones in a similar table that I made in a3cabe9415e1. Important: it is my understanding that these numbers below are *not super reliable*, the large slowdowns may be artifacts of some odd interaction between GC and python module/code complexity. Another changeset of mine (D4351) had shown large timing differences when ~empty, uncalled functions were added to match.py, though only when using --color=never or redirecting to /dev/null. We seem to be on some cusp of complexity or code size that is causing, at my best guess (according to linux `perf` benchmarks) GC to alter behavior and cause a 200-400ms difference in timings. I haven't had a chance to replicate these results on another machine. diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.580 s +- 0.034 s | 1.576 s +- 0.022 s | 99.7% m-u | | x | 1.568 s +- 0.025 s | 1.584 s +- 0.044 s | 101.0% m-u | x | | 1.569 s +- 0.031 s | 1.554 s +- 0.025 s | 99.0% m-u | x | x | 107.3 ms +- 1.6 ms | 106.3 ms +- 1.5 ms | 99.1% l-d-r | | | 232.5 ms +- 5.9 ms | 233.5 ms +- 5.3 ms | 100.4% l-d-r | | x | 236.6 ms +- 6.3 ms | 233.6 ms +- 7.0 ms | 98.7% l-d-r | x | | 118.4 ms +- 2.1 ms | 118.4 ms +- 1.4 ms | 100.0% l-d-r | x | x | 116.8 ms +- 1.5 ms | 118.9 ms +- 1.6 ms | 101.8% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 354.4 ms +- 16.6 ms | 351.0 ms +- 6.9 ms | 99.0% m-u | | x | 207.2 ms +- 3.0 ms | 206.2 ms +- 2.7 ms | 99.5% m-u | x | | 422.0 ms +- 26.0 ms | 351.2 ms +- 6.4 ms | 83.2% <-- m-u | x | x | 166.7 ms +- 2.1 ms | 169.5 ms +- 4.1 ms | 101.7% l-d-r | | | 98.4 ms +- 4.5 ms | 98.5 ms +- 2.1 ms | 100.1% l-d-r | | x | 5.519 s +- 0.060 s | 5.149 s +- 0.042 s | 93.3% <-- l-d-r | x | | 99.1 ms +- 3.2 ms | 102.6 ms +- 9.7 ms | 103.5% <--? l-d-r | x | x | 994.9 ms +- 10.7 ms | 1.026 s +- 0.012 s | 103.1% <--? rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 6.639 s +- 0.168 s | 6.559 s +- 0.097 s | 98.8% m-u | | x | 6.601 s +- 0.143 s | 6.640 s +- 0.207 s | 100.6% m-u | x | | 6.582 s +- 0.098 s | 6.543 s +- 0.098 s | 99.4% m-u | x | x | 678.4 ms +- 57.7 ms | 703.7 ms +- 52.4 ms | 103.7% <--? l-d-r | | | 780.0 ms +- 23.9 ms | 776.0 ms +- 12.6 ms | 99.5% l-d-r | | x | 7.520 s +- 0.255 s | 7.395 s +- 0.044 s | 98.3% l-d-r | x | | 331.9 ms +- 16.5 ms | 327.0 ms +- 3.4 ms | 98.5% l-d-r | x | x | 6.228 s +- 0.113 s | 5.924 s +- 0.044 s | 95.1% status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 330.8 ms +- 7.2 ms | 329.0 ms +- 7.1 ms | 99.5% m-u | | x | 182.9 ms +- 2.7 ms | 183.5 ms +- 2.7 ms | 100.3% m-u | x | | 330.0 ms +- 7.6 ms | 327.1 ms +- 5.4 ms | 99.1% m-u | x | x | 146.2 ms +- 2.4 ms | 147.1 ms +- 1.3 ms | 100.6% l-d-r | | | 95.3 ms +- 1.4 ms | 95.9 ms +- 1.5 ms | 100.6% l-d-r | | x | 5.157 s +- 0.035 s | 5.166 s +- 0.058 s | 100.2% l-d-r | x | | 99.7 ms +- 3.0 ms | 100.2 ms +- 4.4 ms | 100.5% l-d-r | x | x | 993.6 ms +- 13.1 ms | 1.025 s +- 0.015 s | 103.2% <--? status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 2.348 s +- 0.031 s | 2.329 s +- 0.019 s | 99.2% m-u | | x | 2.337 s +- 0.026 s | 2.346 s +- 0.034 s | 100.4% m-u | x | | 2.354 s +- 0.015 s | 2.342 s +- 0.021 s | 99.5% m-u | x | x | 120.6 ms +- 4.3 ms | 119.2 ms +- 2.1 ms | 98.8% l-d-r | | | 731.5 ms +- 11.1 ms | 719.6 ms +- 9.8 ms | 98.4% l-d-r | | x | 729.0 ms +- 15.5 ms | 725.7 ms +- 10.6 ms | 99.5% l-d-r | x | | 211.0 ms +- 3.9 ms | 212.8 ms +- 3.7 ms | 100.9% l-d-r | x | x | 211.5 ms +- 4.2 ms | 211.0 ms +- 3.3 ms | 99.8% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.910 s +- 0.055 s | 3.920 s +- 0.075 s | 100.3% m-u | | x | 3.613 s +- 0.056 s | 3.630 s +- 0.056 s | 100.5% m-u | x | | 3.873 s +- 0.055 s | 3.864 s +- 0.049 s | 99.8% m-u | x | x | 400.4 ms +- 7.4 ms | 403.6 ms +- 5.0 ms | 100.8% l-d-r | | | 531.6 ms +- 10.0 ms | 528.8 ms +- 9.6 ms | 99.5% l-d-r | | x | 10.377 s +- 0.049 s | 9.955 s +- 0.046 s | 95.9% l-d-r | x | | 308.3 ms +- 4.4 ms | 306.8 ms +- 3.7 ms | 99.5% l-d-r | x | x | 1.805 s +- 0.015 s | 1.834 s +- 0.020 s | 101.6% Differential Revision: https://phab.mercurial-scm.org/D4366

contrib: use a monotonic timer in catapipe As spotted by Gregory, we should use a monotonic timer to get better timings. Differential Revision: https://phab.mercurial-scm.org/D4516

contrib: fix catapipe output argument documentation Differential Revision: https://phab.mercurial-scm.org/D4515

tracing: trace command function execution Differential Revision: https://phab.mercurial-scm.org/D4514

extension: add a summary of total loading time per extension Differential Revision: https://phab.mercurial-scm.org/D4513

extensions: trace the total time of running all reposetup callbacks Differential Revision: https://phab.mercurial-scm.org/D4512

extensions: trace the total time of running all extsetup callbacks Differential Revision: https://phab.mercurial-scm.org/D4511

extensions: trace the total time of running all uisetup callbacks Differential Revision: https://phab.mercurial-scm.org/D4510

extensions: add timing for extensions reposetup Differential Revision: https://phab.mercurial-scm.org/D4509

sparse-revlog: set max delta chain length to on thousand The new snapshot system used in the sparse-revlog case gave us some small size benefit so far. However its most important property is to gracefully handle harder limit on delta chainlength. Long delta chain has a very detrimental impact on read (and write) performance in revlog. Being able to shorter them provide a great boost. However, shorting delta used to result significantly lower compression ratio. The intermediate snapshots effectively suppress most of this effect (even all in some case). # Effect on the test repository The repository we use for test is not "realistic" but can still show this in action using an unreasonably low chain limit. Limiting the chain length show a sizeable increase but stay under control: +6% for limit=15; +15% for limit=10. Without the snapshot system the increase is significantly bigger: +45% for limit=15; +80% for limit=10. Even slightly larger than without delta chain limit, the resulting size is still smaller than before we started doing snapshots. Here is a table for comparison. *Since the repository is not branchy, the initial sparse-revlog version does not bring much benefit compare to the non-sparse one): chain length limit | none | limit=15 | limit=10 | without sparse-revlog | 62 818 987 | 112 664 615 | 131 222 574 | without snapshot | 74 365 490 | 108 211 410 | 133 857 764 | with snapshot | 59 230 936 | 63 002 924 | 68 415 329 | # Effect On Real Life Repositories The series provides significant benefits on all kind of repositories. Using `hg debugupgraderepo -o redeltaparent --run`, we recomputed delta chain for various repositories with different settings: - delta chain length: unlimited or 1000 limit - sparse-revlog: enabled or disabled - this series: applied or not applied We can observe multiple types of effect: - On very branchy repositories: * The delta chain limit as low impact on the repo size. * Intermediate snapshot greatly reduces manifest size: - pypy: -80% - netbeans: -95% * The delta chain limit is effective, without a size impact: - netbeans average: 613 -> 282 - private #1 average: 1 068 -> 307 - On more linear repository: * Intermediate snapshot limit the impact of delta chain limit: - mozilla: without the series: +360% with the series: +25% * The delta chain limit provides large improvement: - mozilla's average chain length: unlimited: 15 338 limited: 469 * Despite the chain length limit, the manifest size is reduced: - mercurial: -25% - mozilla: -30% It is clear that the use of chains of intermediate snapshots provide large benefits both in storage size and delta chains quality. We should now switch our effort toward making sure the write performance are acceptable. Then, `sparse-revlog` will be a suitable format for all new repository. # Raw Statistic * no-sparse: general delta repository not using sparse-revlog * no-snapshot: sparse-revlog repository not using this series * snapshot: sparse-revlog repository using this series mercurial Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 8 021 373 | 8 199 366 no-snapshot | 8 103 561 | 8 259 719 snapshot | 6 137 116 | 6 126 433 Manifest Chain length data limit || none || 1000 || value || average | max || average | max || ------------||---------|---------||---------|---------|| no-sparse || 307 | 1456 || 279 | 1000 || no-snapshot || 312 | 1456 || 283 | 1000 || snapshot || 248 | 1208 || 241 | 1000 || Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 51 013 198 | 51 201 574 no-snapshot | 50 930 795 | 51 141 006 snapshot | 48 072 037 | 48 093 572 pypy Manifest Size: limit | none | 1000 ------------|-------------|------------ no-sparse | 193 987 784 | 193 987 784 no-snapshot | 163 171 745 | 163 312 229 snapshot | 34 605 900 | 34 600 750 Manifest Chain length data limit || none || 1000 || value || average | max || average | max || ------------||---------|---------||---------|---------|| no-sparse || 101 | 692 || 101 | 692 || no-snapshot || 151 | 1307 || 148 | 1000 || snapshot || 128 | 1309 || 125 | 1000 || Full Store Size limit | none | 1000 ------------|-------------|------------ no-sparse | 495 931 473 | 495 931 473 no-snapshot | 465 441 017 | 465 581 501 snapshot | 355 467 301 | 355 472 451 Mozilla Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 416 757 148 | 1 869 009 668 no-snapshot | 401 592 370 | 1 843 493 795 snapshot | 224 359 521 | 284 615 500 Manifest Chain length data limit || none || 1000 || value || average | max || average | max || ------------||---------|---------||---------|---------|| no-sparse || 15 333 | 58 980 || 468 | 1 000 || no-snapshot || 15 336 | 58 980 || 469 | 1 000 || snapshot || 15 338 | 58 983 || 469 | 1 000 || Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 2 712 477 887 | 4 164 995 451 no-snapshot | 2 698 887 835 | 4 141 054 304 snapshot | 2 518 130 385 | 2 578 587 596 Netbeans Manifest Size: limit | none | 1000 ------------|----------------|--------------- no-sparse | 4 766 794 101 | 4 870 642 687 no-snapshot | 4 334 806 082 | 4 428 681 309 snapshot | 232 659 666 | 240 330 665 Manifest Chain length data limit || none || 1000 || value || average | max || average | max || ------------||---------|---------||---------|---------|| no-sparse || 597 | 6802 || 254 | 1 000 || no-snapshot || 648 | 6 802 || 305 | 1 000 || snapshot || 613 | 6 804 || 282 | 1 000 || Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 5 807 347 998 | 5 911 196 584 no-snapshot | 5 375 398 602 | 5 469 273 829 snapshot | 1 282 519 928 | 1 290 190 927 Private repo #1 Manifest Size: limit | none | 1000 ------------|-----------------|--------------- no-sparse | 41 389 010 840 | 41 398 162 091 no-snapshot | 9 737 319 435 | 10 223 773 150 snapshot | 744 215 807 | 747 961 822 Manifest Chain length data limit || none || 1000 || value || average | max || average | max || ------------||---------|---------||---------|---------|| no-sparse || 245 | 8 885 || 81 | 1 000 || no-snapshot || 1 225 | 8 885 || 336 | 1 000 || snapshot || 1 068 | 7 909 || 307 | 1 000 || Full Store Size limit | none | 1000 ------------|----------------|--------------- no-sparse | 49 646 065 126 | 49 655 216 377 no-snapshot | 17 924 862 856 | 18 411 316 571 snapshot | 9 009 024 710 | 9 012 770 725 Private repo #2 We currently have less data available for that repository. * Before is a sparse-revlog repository without this series * After is a sparse-revlog repository with this series + 1000 chain limit Manifest Size: Before: 1 531 485 040 bytes After: 1 091 422 451 bytes Manifest Chain: Before: 2 218 avg; 6 575 Max After: 442 avg; 1 000 Max Full Store Size Before: 15 203 955 615 after: 8 207 180 693

snapshot: also consider the snapshot chain of one unrelated revision To maximize the chance of good delta chain reuse, we inject an unrelated delta chain into our search. To do so, we search for the highest revision unrelated to the parents of the current revision and use its snapshot chain too. Adding this extra snapshot into the mix can have a performance impact. We'll deal with performance impact in a later series.

snapshot: extract parent chain computation The final step of this series is to include chain related to "prev" in the search. Before adding that code we do some simple code movement to clarify the next diff.

snapshot: refine candidate snapshot base upward Once we found a suitable snapshot base it is useful to check if it has a "children" snapshot that would provide a better diff. This is useful when base not directly related to stored revision are picked. In those case, we "jumped" to this new chain at an arbitrary point, checking if a higher point is more appropriate will help to provide better results and increase snapshot reuse.

snapshot: try to refine new snapshot base down the chain There are cases where doing a diff against a snapshot's parent will be shorter than against the snapshot itself. Reusing snapshot not directly related to the revision we are trying to store increase this odd. So once we found a possible candidate, we check the snapshots lower in the chain. This will involve extra processing, but this extra processing will only happen when we are doing building a snapshot, a rare situation.

snapshot: make sure we'll never refine delta base from a reused source The point of reusing delta from the source is to avoid doing computation when applying a bundle. Refining such delta would go against that spirit. We do not have refining logic in place yet. This code needed to be moved out of the way before we could start adding such logic.

snapshot: turn _refinedgroups into a coroutine We are now almost ready to start adding refining logic.

snapshot: also use None as a stop value for `_refinegroup` This is yet another small step toward turning `_refinegroups` into a co-routine.

snapshot: add refining logic at the findeltainfo level Once we found a delta, we want to have the candidates logic challenge it, searching for a better candidate. The logic at the lower level is still missing. We'll introduce it later. Adding small changes in individual commits make it simpler to explain the code change. This is another small step toward turning `_refinegroups` into a co-routine.

snapshot: use None as a stop value when looking for a good delta Having clear stop value should help keep clear logic around the co-routine. The alternative of using a StopIteration exception give a messier result. This is one small step toward turning `_refinegroups` into a co-routine.

snapshot: introduce an intermediate `_refinedgroups` generator This method will be used to improve the search for a good snapshot base. To keep things simpler, we introduce the necessary function before doing any delta base logic change. The next handful of commits will focus on refactoring the code to let that new logic land as clearly as possible. # General Idea Right now, the search for a good delta base stop whenever we found a good one. However, when using sparse-revlog, we should probably try a bit harder. We do significant effort to increase delta re-use by jumping on "unrelated" delta chains that provide better results. Moving to another chain for a better result is good, but we have no guarantee we jump at a reasonable point in that new chain. When we consider over the chains related to the parents, we start from the higher-level snapshots. This is a way to consider the snapshot closer to the current revision that has the best chance to produce a small delta. We do benefit from this walk order when jumping to a better "unrelated" stack. To counter-balance this, we'll introduce a way to "refine" the result. After a good delta have been found, we'll keep searching for a better delta, using the current best one as a starting point. # Target Setup The `finddeltainfo` method is responsible for the general search for a good delta. It requests candidates base from `_candidategroups` and decides which one are usable. The `_candidategroups` generator act as a top-level filter, it does not care about how we pick candidates, it just does basic filtering, excluding revisions that have been tested already or that are an obvious misfit. The `_rawgroups` generator is the one with the actual ancestors walking logic, It does not care about what would do a good delta and what was already tested, it just issues the initial candidates. We introduce a new `_refinedgroup` function to bridge the gap between `_candidategroups` and `_rawgroups`. It delegates the initial iteration logic and then performing relevant refining of the valid base once found. (This logic is yet to be added to function) All these logics are fairly independent and easier to understand when standing alone, not mixed with each other. It also makes it easy to test and try different approaches for one of those four layers without affecting the other ones. # Technical details To communicate `finddeltainfo` choice of "current best delta base" to the `_refinegroup` logic, we plan to use python co-routine feature. The `_candidategroups` and `_refinegroup` generators will become co-routine. This will allow `_refinegroup` to detect when a good delta have been found and triggers various refining steps. For now, `_candidategroups` will just pass the value down the stack. After poking at various option, the co-routine appears the best to keep each layers focus on its duty, without the need to spread implementation details across layers.

snapshot: consider unrelated snapshots at a similar level first This new step is inserted before considering using a level-N snapshot as a base for a level-N+1 snapshot. We first check if existing level-N+1 snapshots using the same base would be a suitable base for a level-N+2 snapshot. This increases snapshot reuse and limits the risk of snapshot explosion in very branchy repositories. Using a "deeper" snapshot as the base also results in a smaller snapshot since it builds a level-N+2 intermediate snapshot instead of an N+1 one. This logic is similar for the one we added in a previous commit. In that previous commit is only applied to level-0 "siblings". We can see this effect in the test repository. Snapshots moved from lower levels to higher levels.

snapshot: consider all snapshots in the parents' chains There are no reasons to only consider full snapshot as a possible base for an intermediate snapshot. Now that the basic principles have been set, we can start adding more levels of snapshots. We now consider all snapshots in the parent's chains (full or intermediate). This creates a chain of intermediate snapshots, each smaller than the previous one. # Effect On The Test Repository In the test repository, we can see a decrease in the revlog size and slightly shorter delta chain. However, that approach creates snapshots more frequently, increasing the risk of ending into problematic cases in very branchy repositories (not triggered by the test repository). The next changesets will remove that risk by adding logic that increases deltas reuse.

snapshot: search for unrelated but reusable full-snapshot # New Strategy Step: Reusing Snapshot Outside Of Parents' Chain. If no suitable bases were found in the parent's chains, see if we could reuse a full snapshot not directly related to the current revision. Such search can be expensive, so we only search for snapshots appended to the revlog *after* the bases used by the parents of the current revision (the one we just tested). We assume the parent's bases were created because the previous snapshots were unsuitable, so there are low odds they would be useful now. This search gives a chance to reuse a delta chain unrelated to the current revision. Without this re-use, topological branches would keep reopening new full chains. Creating more and more snapshots as the repository grow. In repositories with many topological branches, the lack of delta reuse can create too many snapshots reducing overall compression to nothing. This results in a very large repository and other usability issues. For now, we still focus on creating level-1 snapshots. However, this principle will play a large part in how we avoid snapshot explosion once we have more snapshot levels. # Effects On The Test Repository In the test repository we created, we can see the beneficial effect of such reuse. We need very few level-0 snapshots and the overall revlog size has decreased. The `hg debugrevlog` call, show a "lvl-2" snapshot. It comes from the existing delta logic using the `prev` revision (revlog's tip) as the base. In this specific case, it turns out the tip was a level-1 snapshot. This is a coincidence that can be ignored. Finding and testing against all these unrelated snapshots can have a performance impact at write time. We currently focus on building good deltas chain we build. Performance concern will be dealt with later in another series.

snapshot: try intermediate snapshot against parents' base # Regarding The Series Started By This Changeset This is the first changesets of a group adjusting delta chain strategy to build a useful chain of intermediate snapshots. The series will introduce a full strategy to produce chains of multiple snapshots on top of which a "usual" delta chain will be built. That strategy will have multiple steps to maximize snapshot reuse, avoiding pathological cases and improving overall compression in very branchy repositories. An important property of sparse-revlog using such snapshot-chain is that they can use very short delta chain without problematic impact on the resulting compression. Shorter delta chains are important to achieve good performance. To make each step clear, we'll introduce them one by one. See the end of this series for full details. # Regarding This Changeset Before this change, if we cannot store the current revision as a delta against a "simple" candidate (p1, p2, prev), we created a new level-0 snapshot (also called full snapshot). As the first step, we introduce a simple strategy: try an intermediate level-1 snapshot against the chain base of the "current revision" parents. The "current revision" is the one we are currently trying to store in the revlog, triggering this search for a good delta base. The first item in the chain is always a level-0 snapshot. # Effect On The Test Repository We can already see the effect on the test-repository. Most of the snapshots have shifted from level 0 to level 1. The overall size has slightly decreased. (However, keep in mind that this repository only emulates real data) # Regarding Statistic The current series focuses on improving the chain built. Improving the performance of this logic will be done as a second step. Sparse-revlog is still experimental and disabled by default. We'll provide more statistic about resulting size and delta chain at the end of this series.

sparse-revlog: add a test checking revlog deltas for a churning file The test repository contains 5000 revisions and is therefore slow to build: five minutes with CHG, over fifteen minutes without. It is too slow to build during the test. Bundling all content produce a sizeable result, 20BM, too large to be committed. Instead, we commit a script to build the expected bundle and the test checks if the bundle is available. Any run of the script will produce the same repository content, using resulting in the same hashes. Using smaller repositories was tried, however, it misses most of the cases we are planning to improve. Having them in a 5000 repository is already nice, we usually see these case in repositories in the order of magnitude of one million revisions. This test will be very useful to check various changes strategy for building delta to store in a sparse-revlog. In this series we will focus our attention on the following metrics: The ones that will impact the final storage performance (size, space): * size of the revlog data file (".hg/store/data/*.d") * chain length info The ones that describe the deltas patterns: * number of snapshot revision (and their level) * size taken by snapshot revision (and their level)

tests: add a `tests/artifacts/` directory That directory is meant to cache large items used by tests that are slow to generate. See 'PURPOSE' file for details and next changesets for a first user.

verify: make output less confusing (issue5924) output before: "500 files, 2035 changesets, 2622 total revisions" output after: "checked 2035 changesets with 2622 changes to 500 files" new one was suggested in the comments inside the issue. Differential Revision: https://phab.mercurial-scm.org/D4476

revlog: clarify the comment attached to delta reuse The previous version was a bit complicated and referred to a deprecated configuration option.

revlog: drop duplicated code This code probably got duplicated by a rebase/evolve conflict. We drop the extra copy, the other copy is right below. This had no real effects since other logic ensure that we never test the same revision twice.

wireprotov2peer: properly format errors formatrichmessage() expects an iterable containing dicts with well-defined keys. We were passing in something else. This caused an exception. Change the code to call formatrichmessage() with the proper argument. And add a TODO to potentially emit the proper data structure from the server in the first place. Differential Revision: https://phab.mercurial-scm.org/D4441