tags-cache: skip the filternode step if we are not going to use it When warming the hgtagsfnodescache, we don't need the actual result, so we can simply skip the part that "filter" fnode we read from the cache. So provide a quite visible speed up to the top level `hg debugupdatecache` function. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.debug.debug-update-cache # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.pre-state = warm before: 19.947581 after: 18.916804 (-5.17%, -1.03)

tags-cache: add a dedicated warm cache function to hgtagsfnodescache Having a dedicated API point will help to optimize that specific usage. Right doing a full phases weam takes a long time, even when the cache is already filled.

outgoing: add a simple fastpath when there is no common This further speed up case like `hg bundle --all` for larger repository. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 before: 316.749699 after: 311.165461 (-1.76%, -5.58) There is further work to be done in this area like not doing any outgoing computation in the stream case for example. however the recent changes already gives use a large win for a small amount of local work. ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog pre-%ln-change: 1.263859 the-%ln-change: 0.700229 (-44.60%, -0.56) prev-changeset: 0.496050 (-60.75%, -0.77) this-changeset: 0.495243 (-60.81%, -0.77) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog pre-%ln-change: 2.975765 the-%ln-change: 1.870798 (-37.13%, -1.10) prev-changeset: 1.461583 (-50.88%, -1.51) this-changeset: 1.469185 (-50.63%, -1.51) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog pre-%ln-change: 4.540080 the-%ln-change: 3.401700 (-25.07%, -1.14) prev-changeset: 2.915810 (-35.78%, -1.62) this-changeset: 2.911643 (-35.87%, -1.63) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog pre-%ln-change: 10.138396 the-%ln-change: 7.750458 (-23.55%, -2.39) prev-changeset: 6.665565 (-34.25%, -3.47) this-changeset: 6.672078 (-34.19%, -3.47) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog pre-%ln-change: 399.484481 the-%ln-change: 346.508952 (-13.26%, -52.98) prev-changeset: 316.749699 (-20.71%, -82.73) this-changeset: 311.165461 (-22.11%, -88.32)

outgoing: rework the handling of the `missingroots` case to be faster The previous implementation was slow, to the point it was taking a significant amount of `hg bundle --type none-streamv2` call. We rework the code to compute the same value much faster, making the operation disappear from the `hg bundle --type none-streamv2` profile. Someone would remark that producing a streamclone does not requires an `outgoing` object. However that is a matter for another day. There is other user of `missingroots` (non stream `hg bundle` call for example), and they will also benefit from this rework. We implement an old TODO in the process, directly computing the missing and common attribute as we have most element at hand already. ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 7.750458 after: 6.665565 (-14.00%, -1.08) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 0.700229 after: 0.496050 (-29.16%, -0.20) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog before: 346.508952 after: 316.749699 (-8.59%, -29.76) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 3.401700 after: 2.915810 (-14.28%, -0.49) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog before: 1.870798 after: 1.461583 (-21.87%, -0.41) note: this whole `missingroots` of outgoing has a limited number of callers and could likely be replace by something simpler (like taking an explicit "missing_revs" set for example). However this is a wider change and we focus on a small impact, quick rework that does not change the API for now.

proxy-vfs: also proxy the `audit` attribute In the previous changeset, we had to do a little dance to access the useful `audit` attribute. We now provide a proper accessors to it. We don't update the code in `perf.py` because it has to remain compatible with older version of Mercurial. This will just be nicer in the future.

perf: clear vfs audit_cache before each run When generating a stream clone, we spend a large amount of time auditing path. Before this changes, the first run was warming the vfs cache for the other runs, leading to a large runtime difference and a "faulty" reported timing for the operation. We now clear this important cache between run to get a more realistic timing. Below are some example of median time change when clearing these cases. The maximum time for a run did not changed significantly. ### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog # benchmark.name = hg.perf.exchange.stream.generate # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.version = latest no-clearing: 17.289905 cache-clearing: 21.587965 (+24.86%, +4.30) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog no-clearing: 32.670748 cache-clearing: 40.467095 (+23.86%, +7.80) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog no-clearing: 37.838858 cache-clearing: 46.072749 (+21.76%, +8.23) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog no-clearing: 32.969395 cache-clearing: 39.646209 (+20.25%, +6.68) In addition, this significantly reduce the timing difference between the performance command, from the perf extensions and a `real `hg bundle` call producing a stream bundle. Some significant differences remain especially on the "mozilla-try" repositories, but they are now smaller. Note that some of that difference will actually not be attributable to the stream generation (like maybe phases or branch map computation). Below are some benchmarks done on a currently draft changeset fixing some unrelated slowness in `hg bundle` (34a78972af409d1ff37c29e60f6ca811ad1a457d) ### data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default hg.perf.exchange.stream.generate: 21.587965 hg.command.bundle: 24.301799 (+12.57%, +2.71) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog hg.perf.exchange.stream.generate: 40.467095 hg.command.bundle: 44.831317 (+10.78%, +4.36) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog hg.perf.exchange.stream.generate: 39.646209 hg.command.bundle: 45.395258 (+14.50%, +5.75) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog hg.perf.exchange.stream.generate: 46.072749 hg.command.bundle: 55.882608 (+21.29%, +9.81) ## data-env-vars.name = mozilla-try-2023-03-22-zlib-general-delta hg.perf.exchange.stream.generate: 334.716708 hg.command.bundle: 377.856767 (+12.89%, +43.14) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog hg.perf.exchange.stream.generate: 302.972301 hg.command.bundle: 326.098755 (+7.63%, +23.13)

perf: start recording total time after warming The warming might be costly and this should not affect the "time profile" of the actual collection.

perf: run the gc before each run The python garbage collector is a large source of performance troubles, we run it right before the timed section to reduce the change for the gc to add noise to the benchmark.

perf: allow profiling of more than one run By default, we still profile the first run only. However profiling more run help to understand side effect from one run to the other. So we add an option to be able to do so.

profiler: flush after writing the profiler output Otherwise, the profiler output might only partially appears until the next flush of the buffer. Since profiling often happens for long operation, the next flush can be a long time away.

stream-clone: disable gc for the entry listing section for the v2 format This is similar to the change we did for the v3 format in 6e4c8366c5ce. The benchmark bellow show this gives us a notable gains, especially on larger repositories. ### benchmark.name = hg.perf.stream-locked-section # benchmark.name = hg.perf.stream-locked-section # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.version = v2 ## data-env-vars.name = pypy-2018-08-01-zstd-sparse-revlog 5e931bf8707c: 0.503820 ~~~~~ 1106d1bf695e: 0.470078 (-6.70%, -0.03) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog 5e931bf8707c: 0.535756 ~~~~~ 1106d1bf695e: 0.490249 (-8.49%, -0.05) ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog 5e931bf8707c: 1.327041 ~~~~~ 1106d1bf695e: 1.174636 (-11.48%, -0.15) ## data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog 5e931bf8707c: 2.439158 ~~~~~ 1106d1bf695e: 2.220515 (-8.96%, -0.22) ## data-env-vars.name = netbeans-2019-11-07-zstd-sparse-revlog 5e931bf8707c: 2.630794 ~~~~~ 1106d1bf695e: 2.261473 (-14.04%, -0.37) ## data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog 5e931bf8707c: 5.769002 ~~~~~ 1106d1bf695e: 5.062000 (-12.26%, -0.71) ## data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog 5e931bf8707c: 13.351750 ~~~~~ 1106d1bf695e: 12.346655 (-7.53%, -1.01) ## data-env-vars.name = mozilla-central-2024-03-22-zstd-sparse-revlog 5e931bf8707c: 10.772939 ~~~~~ 1106d1bf695e: 9.495407 (-11.86%, -1.28) ## data-env-vars.name = mozilla-unified-2024-03-22-zstd-sparse-revlog 5e931bf8707c: 10.864297 ~~~~~ 1106d1bf695e: 9.475597 (-12.78%, -1.39) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog 5e931bf8707c: 17.448335 ~~~~~ 1106d1bf695e: 16.027474 (-8.14%, -1.42)

phases: rework the logic of _pushdiscoveryphase to bound complexity This rework the various graph traversal in _pushdiscoveryphase to keep the complexity in check. This is done though a couple of things: - first, limiting the space we have to explore, for example, if we are not in publishing push, we don't need to consider remote draft roots that are also draft locally, as there is nothing to be moved there. - avoid unbounded descendant computation, and use the faster "rev between" computation. This provide a massive boost to performance when exchanging with repository with a massive amount of draft, like mozilla-try: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = default ## benchmark.variants.revs = any-1-extra-rev before: 20.346590 seconds after: 11.232059 seconds (-38.15%, -7.48 seconds) ## benchmark.variants.revs = any-100-extra-rev before: 24.752051 seconds after: 15.367412 seconds (-37.91%, -9.38 seconds) After this changes, the push operation is still quite too slow. Some of this can be attributed to general phases slowness (reading all the roots from disk for example) and other know slowness (not using persistent-nodemap, branchmap, tags, etc. We are also working on them, but with this series, phase discovery during push no longer showing up in profile and this is a pretty nice and bit low-hanging fruit out of the way. ### (same case as the above) # benchmark.variants.revs = any-1-extra-rev pre-%ln-change: 44.235070 this-changeset: 11.232059 seconds (-74.61%, -33.00 seconds) # benchmark.variants.revs = any-100-extra-rev pre-%ln-change: 49.234697 this-changeset: 15.367412 seconds (-68.79%, -33.87 seconds) Note that with this change, the `hg push` performance is now much closer to the `hg pull` performance, even it still lagging behind a bit. (and the overall performance are still too slow). ### data-env-vars.name = mozilla-try-2023-03-22-ds2-pnm # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.pulled-delta-reuse-policy = default # bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 6.517450 hg.command.push: 11.219888 ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 10.160991 hg.command.push: 14.251107 ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.pulled-delta-reuse-policy = default ## bin-env-vars.hg.flavor = default ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 8.577772 hg.command.push: 11.232059 ## bin-env-vars.hg.flavor = default ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 13.152976 hg.command.push: 15.367412 ## bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-1-extra-rev hg.command.pull: 8.731982 hg.command.push: 11.178751 ## bin-env-vars.hg.flavor = rust ## benchmark.variants.revs = any-100-extra-rev hg.command.pull: 13.184236 hg.command.push: 15.620843

phases: introduce a performant efficient way to access revision in a set This will be useful in the next changesets.

phases: use revision number in `_pushdiscoveryphase` We now reach our target checkpoint in terms of rev-num conversion. The `_pushdiscoveryphase` function is now performing graph computation based on revision number only. Avoiding repeated conversion from node-id to rev-num. See previous changeset updated `new_heads` for rationnal. Again, time saved in the 100 milliseconds order of magnitude for the mozilla-try benchmark I have been using. However, wow that the logic is done using revision number, we can look into having better logic in the next changesets, which will provide a much bigger speedup.

phases: move RemotePhasesSummary to revision number This continue our quest to align more logic on revision number instead of node-ids. The motivation is similar to the change to `new_heads` and `analyze_remote_phases` a few changeset earlier. Again, we take this as an opportunity to rename the class, and the attribute to the new naming scheme. This will highlight the need for code update for any code using it an expecting node-ids. Many of the rev-num → node-id conversion we had to introduce in the previous changesets can now be removed. More will be removed in the future as we continue to align code toward rev-num usage. time saved in the 100 milliseconds order of magnitude for the mozilla-try benchmark I have been using.

phases: stop using `repo.set` in `remotephasessummary` The `repository.set` create changectx on the fly, an expensive operation. Using `repo.revs` and a direct rev-num → node-id translation will be significantly faster. This is especially true as we prepare ourself to no longer do the rev-num → node-id transalation there. The speedup is a bit lost in the overall noisyness of the slow phase discovery algorithm, but it save a small amount of time in my benchmark.

phases: use revision number in analyze_remote_phases Same logic as the previous change to `new_heads`, see rationnal there. This avoids a small number of `nodes -> revs` conversion speeding thing up in the 100 milliseconds order of magnitude for the worses cases. However, the rest of the logic is noisy enough that it hardly matters for now.

phases: use revision number in new_heads All graph operations will be done using revision numbers, so passing nodes only means they will eventually get converted to revision numbers internally. As part of an effort to align the code on using revision number we make the `phases.newheads` function operated on revision number, taking them as input and using them in returns, instead of the node-id it used to consume and produce. This is part of multiple changesets effort to translate more part of the logic, but is done step by step to facilitate the identification of issue that might arise in mercurial core and extensions. To make the change simpler to handle for third party extensions, we also rename the function, using a more modern form. This will help detecting the different between the node-id version and the rev-num version. I also take this as an opportunity to add some comment about possible performance improvement for the future. They don't matter too much now, but they are worse exploring in a while.

phases: convert remote phase root to node while reading them This is currently a bit silly as we will convert them back to node right after, but that is an intermediate step before doing more disruptive changes.

phases: more compact error handling in analyzeremotephases using an intermediate variable result in more readable code, so let us use it.

push: rework the computation of fallbackheads to be correct The previous computation tried to be smart but ended up being wrong. This was caught by phase movement test while reworking the phase discovery logic to be faster. The previous logic was failing to catch case where the pushed set was not based on a common heads (i.e. when the discovery seemed to have "over discovered" content, outside the pushed set) In the following graph, `e` is a common head and we `hg push -r f`. We need to detect `c` as a fallback heads and we previous failed to do so:: e | d f |/ c | b | a The performance impact of the change seems minimal. On the most impacted repository at hand (mozilla-try), the slowdown seems mostly mixed in the overall noise `hg push` but seems to be in the hundred of milliseconds order of magnitude. When using rust, we seems to be a bit faster, probably because we leverage more accelaratd internals. I added a couple of performance related common for further investigation later on.

revset: stop serializing node when using "%ln" Turning hundred of thousand of node from node to hex and back can be slow… what about we stop doing it? In many case were we are using node id we should be using revision id. However this is not a good reason to have a stupidly slow implementation of "%ln". This caught my attention again because the phase discovery during push make an extensive use of "%ln" or huge set. In absolute, that phase discovery probably should use "%ld" and need to improves its algorithmic complexity, but improving "%ln" seems simple and long overdue. This greatly speeds up `hg push` on repository with many drafts. Here are some relevant poulpe benchmarks: ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.command.push # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.explicit-rev = all-out-heads # benchmark.variants.issue6528 = disabled # benchmark.variants.protocol = ssh # benchmark.variants.reuse-external-delta-parent = default ## benchmark.variants.revs = any-1-extra-rev before: 44.235070 after: 20.416329 (-53.85%, -23.82) ## benchmark.variants.revs = any-100-extra-rev before: 49.234697 after: 26.519829 (-46.14%, -22.71) ### benchmark.name = hg.command.bundle # bin-env-vars.hg.flavor = default # bin-env-vars.hg.py-re2-module = default # benchmark.variants.revs = all # benchmark.variants.type = none-streamv2 ## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog before: 10.138396 after: 7.750458 (-23.55%, -2.39) ## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog before: 1.263859 after: 0.700229 (-44.60%, -0.56) ## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog before: 399.484481 after: 346.5089 (-13.26%, -52.98) ## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog before: 4.540080 after: 3.401700 (-25.07%, -1.14) ## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog before: 2.975765 after: 1.870798 (-37.13%, -1.10)

bundlespec: drop unused _bundlespecvariants dictionary Why do we have a `_bundlespecvariants`?

bundlespec: type the _bundlespeccontentopts dictionary If only we had a tool to detect the kind of stupid error we just fixed… ho wait.

bundlespec: fix the "streamv2" and "streamv3-exp" variant In c4aab3661f25, we broken this feature by adding unicode instead of bytes to the dictionary. On the other hand, this feature was never tested, so augment the tests to tests this.

wireprotoserver: ensure that output stream gets flushed on exception Previously flush was happening due to Python finalizer being run on `BufferedWriter`. With upgrade to Python 3.11 this started randomly failing. My guess is that the finalizer on the raw `FileIO` object may be running before the finalizer of `BufferedWriter` has a chance to run. At any rate, since we're not relying on finalizers in the happy case we should also not rely on them in case of exception.

match: strengthen visit_children_set invariant, Recursive means "all files" My previous interpretation of "Recursive" was too relaxed: I thought it instructed the caller to do something like this: > you can stop calling `visit_children_set` because you'll need to descend into > every directory recursively, but you should still check every file if it > matches or not Whereas the real instruction seems to be: > I guarantee that everything in this subtree matches, you can stop > querying the matcher for all files and dirs altogether. The evidence to support this: - the test actually passes with the stronger invariant, revealing no exceptions from this rule - the implementation of `visit_children_set` for `DifferenceMatcher` clearly relies on this requirement, so it must hold for that not to lead to bugs.

match: fix the rust-side bug in visit_children_set for rootfilesin matchers The fix is checked by `test_pattern_matcher_visit_children_set` test, which is what caught the bug in the first place, but also by an end-to-end test that I made for this purpose. Accept the new results of Cargo tests Many of these were already annotated with "FIXME", which is a good sign.

match: fix the "visitdir" method on "rootfilesin" matchers This fixes just the Python side, the fix for the rust side will follow shortly.

match: rename RootFiles to RootFilesIn for more consistency

match: small tweak to PatternMatcher.visit_children_set This makes it a bit more efficient (avoid a computation in case of early return), and in my opinion clearer.

matchers: fix the bug in rust PatternMatcher that made it cut off early This brings the rust output in line with the Python output.

tests: add an end-to-end test to show a bug in `visit_children_set` Concretely, `rootfilesin` is completely broken with respect to `visit_children_set` optimization.

tests: add tests and document expectations from visit_children_set in rust The tests this patch are adding have the form of formal spec in invariants::visit_children_set::holds, and then a series of checks that all examples must satisfy this formal spec. I tried to make the spec consistent with how this function is used and how it was originally conceived. This is in conflict with how it's documented in Rust. Some of the implementations also fail to implement this spec, which leads to bugs, in particular when complicated patterns are used with `hg status`.

tests: add a test that demonstrates a bug in rhg status pattern handling The bug is in [visit_children_set], will be elaborated on in follow-up changes.

bundle-spec: properly parse boolean configuration as boolean Before this changesets "v2;revbranchcache=no" would actually request the addition for a revbranchcache part as the non-empty string `"0"` is `True`

bundle-spec: properly identify changegroup-less bundle It is possible to produce a bundle without changegroup. For example if we want to only send phases or obsolescence information. However that lead to crash for command that identifies bundle content. So we fix that. The test will come in the next changesets, when we fix another bug preventing to generate such bundle by hand.

perf: create the temporary target next to the source in stream-consume See inline comment for rational.

setup: display return code information about failed `hg` call This help to understand what is going wrong when things goes wrong.

bundlespec: rationalize the way we specify stream bundle version Instead of having weird dedicated option for each version (v2, v3, etc) we reuse the same "stream" parameters. This is consistent with the ability to request a stream clone using "none-v2;stream=v2". This changeset introduce no user visible change, this is pure internal cleaning.

bundle: do no check the changegroup version if no changegroup is included We don't need to check the compatibility of something we will not use. In practice this was getting in the was of `streamv2` bundles on a narrow repository as the 'cg.version=02' value was rejected by this checks.

perf-stream-consume: use the source repository config when applying This might contains critical configuration for the benchmark, like enabling of extensions like narrow.

unbundle: move most of the logic on cmdutil to help debug::unbundle reuse This make sure `hg debug::unbundle` focus on the core logic.

postincoming: move to cmdutil This looks like a good place for it to live.

postincoming: avoid computing branchhead if no report will be posted This otherwise defeat some of the branch v3 optimization.

streamclone: stop listing files for entries that have no volatile files This will save a lot of python related time. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) prev-change: 20.88 seconds (-40%) prev-change: 14.22 seconds (-60%) this-change: 11.58 seconds (-67% from baseline; -18% from prev)

stream-clone: disable gc for the initial section for the v3 format The number of small container created turn Python in a gc-frenzy that seriously impact performance. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) prev-change: 20.88 seconds (-40%) this-change: 14.22 seconds (-60% from baseline; -31% from prev)

stream-clone: disable gc for `_entries_walk` duration The number of small container created turn Python in a gc-frenzy that seriously impact performance. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) this-change: 20.88 seconds (-40% from baseline; -15% from previous changes)

nocg: make the utility work are both a decorator and context manager In some case, the context manager version will be simpler.

stream-clone: stop getting the file size of all file in v3 The point of v3 is to do less work in the locked section. It was currently not the case. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.03 seconds this-change: 24.50 seconds (-30%)

stream: in v3, skip the "size" fast path if the entries as some unknown size We are about to prefetch size during the lock less in the v3 case. So we need to avoid trying to use that prefetched size when it is not available. See next changeset for the motivation.

perf-stream-locked-section: advertise the right version key in the help As the v3 format is still experimental, its key is "v3-exp". The help text was not pointing that out. (we also fix `perf::stream-generate` in the process)

perf-stream-locked-section: fix the call to the v3 generator That generator simply return chunks so we should not assign the return to a tuple.

perf-stream-locked-section: actually use v1 generation when requested We were fetching a v1 generator but actually using the v2 function…

branching: merge stable into default

Added signature for changeset 803e61387e86

Added tag 6.7.2 for changeset 803e61387e86

relnotes: add 6.7.2

bundle2: make the "hgtagsfnodes" part advisory This bundle2 part is about helping the client to warms its cache. There is no reason for it to be mandatory. So we mark it advisory.

branching: merge stable into default

branchcache: allow to detect "pure topological case" for branchmap We don't rum this detection every time we run the branchcache, that would be costly. However we now do it when running `hg debugupdatecache`. This will help existing repository to benefit from the fastpath when possible.

branchcache: add a "pure topological head" fast path In a narrow but actually quick common case, all topological heads are all on the same branch and all open. In this case, computing the branch map is very simple. We can quickly detect situation where this situation will not change. So we update the V3 format to be able to express this situation and upgrade the update code to detect we remains in that mode. The branch cache is populated with the actual value when the branch map is accessed, but the update_disk method can do the update without needing to populate it.

branchcache: move the processing of the new data in a dedicated method In a future changeset, this will allow the V3 of the branch cache to use a fast path when possible.

branchcache: gather newly closed head in a dedicated set This is part of a series to more clearly split the update in two step. This will allow us to introduce a fast path during update in a future changeset.

branchcache: gather new obsolete revision in a set This is part of a series to more clearly split the update in two step. This will allow us to introduce a fast path during update in a future changeset.

branchcache: filter obsolete revisions sooner Since we won't do anything with the obsolete revisions, we can just ignore them sooner.

branchcache: skip entries that are topological heads in the on disk file In the majority of cases, topological heads are also branch heads. We have efficient way to get the topological heads and efficient way to retrieve their branch information. So there is little value in putting them in the branch cache file explicitly. On the contrary, writing them explicitly tend to create very large cache file that are inefficient to read and update. So the branch cache v3 format is no longer including them. This changeset focus on the format aspect and have no focus on the performance aspect. We will cover that later.

branchcache: simplify the branch rev cache test We don't need that many content dump and this gets in the way in change in access pattern (e.g. accessing revision in a different order change the order of branches in the "names" file). So we simplify this test in advance.

branchcache: store filtered hash and obsolete hash independently for V3 This will avoid the bug covered in tests/test-branches-obsolete.t when we stop storing all heads explicitly in V3.

branchcache: show the cache file content in test-branches-obsoletes.t This help to track the changes in format between v2 and v3.

branchcache: rework the `filteredhash` logic to be more generic We now have a more flexible `key_hashes` tuple. We duplicated various logic in the V2 and V3 version of the cache as the goal is to start changing the logic for V3 in the next few changesets.

filteredhash: rename the filteredhash function The new name is less ambiguous, as we are about to introduce an alternative function it seems like a good idea to have clearer name to distinct the two.

filteredhash: split the computation of revision sets The branch2's filteredhash combines the filtered revisions and the obsolete ones, this will creates issue for implicit reference to heads we want to introduce for the v3 of the branch cache format. So we isolate this logic for alternative use.

filteredhash: move the hashing in its own function This will help us to reuse this logic in variants of the hashes used for branch cache validation.

branchcache: cleanup the final key generation after update A lot of duplicated work seemed to be done, as we already update the tiprev and tipnode when needed right before. So we simplify that part to focus on the filtered hash. See inline comment for details.

branchcache: add more test for the logic around obsolescence and branch heads While working on branch-cache-v3, we noticed some ambiguity in the filtered+obsolete hash. However this was only caught by a rebase test by chance. It seems important to explicitly tests these cases.

branchcache-v3: use more explicit header line The key-value approach is clearer and gives more rooms to have the format evolve in a clear way. It also provides extension (like topic) simpler way to extend the validation scheme. This is just a small evolution, the V3 format is still a work in progress.

branchcache-v3: introduce a v3 format For now the format is the very same, however we will start changing it in future changesets.

branchcache: use an explicit class for the v2 version This prepare the introduction of an experimental v3 format version. In the process, we move the description of the format in that new class.

branchcache: add some blank line in a test This helps each section to stand out.

phases: update the phase set as we go during retract boundary Apparently iterating over the `changed_revs` dictionary is very expensive. On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets bundle gives give use the following timing. e57d4b868a3e: 4.6 seconds ac1c75188440: 102.5 seconds prev-changeset: 30.0 seconds this-changeset: 4.6 seconds So, the performance regression is gone. Once again: thanks to marvelous Python!

phases: avoid a potentially costly dictionary interation in some case If we retract for the draft phase, there is not non-public item to be retracted and we can skip this part. This part is was apparently super costly thanks to Python. On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets bundle gives give use the following timing. e57d4b868a3e: 4.6 seconds ac1c75188440: 102.5 seconds this-changeset: 30.0 seconds So we recovered about ⅔ of the regression, the next changeset will give us the rest back.

branching: merge stable into default

Added signature for changeset 2e6fde2ed01e

Added tag 6.7.1 for changeset 2e6fde2ed01e

relnotes: add 6.7.1

subrepo: fix normalizing paths with scheme After revision 0afe96e374a7, subrepo paths were normalized using posixpath.normpath and that resulted in ssh paths being wrongly converted from ssh://host/path to ssh:/host/path This fix applies the same logic used in urlutil.url to split the path scheme from the rest and only use posixpath.normpath to the string after scheme://

sshpeer: fix path when handling invalid url exception In 73ed1d13c0bf the code was refactored but the error handling seems to have been missed (or maybe the object shoud have implemented __bytes__)

delta-search: fix crash caused by unbound variable This code path was apparently not tested. This fixes a crash when cloning the Tryton repo.

branching: merge stable into default

Added signature for changeset c9ceb4f60256

Added tag 6.7 for changeset c9ceb4f60256

phases: avoid N² behavior in `advanceboundary` We allowed duplicated entries in the deque, which each entry could potentially insert all its ancestors. So advancing boundary for the full repository would mean each revision would walk all its ancestors, resulting in O(N²) iteration. For repository of any decent size, N² is quickly insane. We introduce a simple set to avoid this and get back to reasonable performance.

relnotes: add 6.7

admin-commands: move the chainsaw extension to the admin commands module Activating an extension is always a little bit of a chore and the long name, options and "chainsaw" bits are deterrent enough. This also allows us to help the discoverability for people looking for repo "administration" tools, with the widest semantic of "administration".

obsutil: sort metadata before comparing in geteffectflag() This is probably less important now that we dropped Python 2. We do still support Python 3.6 though, and the dictionaries aren't ordered there either (that was a big change that came with 3.7). Still, maybe it's a good idea to sort metadata explicitly.

tests: disable revlog compression in test-generaldelta.t (issue6867) The revlog compression makes a lot of numbers unstable. Since checking revlog compression is not the goal of this test, we disable the compression to get stable numbers. This should avoid wasting more time on this kind of changes in the future.

test-general-delta: actually test optimize-delta-parent-choice=no Since the configuration was not explicit, the case stopped testing what it intended to test when the default value changed.

test-chg: stabilize the log checking The "worker process exited" line have been making the CI flaky for a long time. Lets sort this out.

tests: fix test-patchbomb-tls.t instability The flakiness on chg is caused by a client that exit faster than the server output log. So actively wait for the server to issue the expected output (with a small timeout)

test-lock: use synchronisation file instead of sleep This will prevent the test to be flaky on load.

branchcache: use update_disk to refresh 'served' and 'served.hidden' The `update_disk` method is dedicated to this kind of usecase. Now that the writting patterns are more consistent, we can use it to warm these two important cache. I am dropping the first comment about "refreshing all the others" because it is false. If a branchmap already exist for "served", none of the subset will be updated.

branchcache: explictly update disk state only if no transaction exist If a transaction exist the `write_dirty` call will eventually be done and the state will be synched on disk. It is better to no interfer with that.

branchcache: do not use `__getitem__` in updatecache The `update_disk` method uses `updatecache` and the point of `update_disk` is to be able to do alternative processing to the one we do in `__getitem__`. So we calling `__getitem__` in `updatecache` defeat this purpose. Instead we do the equivalent explicitly to preserve the spirit of `update_disk` (that we will actually put to use soon, I promise)

branchcache: explicitly track inheritence "state" We move from a binary "dirty" flag to a three value "state": "clean", "inherited", "dirty". The "inherited" means that the branch cache is not only "clean", but it is a duplicate of its parent filter. If a branch cache is "inherited", we can non only skip writing its value on disk, but it is a good idea to delete any stale value on disk, as those will just waste time (and possibly induce bug) in the future. We only do this in the update related to transaction or explicit cache update (e.g `hg debugupdatecache`). Deleting the file when we simply detected a stall cache during a read only operation seems more dangerous. We rename `copy` to `inherit_for` to clarify we associate a stronger semantic to the operation.

branchcache: stop writing more branchcache file on disk than needed Before this change, we were unconditionally writing a branchmap file for the filter level passed to `update_disk`. This is actually counter productive if no update were needed for this filter level. In many case, the branch cache for a filter level is identical to its parent "subset" and it is better to simply keep the subset update and reuse it every time instead of having to do identical work for similar subset. So we change the `update_disk` method to only write a file when that filter level differ from its parent. This removes many cases where identical files were written, requiring multiple boring update in the test suite. The only notable changes is the change to `test-strip-branch-cache.t`, this case was checking a scenario that no longer reproduce the bug as writing less branchmap file result in less stalled cache on disk. Strictly speaking, we could create a more convoluted scenario that create a similar issue. However the next changeset would also cover that scenario so we directly updated that test case to a "no longer buggy" state.

branchcache: do not copy the `_dirty` flag If the inherited branch cache is dirty, it will be written on disk, and the super-set did not need to modify it, the on disk value for the subset will be re-useable as is. So the super set does not needs to write the very same content itself.

branchcache: explicitly assert that copy is always about inheritance This would catch cases where copy is used for something else if any existed.

branchcache: stop using `copy(…)` in `replace(…)` The `copy` method is mostly used for a filter level to inherit the branchmap from a subset. So we stop using (abusing) it in "replace" to ensure `copy` is used only for inheritance purposes. Since `replace` is a method of the BranchMapCache, it seems fine to do lower level operation there.

branchcache: change the _delayed flag to an explicit `_dirty` flag This is more consistent with the logic we use for other object and it open the way to a clearer management of the cache state. Now, cache are created clean, cache update mark them dirty, writing them on disk mark them clean again.

branchcache: write branchmap in subset inheritance order This way, we can guarantee a valid subset has been written before touching the branchmap of another filter. This is especially useful as we are bout to start deleting outdated branchmap file.

branchcache: do not accept "empty update" This currently does not happens and it will be simpler that is remains that way. If all update do something, we will be able to simply declare, in a later changesets, that all update to result in a dirty branchcache.

branchcache: avoid created a `None` filter repoview when writing The repoview class is not intended to be used for unfiltered repository.

stream-clone-tests: stop filtering non existent warning This filtering was introduced in 74c004a515bc, however there is already no warning in that changeset. So I guess the warnings existed when we the patch was created but the problem was solved in another changeset that 74c004a515bc, rebased on.

stream-clone-test: simplify case testing obsolescence There is only two important things in this test: - the number of file we send, to show we picked the obsstore. - the resulting state, to show we did alter things in the process. The rest are of the number are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: simplify the case testing phases There is only two important things in this test: - the number of file we send, to show we picked the phase roots. - the resulting phases, to show we did not modified them. The rest are of the number are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: simplify bookmark clone The important things to test here is the number of file included (to catch that the bookmark file was sent). So we keep that part non glob'ed but glob the rest. The glob'ed numbers are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: add a verify call to the "clone while changing" case It seems useful to very that the clone did not result in a corrupted copy.

stream-clone-test: add title to various test cases These case are fine as is, but as we are adding title to all the other as we simplify them, lets add title for all cases.

stream-clone-test: simplify testing of secret cloning restriction Here, we just want to check if the streaming clone is allowed and used or not. We do not care about the details of the clone itself.