phases: large rework of advance boundary In a similar spirit as the rework of retractboundary, the new algorithm is doing an amount of work in the order of magnitude of the amount of changeset that changes phases. (except to find new roots in impacted higher phases if any may exists). This result in a very significant speedup for repository with many old draft like mozilla try. runtime of perf:unbundle for a bundle constaining a single changeset (C code): before 6.7 phase work: 14.497 seconds before this change: 6.311 seconds (-55%) with this change: 2.240 seconds (-85%) Combined with the other patches that fixes the phases computation in the Rust index, the rust code with a persistent nodemap get back to quite interresting performances with 2.026 seconds for the same operation, about 10% faster than the C code.

phases: apply similar early filtering to advanceboundary advanceboundary is called the push's unbundle (but not the other unbundle) so advanceboundary did not show up the profile I looked at so far. We start with simple pre-filtering to avoid doing any work if we don't needs too.

phases: filter revision that are already in the right phase No need to compute new roots if everything is already in order.

phases: invalidate the phases set less often on retract boundary We already have the information to update the phase set, so we do so directly instead of invalidating the cache. This show a sizeable speedup in our `perf::unbundle` benchmark on the many-draft mozilla-try repository. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.perf.perf-unbundle # bin-env-vars.hg.flavor = no-rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.revs = last-10 before: 2.055259 seconds after: 1.887064 seconds (-8.18%) # benchmark.variants.revs = last-100 before: 2.409239 seconds after: 2.222429 seconds (-7.75%) # benchmark.variants.revs = last-1000 before: 3.945648 seconds after: 3.762480 seconds (-4.64%)

phases: incrementally update the phase sets when reasonable When the amount of manual walking is small, we update the phases set manually instead of computing them from scratch. This should help small update. The next changesets will make this used more often by reducing the amount of full invalidation we do on roots upgrade. The criteria for using an incremental upgrade are arbitrary, however, it "should never hurt".

phasees: properly shallow caopy the phase sets dictionary We are about to increments the set more incrementally in some case, so we need to make a proper shallow copy of it.

phases: pass an unfiltered repository to _ensure_phase_sets It seems better for such a low level function to be able to assume it operate on a real repository.

phases: drop set building in `hasnonpublicphases` We don't actually use the set, so why do we ensure they are built? (we should also clean up the use of repository argument but that's a quest for later).

phases: gather the logic for phasesets update in a single method This logic is duplicated around for no good reason, we gather it in a single place. The conditional is the new function are a bit weird as we about going to extend it soon.

phases: change the way we warm the phasecache in repocache Same logic as for the previous chngeset. We are about to rename and change the method used here.

phases: use a more generic way to trigger a phases computation for perf Querying the tip most revision will require the cache to warm the same as calling the dedicated method. This avoid using a method that is mostly meant for internal use and will be renamed in a coming changesets.

phases: fix an overzealous invalidation of the phase sets If `len(cl) == self._loadedrevslen` the cache is up to date.

phases: type annotation for `_phasesets` Does not hurt.

phases: leverage the collected information to record phase update Since the lower level function already gather this information, we can directly use it. This comes with a small change to the test that are actually fixing them. The previous version over-reported some phase change that did not exists. In both case, we are force revision `1` to be secret and `0` remains draft`, the previous code wrongly reported `0` as moving to secret while it properly remained draft in the repository.

phases: large rewrite on retract boundary The new code is still pure Python, so we still have room to going significantly faster. However its complexity of the complex part is `O(|[min_new_draft, tip]|)` instead of `O(|[min_draft, tip]|` which should help tremendously one repository with old draft (like mercurial-devel or mozilla-try). This is especially useful as the most common "retract boundary" operation happens when we commit/rewrite new drafts or when we push new draft to a non-publishing server. In this case, the smallest new_revs is very close to the tip and there is very few work to do. A few smaller optimisation could be done for these cases and will be introduced in later changesets. We still have iterate over large sets of roots, but this is already a great improvement for a very small amount of work. We gather information on the affected changeset as we go as we can put it to use in the next changesets. This extra data collection might slowdown the `register_new` case a bit, however for register_new, it should not really matters. The set of new nodes is either small, so the impact is negligible, or the set of new nodes is large, and the amount of work to do to had them will dominate the overhead the collecting information in `changed_revs`. As this new code compute the changes on the fly, it unlock other interesting improvement to be done in later changeset.

phases: fast path public phase advance when everything is public Everything is already public, so we have nothing to do here.

phases: fast path retract of public phase There are no boundary to retract, so lets do nothing.

phases: keep internal state as rev-num instead of node-id Node-id are expensive to work with, dealing with revision is much simple and faster. The fact we still used node-id here shows how few effort have been put into making the phase logic fast. We tend to no longer use node-id internally for about ten years. This has a large impact of repository with many draft roots. For example this Mozilla-try copy have ½ Million draft roots and `perf::unbundle` see a significant improvement. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.perf.perf-unbundle # bin-env-vars.hg.flavor = no-rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.revs = last-1 before:: 1.746791 seconds after:: 1.278379 seconds (-26.82%) # benchmark.variants.revs = last-10 before:: 3.145774 seconds after:: 2.103735 seconds (-33.13%) # benchmark.variants.revs = last-100 before:: 3.487635 seconds after:: 2.446749 seconds (-29.85%) # benchmark.variants.revs = last-1000 before:: 5.007568 seconds after:: 3.989923 seconds (-20.32%)

phases: do filtering at read time This remove the need for the `filterunknown` method at all.

phases: always write with a repo In the future change that move the internal representation of phase-roots from node-id to rev-num, we will use a repository to translate revision numbers back to node at write time. Since that future change is quite complicated already, we do this small API change beforehand.

phases: mark `phasecache.phaseroots` private We are about to change its content from nodeid to revnum. So anyone directly using the content might be in unexpected troubles. We start by making it private to explicitly break any such user (and discourage them to do so).

phases: check secret presence the right way during discovery There is an official function for this, lets use it. This will prevent the code to break in the future while we refactor the phase code.

phases: explicitly filter stripped revision at strip time Explicit is better than implicit. The current logic is bit subtle and fragile. It also get in the way of using something else than node-id as internal storage. We replace it with a more explicit filtering while striping.

debug: add a debug::unbundle command that simulate the unbundle from a push The code have different behavior when the unbundle comes from a push, so we introduce a command that can simulate such unbundle. For our copy of mozilla-try-2023-03-22, this make the unbundle jump from 2.5 seconds (with `hg unbundle`) to 15 seconds (with `hg debug::unbundle`). That 15 seconds timings is consistent with the issue seen in production.

annotate: limit output to range of lines

revlog: add a Rust implementation of `headrevsdiff` Python implementation of `headrevsdiff` can be very slow in the worst case compared with the `heads` computation it replaces, since the latter is done in Rust. Even the average case of this Python implementation is still noticeable in the profiles. This patch makes the computation much much faster by doing it in Rust.

revlog: add a C implementation of `headrevsdiff` Python implementation of `headrevsdiff` can be very slow in the worst case compared with the `heads` computation it replaces, since the latter is done in C. Even the average case of this Python implementation is still noticeable in the profiles. This patch makes the computation much much faster by doing it in C.

unbundle: faster computation of changed heads To compute the set of changed heads it's sufficient to look at the recent commits, instead of looking at all heads currently in existence.

branching: merge stable into default

hg-core: separate timestamp and extra methods