stream-clone-test: simplify testing of secret cloning restriction Here, we just want to check if the streaming clone is allowed and used or not. We do not care about the details of the clone itself.

stream-clone-test: simplify the background file closing test Here we just care about the fact the background file closing logic actually ran. We don't need to check the details of the cloning. The details of the output is very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol. So we filter it out.

stream-clone-test: simplify the --uncompressed alias check To check that --uncompressed is an alias we just need to check it trigger a stream clone, we don't need to check anything else.

stream-clone-test: drop an automatic pattern replacement That pattern is nice, but it prevent us to glob the number of bytes when we don't care about them. We don't care about them more often that what we currently checks so dropping this pattern will help use to simplify various tests.

stream-clone-test: simplify the test for getbundle with stream=1 The core of this tests is about checking we receive a stream bundle with such request. We don't need to look at too much of the details of the stream itself. Since the content of the stream if shifting overtime, Such check is very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol. So we reduce the size of what we check to focus on "is this a stream clone" question.

stream-clone-test: factor some piece of basic clone test out Multiple parts of this case (listing cache, checking error) are common to all cases and don't need to be in the conditionnal block. This simplify the test update.

stream-clone-test: simplify the case where server disabled it We have an option to disable it, we don't need to test it with all protocol variants. In addition there is little value in looking at the bytes to bytes details of the reply. Such check is very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

rust-matchers: raw regular expression builder Extracting this `re_builder()` from `re_matcher()` makes it reusable in more general cases than matching `HgPath` instances and would help reducing code duplication in RHGitaly.

rust-filepatterns: export glob_to_re function Making this function public should not risk freezing the internal API, and it can be useful for all downstream code that needs to perform glob matching against byte strings, such as RHGitaly where it will be useful to match on branches and tags.

repoview: prevent `None` to be passed as the filtername We let such instantiation slip in a previous commit, so we add an explicit check to prevent it to happen in the future.

branchcache: avoid created a `None` filter repoview when writing The repoview class is not intended to be used for unfiltered repository.

rust-index: don't use mutable borrow to computed filtered heads This does not need to mutate the index. This is the prime suspect for some RuntimeError raised during some pushes.

rust-index: don't use mutable borrow for head-diff computation It does not needs to mutate the index. This is one of the two suspects of RuntimeError being thrown during push.

branchcache: move head writing in a `_write_headers` method Same rational: this will help having format variants.

branchcache: move head writing in a `_write_heads` method Same rational: this will help having format variants.

branchcache: move the header loading in a `_load_header` class method This will help changing header parsing in format variants.

branchcache: simplify a long line Gratuitous change to help code readability.

branchcache: rename `load` to `_load_heads` We are about to have more similar function, we rename the existing one to a more meaningful name and mark it private in the process.

branchcache: move the filename to a class attribute This prepare the introduction of more variant of cache.

test-clonebundles: simplify matching to be less flavor depends We keep the files and bytes output for the first call, but then we mostly check that we are being served a stream-clone bundle, not the actual content and size of the bundle. That aspect being tested by the stream clone test themselves.

repoview: fix changelog.__contains__ method This have been around for ten years, so we can safely that this method have few callers. However I am about to add one.

branchcache: unconditionally write delayed branchmap

branchcache: drop the unused `_verifyclosed` This code appears dead since its introduction about 5 years ago in this three consecutive commits: - 6578654916ae → introduce the method with two calls - 7c9d4cf23adf → remove first call - be5eeaf5c24a → remove second call o changeset: be5eeaf5c24a | user: Pulkit Goyal <pulkit@yandex-team.ru> | date: Fri Apr 05 15:57:09 2019 +0300 | summary: branchcache: don't verify closed nodes in _branchtip() | o changeset: 7c9d4cf23adf | user: Pulkit Goyal <pulkit@yandex-team.ru> | date: Fri Apr 05 15:56:33 2019 +0300 | summary: branchcache: don't verify closed nodes in iteropen() | o changeset: 6578654916ae | user: Pulkit Goyal <pulkit@yandex-team.ru> ~ date: Mon Apr 01 13:56:47 2019 +0300 summary: branchcache: lazily validate nodes from the branchmap

branchcache: dispatch the code into the dedicated subclass The code useful only to the local brancache have now been moved into the dedicated subclass. This will help improving the branchcache code without subtle breaking the remote variants.

branchcache: introduce a base class for branchmap This will help define a clear boundary between the two.

branchcache: fix the copy code We copy some internal attribute along too. This should prevent inconsistency in the resulting branchmap.

branchcache: pass a "verify_node" attribut to __init__ instead of hasnode The hasnode callback cannot be inherited and is dropped on copy, which seems like a bad idea. Instead we pass the actual semantic as a parameter and let the internal logic deal with it.

branchcache: stop storing a repository instance on the cache altogether We did not really needed it and we do not needs it anymore at all. So lets make things simpler for consistency and garbage collecting and stop storing it altogether.

branchcache: pass the target repository when copying Branchmap are usually copied to be used on a different repoview using a different filter level. Passing the repository around means the repository in `branchcache._repo` will drift from the actual branchmap filter. This is currently "fine" because the repo is only used to retrieve the `nullid` value. However, this is a fairly big trap for any extension or future code using the `_repo` attribute. The replace logic is now using a copy to ensure the right repository view is used to initialized the cached value. We add a couple of assert for make sure this inconsistency does not sneak back.

branchcache: have an explicit method to update the on disk cache Explicit is better and will give use more flexibility for future evolution of the storage.

crecord: drop calls to `curses.endwin()` We got a bug report where `curses.endwin()` failed with `_curses.error: endwin() returned ERR`. Looking at e306d552dfb12, it seems like we should be able to just remove these calls.

config: move the option to mmap rev branch cache in the storage section See previous commit for rational.

config: document the storage and format sections This should help people to put configuration in the right section.

rust-index: drop offset_override The inline `offsets` value diverge from the one on disk for added value, so the offset_override tricks is not going to work well once we start having the full revlog logic in Rust. We remove it beforehand and align the Rust logic to the Python one (adjusting the segment offset at read time for inline revlog).

rust-index: stop calling `with_offset` in the tests We are not adding any data, so why are we setting any offset?

branching: merge stable into default

Added signature for changeset d1d48d18db37

Added tag 6.7rc0 for changeset d1d48d18db37

relnotes: add 6.7rc0

relnotes: remove outdated message from `next`

branching: merge default into stable for 6.7rc0

branching: merge stable into default

perf: add a --as-push option to perf::unbundle This turned out to make a quite significant difference.

chainsaw-update: exit early if one of the intermediate command fails That will prevent the user to be presented with a start that pretend to be consistent with the request, but is not.

chainsaw-update: lock the repository for the duration of the operation This should prevent and catch some misusage where something else try to touch the repository.

chainsaw-update: taking care of initial cloning Perhaps we should go just a bit lower level than this `instance()`, since the main added value in our use-case is full path resolution, that we need to do anyway for the rmtree cleanup.

chainsaw-update: use a graph with branching in graph This will be relevant for the next improvement of `chainsaw-update`.

chainsaw-update: log actual locks breaking Previously, the command would simply state that it was about to break locks, not if there was actually some to break. This version is race-free. It would be also possible to display the content of the lock before hand (not race-free but informative in almost all cases).

vfs: have tryunlink tell what it did It is useful in certain circumstances to know whether vfs.tryunlink() actually removed something or not, be it for logging purposes.

chainsaw: new extension for dangerous operations The first provided command is `chainsaw-update`, whose one and single job is to make sure that it will pull, update and purge the target repository, no matter what may be in the way (locks, notably), see docstring for rationale.

rust: disable the RustIndex without persistent nodemap See rational inline.

rust: stop claiming the C index is compatible with the rust code This is no longer the case since the introduction of the pure Rust Index, and was probably not the case since the MixedIndex itself. So we fix the dedicated attribute value.

rust-index: remove one collect when converting back Turns out this is slightly faster. Sending the results back to Python is still the most costly (like 75% of the time) of the whole method, but it's about as fast as it can be now. hg perf::phases on mozilla-try-2023-03-22 before: 0.267114 after: 0.247101

rust-index: improve phase computation speed While less memory efficient, using an array is *much* faster than using a HashMap, especially with the default hasher. It even makes the code simpler, so I'm not really sure what I was thinking in the first place, maybe it's more obvious now. This fix a significant performance regression when using the rust version of the code. (however, the C code still outperform rust on this operation) hg perf::phases on mozilla-try-2023-03-22 - 6.6.3: 0.451239 seconds - before: 0.982495 seconds - after: 0.265347 seconds - C code: 0.183241 second

phases: directly update the phase sets in advanceboundary This is similar to what we do in retractboundary. There is no need to invalidate the cache if we have everything at hand to update it.

phases: large rework of advance boundary In a similar spirit as the rework of retractboundary, the new algorithm is doing an amount of work in the order of magnitude of the amount of changeset that changes phases. (except to find new roots in impacted higher phases if any may exists). This result in a very significant speedup for repository with many old draft like mozilla try. runtime of perf:unbundle for a bundle constaining a single changeset (C code): before 6.7 phase work: 14.497 seconds before this change: 6.311 seconds (-55%) with this change: 2.240 seconds (-85%) Combined with the other patches that fixes the phases computation in the Rust index, the rust code with a persistent nodemap get back to quite interresting performances with 2.026 seconds for the same operation, about 10% faster than the C code.

phases: apply similar early filtering to advanceboundary advanceboundary is called the push's unbundle (but not the other unbundle) so advanceboundary did not show up the profile I looked at so far. We start with simple pre-filtering to avoid doing any work if we don't needs too.

phases: filter revision that are already in the right phase No need to compute new roots if everything is already in order.

phases: invalidate the phases set less often on retract boundary We already have the information to update the phase set, so we do so directly instead of invalidating the cache. This show a sizeable speedup in our `perf::unbundle` benchmark on the many-draft mozilla-try repository. ### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog # benchmark.name = hg.perf.perf-unbundle # bin-env-vars.hg.flavor = no-rust # bin-env-vars.hg.py-re2-module = default # benchmark.variants.issue6528 = disabled # benchmark.variants.revs = last-10 before: 2.055259 seconds after: 1.887064 seconds (-8.18%) # benchmark.variants.revs = last-100 before: 2.409239 seconds after: 2.222429 seconds (-7.75%) # benchmark.variants.revs = last-1000 before: 3.945648 seconds after: 3.762480 seconds (-4.64%)

phases: incrementally update the phase sets when reasonable When the amount of manual walking is small, we update the phases set manually instead of computing them from scratch. This should help small update. The next changesets will make this used more often by reducing the amount of full invalidation we do on roots upgrade. The criteria for using an incremental upgrade are arbitrary, however, it "should never hurt".