perf: create the temporary target next to the source in stream-consume See inline comment for rational.

bundlespec: rationalize the way we specify stream bundle version Instead of having weird dedicated option for each version (v2, v3, etc) we reuse the same "stream" parameters. This is consistent with the ability to request a stream clone using "none-v2;stream=v2". This changeset introduce no user visible change, this is pure internal cleaning.

bundle: do no check the changegroup version if no changegroup is included We don't need to check the compatibility of something we will not use. In practice this was getting in the was of `streamv2` bundles on a narrow repository as the 'cg.version=02' value was rejected by this checks.

perf-stream-consume: use the source repository config when applying This might contains critical configuration for the benchmark, like enabling of extensions like narrow.

unbundle: move most of the logic on cmdutil to help debug::unbundle reuse This make sure `hg debug::unbundle` focus on the core logic.

postincoming: move to cmdutil This looks like a good place for it to live.

postincoming: avoid computing branchhead if no report will be posted This otherwise defeat some of the branch v3 optimization.

streamclone: stop listing files for entries that have no volatile files This will save a lot of python related time. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) prev-change: 20.88 seconds (-40%) prev-change: 14.22 seconds (-60%) this-change: 11.58 seconds (-67% from baseline; -18% from prev)

stream-clone: disable gc for the initial section for the v3 format The number of small container created turn Python in a gc-frenzy that seriously impact performance. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) prev-change: 20.88 seconds (-40%) this-change: 14.22 seconds (-60% from baseline; -31% from prev)

stream-clone: disable gc for `_entries_walk` duration The number of small container created turn Python in a gc-frenzy that seriously impact performance. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.04 seconds prev-change: 24.51 seconds (-30%) this-change: 20.88 seconds (-40% from baseline; -15% from previous changes)

nocg: make the utility work are both a decorator and context manager In some case, the context manager version will be simpler.

stream-clone: stop getting the file size of all file in v3 The point of v3 is to do less work in the locked section. It was currently not the case. This significantly boost performance. The following number comes from a large private repository using perf::stream-locked-section: base-line: 35.03 seconds this-change: 24.50 seconds (-30%)

stream: in v3, skip the "size" fast path if the entries as some unknown size We are about to prefetch size during the lock less in the v3 case. So we need to avoid trying to use that prefetched size when it is not available. See next changeset for the motivation.

perf-stream-locked-section: advertise the right version key in the help As the v3 format is still experimental, its key is "v3-exp". The help text was not pointing that out. (we also fix `perf::stream-generate` in the process)

perf-stream-locked-section: fix the call to the v3 generator That generator simply return chunks so we should not assign the return to a tuple.

perf-stream-locked-section: actually use v1 generation when requested We were fetching a v1 generator but actually using the v2 function…

branching: merge stable into default

branching: merge stable into default

branchcache: allow to detect "pure topological case" for branchmap We don't rum this detection every time we run the branchcache, that would be costly. However we now do it when running `hg debugupdatecache`. This will help existing repository to benefit from the fastpath when possible.

branchcache: add a "pure topological head" fast path In a narrow but actually quick common case, all topological heads are all on the same branch and all open. In this case, computing the branch map is very simple. We can quickly detect situation where this situation will not change. So we update the V3 format to be able to express this situation and upgrade the update code to detect we remains in that mode. The branch cache is populated with the actual value when the branch map is accessed, but the update_disk method can do the update without needing to populate it.

branchcache: move the processing of the new data in a dedicated method In a future changeset, this will allow the V3 of the branch cache to use a fast path when possible.

branchcache: gather newly closed head in a dedicated set This is part of a series to more clearly split the update in two step. This will allow us to introduce a fast path during update in a future changeset.

branchcache: gather new obsolete revision in a set This is part of a series to more clearly split the update in two step. This will allow us to introduce a fast path during update in a future changeset.

branchcache: filter obsolete revisions sooner Since we won't do anything with the obsolete revisions, we can just ignore them sooner.

branchcache: skip entries that are topological heads in the on disk file In the majority of cases, topological heads are also branch heads. We have efficient way to get the topological heads and efficient way to retrieve their branch information. So there is little value in putting them in the branch cache file explicitly. On the contrary, writing them explicitly tend to create very large cache file that are inefficient to read and update. So the branch cache v3 format is no longer including them. This changeset focus on the format aspect and have no focus on the performance aspect. We will cover that later.

branchcache: simplify the branch rev cache test We don't need that many content dump and this gets in the way in change in access pattern (e.g. accessing revision in a different order change the order of branches in the "names" file). So we simplify this test in advance.

branchcache: store filtered hash and obsolete hash independently for V3 This will avoid the bug covered in tests/test-branches-obsolete.t when we stop storing all heads explicitly in V3.

branchcache: show the cache file content in test-branches-obsoletes.t This help to track the changes in format between v2 and v3.

branchcache: rework the `filteredhash` logic to be more generic We now have a more flexible `key_hashes` tuple. We duplicated various logic in the V2 and V3 version of the cache as the goal is to start changing the logic for V3 in the next few changesets.

filteredhash: rename the filteredhash function The new name is less ambiguous, as we are about to introduce an alternative function it seems like a good idea to have clearer name to distinct the two.

filteredhash: split the computation of revision sets The branch2's filteredhash combines the filtered revisions and the obsolete ones, this will creates issue for implicit reference to heads we want to introduce for the v3 of the branch cache format. So we isolate this logic for alternative use.

filteredhash: move the hashing in its own function This will help us to reuse this logic in variants of the hashes used for branch cache validation.

branchcache: cleanup the final key generation after update A lot of duplicated work seemed to be done, as we already update the tiprev and tipnode when needed right before. So we simplify that part to focus on the filtered hash. See inline comment for details.

branchcache: add more test for the logic around obsolescence and branch heads While working on branch-cache-v3, we noticed some ambiguity in the filtered+obsolete hash. However this was only caught by a rebase test by chance. It seems important to explicitly tests these cases.

branchcache-v3: use more explicit header line The key-value approach is clearer and gives more rooms to have the format evolve in a clear way. It also provides extension (like topic) simpler way to extend the validation scheme. This is just a small evolution, the V3 format is still a work in progress.

branchcache-v3: introduce a v3 format For now the format is the very same, however we will start changing it in future changesets.

branchcache: use an explicit class for the v2 version This prepare the introduction of an experimental v3 format version. In the process, we move the description of the format in that new class.

branchcache: add some blank line in a test This helps each section to stand out.

branching: merge stable into default

branching: merge stable into default

branchcache: use update_disk to refresh 'served' and 'served.hidden' The `update_disk` method is dedicated to this kind of usecase. Now that the writting patterns are more consistent, we can use it to warm these two important cache. I am dropping the first comment about "refreshing all the others" because it is false. If a branchmap already exist for "served", none of the subset will be updated.

branchcache: explictly update disk state only if no transaction exist If a transaction exist the `write_dirty` call will eventually be done and the state will be synched on disk. It is better to no interfer with that.

branchcache: do not use `__getitem__` in updatecache The `update_disk` method uses `updatecache` and the point of `update_disk` is to be able to do alternative processing to the one we do in `__getitem__`. So we calling `__getitem__` in `updatecache` defeat this purpose. Instead we do the equivalent explicitly to preserve the spirit of `update_disk` (that we will actually put to use soon, I promise)

branchcache: explicitly track inheritence "state" We move from a binary "dirty" flag to a three value "state": "clean", "inherited", "dirty". The "inherited" means that the branch cache is not only "clean", but it is a duplicate of its parent filter. If a branch cache is "inherited", we can non only skip writing its value on disk, but it is a good idea to delete any stale value on disk, as those will just waste time (and possibly induce bug) in the future. We only do this in the update related to transaction or explicit cache update (e.g `hg debugupdatecache`). Deleting the file when we simply detected a stall cache during a read only operation seems more dangerous. We rename `copy` to `inherit_for` to clarify we associate a stronger semantic to the operation.

branchcache: stop writing more branchcache file on disk than needed Before this change, we were unconditionally writing a branchmap file for the filter level passed to `update_disk`. This is actually counter productive if no update were needed for this filter level. In many case, the branch cache for a filter level is identical to its parent "subset" and it is better to simply keep the subset update and reuse it every time instead of having to do identical work for similar subset. So we change the `update_disk` method to only write a file when that filter level differ from its parent. This removes many cases where identical files were written, requiring multiple boring update in the test suite. The only notable changes is the change to `test-strip-branch-cache.t`, this case was checking a scenario that no longer reproduce the bug as writing less branchmap file result in less stalled cache on disk. Strictly speaking, we could create a more convoluted scenario that create a similar issue. However the next changeset would also cover that scenario so we directly updated that test case to a "no longer buggy" state.

branchcache: do not copy the `_dirty` flag If the inherited branch cache is dirty, it will be written on disk, and the super-set did not need to modify it, the on disk value for the subset will be re-useable as is. So the super set does not needs to write the very same content itself.

branchcache: explicitly assert that copy is always about inheritance This would catch cases where copy is used for something else if any existed.

branchcache: stop using `copy(…)` in `replace(…)` The `copy` method is mostly used for a filter level to inherit the branchmap from a subset. So we stop using (abusing) it in "replace" to ensure `copy` is used only for inheritance purposes. Since `replace` is a method of the BranchMapCache, it seems fine to do lower level operation there.

branchcache: change the _delayed flag to an explicit `_dirty` flag This is more consistent with the logic we use for other object and it open the way to a clearer management of the cache state. Now, cache are created clean, cache update mark them dirty, writing them on disk mark them clean again.

branchcache: write branchmap in subset inheritance order This way, we can guarantee a valid subset has been written before touching the branchmap of another filter. This is especially useful as we are bout to start deleting outdated branchmap file.

branchcache: do not accept "empty update" This currently does not happens and it will be simpler that is remains that way. If all update do something, we will be able to simply declare, in a later changesets, that all update to result in a dirty branchcache.

branchcache: avoid created a `None` filter repoview when writing The repoview class is not intended to be used for unfiltered repository.

stream-clone-tests: stop filtering non existent warning This filtering was introduced in 74c004a515bc, however there is already no warning in that changeset. So I guess the warnings existed when we the patch was created but the problem was solved in another changeset that 74c004a515bc, rebased on.

stream-clone-test: simplify case testing obsolescence There is only two important things in this test: - the number of file we send, to show we picked the obsstore. - the resulting state, to show we did alter things in the process. The rest are of the number are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: simplify the case testing phases There is only two important things in this test: - the number of file we send, to show we picked the phase roots. - the resulting phases, to show we did not modified them. The rest are of the number are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: simplify bookmark clone The important things to test here is the number of file included (to catch that the bookmark file was sent). So we keep that part non glob'ed but glob the rest. The glob'ed numbers are very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol.

stream-clone-test: add a verify call to the "clone while changing" case It seems useful to very that the clone did not result in a corrupted copy.

stream-clone-test: add title to various test cases These case are fine as is, but as we are adding title to all the other as we simplify them, lets add title for all cases.

stream-clone-test: simplify testing of secret cloning restriction Here, we just want to check if the streaming clone is allowed and used or not. We do not care about the details of the clone itself.

stream-clone-test: simplify the background file closing test Here we just care about the fact the background file closing logic actually ran. We don't need to check the details of the cloning. The details of the output is very fragile and consume a lot of time for little value when adjusting formats, caches, and protocol. So we filter it out.