Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 15:54:22 +0100] rev 51532
branchcache: filter obsolete revisions sooner
Since we won't do anything with the obsolete revisions, we can just ignore them
sooner.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 10:55:22 +0100] rev 51531
branchcache: skip entries that are topological heads in the on disk file
In the majority of cases, topological heads are also branch heads. We have
efficient way to get the topological heads and efficient way to retrieve
their branch information. So there is little value in putting them in the branch
cache file explicitly. On the contrary, writing them explicitly tend to create
very large cache file that are inefficient to read and update.
So the branch cache v3 format is no longer including them. This changeset focus
on the format aspect and have no focus on the performance aspect. We will cover
that later.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 01:35:43 +0100] rev 51530
branchcache: simplify the branch rev cache test
We don't need that many content dump and this gets in the way in change in
access pattern (e.g. accessing revision in a different order change the order of
branches in the "names" file).
So we simplify this test in advance.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 11:39:44 +0100] rev 51529
branchcache: store filtered hash and obsolete hash independently for V3
This will avoid the bug covered in tests/test-branches-obsolete.t when we stop
storing all heads explicitly in V3.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 12:07:31 +0100] rev 51528
branchcache: show the cache file content in test-branches-obsoletes.t
This help to track the changes in format between v2 and v3.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 02:20:53 +0100] rev 51527
branchcache: rework the `filteredhash` logic to be more generic
We now have a more flexible `key_hashes` tuple. We duplicated various logic in
the V2 and V3 version of the cache as the goal is to start changing the logic
for V3 in the next few changesets.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 01:53:52 +0100] rev 51526
filteredhash: rename the filteredhash function
The new name is less ambiguous, as we are about to introduce an alternative
function it seems like a good idea to have clearer name to distinct the two.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 06 Mar 2024 01:43:51 +0100] rev 51525
filteredhash: split the computation of revision sets
The branch2's filteredhash combines the filtered revisions and the obsolete
ones, this will creates issue for implicit reference to heads we want to
introduce for the v3 of the branch cache format. So we isolate this logic for
alternative use.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 05 Mar 2024 15:21:18 +0100] rev 51524
filteredhash: move the hashing in its own function
This will help us to reuse this logic in variants of the hashes used for branch
cache validation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 25 Feb 2024 23:31:50 +0100] rev 51523
branchcache: cleanup the final key generation after update
A lot of duplicated work seemed to be done, as we already update the tiprev and
tipnode when needed right before. So we simplify that part to focus on the
filtered hash.
See inline comment for details.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 12:56:08 +0100] rev 51522
branchcache: add more test for the logic around obsolescence and branch heads
While working on branch-cache-v3, we noticed some ambiguity in the
filtered+obsolete hash. However this was only caught by a rebase test by
chance.
It seems important to explicitly tests these cases.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:44:44 +0100] rev 51521
branchcache-v3: use more explicit header line
The key-value approach is clearer and gives more rooms to have the format evolve
in a clear way. It also provides extension (like topic) simpler way to extend
the validation scheme.
This is just a small evolution, the V3 format is still a work in progress.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 14:20:36 +0100] rev 51520
branchcache-v3: introduce a v3 format
For now the format is the very same, however we will start changing it in
future changesets.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 27 Feb 2024 14:04:29 +0100] rev 51519
branchcache: use an explicit class for the v2 version
This prepare the introduction of an experimental v3 format version.
In the process, we move the description of the format in that new class.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 27 Feb 2024 15:33:21 +0100] rev 51518
branchcache: add some blank line in a test
This helps each section to stand out.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 25 Mar 2024 02:09:15 +0100] rev 51517
phases: update the phase set as we go during retract boundary
Apparently iterating over the `changed_revs` dictionary is very expensive.
On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets
bundle gives give use the following timing.
e57d4b868a3e: 4.6 seconds
ac1c75188440: 102.5 seconds
prev-changeset: 30.0 seconds
this-changeset: 4.6 seconds
So, the performance regression is gone.
Once again: thanks to marvelous Python!
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 25 Mar 2024 01:50:31 +0100] rev 51516
phases: avoid a potentially costly dictionary interation in some case
If we retract for the draft phase, there is not non-public item to be retracted
and we can skip this part. This part is was apparently super costly thanks to
Python.
On mozilla-try-2019-02-18, a perf::unbundle call with a 10 000 changesets
bundle gives give use the following timing.
e57d4b868a3e: 4.6 seconds
ac1c75188440: 102.5 seconds
this-changeset: 30.0 seconds
So we recovered about ⅔ of the regression, the next changeset will give us the
rest back.
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:26:46 +0100] rev 51515
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:24:42 +0100] rev 51514
Added signature for changeset
2e6fde2ed01e
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:24:36 +0100] rev 51513
Added tag 6.7.1 for changeset
2e6fde2ed01e
Raphaël Gomès <rgomes@octobus.net> [Thu, 21 Mar 2024 12:23:25 +0100] rev 51512
relnotes: add 6.7.1
Felipe Resende <felipe@fcresende.dev.br> [Sat, 16 Mar 2024 21:02:19 -0300] rev 51511
subrepo: fix normalizing paths with scheme
After revision
0afe96e374a7, subrepo paths were normalized using
posixpath.normpath and that resulted in ssh paths being wrongly converted
from ssh://host/path to ssh:/host/path
This fix applies the same logic used in urlutil.url to split the path scheme
from the rest and only use posixpath.normpath to the string after scheme://
Felipe Resende <felipe@fcresende.dev.br> [Sat, 16 Mar 2024 18:37:07 -0300] rev 51510
sshpeer: fix path when handling invalid url exception
In
73ed1d13c0bf the code was refactored but the error handling seems to have
been missed (or maybe the object shoud have implemented __bytes__)
Raphaël Gomès <rgomes@octobus.net> [Mon, 18 Mar 2024 11:25:21 +0100] rev 51509
delta-search: fix crash caused by unbound variable
This code path was apparently not tested. This fixes a crash when cloning the
Tryton repo.
Raphaël Gomès <rgomes@octobus.net> [Fri, 15 Mar 2024 10:52:51 +0100] rev 51508
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Fri, 15 Mar 2024 10:49:44 +0100] rev 51507
Added signature for changeset
c9ceb4f60256
Raphaël Gomès <rgomes@octobus.net> [Fri, 15 Mar 2024 10:49:40 +0100] rev 51506
Added tag 6.7 for changeset
c9ceb4f60256
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 15 Mar 2024 01:31:57 +0100] rev 51505
phases: avoid N² behavior in `advanceboundary`
We allowed duplicated entries in the deque, which each entry could potentially
insert all its ancestors. So advancing boundary for the full repository would
mean each revision would walk all its ancestors, resulting in O(N²) iteration.
For repository of any decent size, N² is quickly insane.
We introduce a simple set to avoid this and get back to reasonable performance.
Raphaël Gomès <rgomes@octobus.net> [Thu, 14 Mar 2024 16:25:46 +0100] rev 51504
relnotes: add 6.7
Raphaël Gomès <rgomes@octobus.net> [Thu, 14 Mar 2024 11:24:52 +0100] rev 51503
admin-commands: move the chainsaw extension to the admin commands module
Activating an extension is always a little bit of a chore and the long name,
options and "chainsaw" bits are deterrent enough.
This also allows us to help the discoverability for people looking for
repo "administration" tools, with the widest semantic of "administration".
Anton Shestakov <av6@dwimlabs.net> [Wed, 13 Mar 2024 16:22:13 -0300] rev 51502
obsutil: sort metadata before comparing in geteffectflag()
This is probably less important now that we dropped Python 2. We do still
support Python 3.6 though, and the dictionaries aren't ordered there either
(that was a big change that came with 3.7).
Still, maybe it's a good idea to sort metadata explicitly.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 11 Mar 2024 11:11:34 +0100] rev 51501
tests: disable revlog compression in test-generaldelta.t (
issue6867)
The revlog compression makes a lot of numbers unstable. Since checking revlog
compression is not the goal of this test, we disable the compression to get
stable numbers.
This should avoid wasting more time on this kind of changes in the future.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 11 Mar 2024 11:09:29 +0100] rev 51500
test-general-delta: actually test optimize-delta-parent-choice=no
Since the configuration was not explicit, the case stopped testing what it
intended to test when the default value changed.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 11 Mar 2024 13:09:01 +0100] rev 51499
test-chg: stabilize the log checking
The "worker process exited" line have been making the CI flaky for a long time.
Lets sort this out.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 11 Mar 2024 12:03:40 +0100] rev 51498
tests: fix test-patchbomb-tls.t instability
The flakiness on chg is caused by a client that exit faster than the server
output log.
So actively wait for the server to issue the expected output (with a small
timeout)
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 11 Mar 2024 16:05:28 +0100] rev 51497
test-lock: use synchronisation file instead of sleep
This will prevent the test to be flaky on load.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 10 Mar 2024 03:29:12 +0100] rev 51496
branchcache: use update_disk to refresh 'served' and 'served.hidden'
The `update_disk` method is dedicated to this kind of usecase. Now that the writting patterns are more consistent, we can use it to warm these two important cache.
I am dropping the first comment about "refreshing all the others" because it is
false. If a branchmap already exist for "served", none of the subset will be
updated.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 10 Mar 2024 03:25:04 +0100] rev 51495
branchcache: explictly update disk state only if no transaction exist
If a transaction exist the `write_dirty` call will eventually be done and the state will be synched on disk. It is better to no interfer with that.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 10 Mar 2024 03:32:50 +0100] rev 51494
branchcache: do not use `__getitem__` in updatecache
The `update_disk` method uses `updatecache` and the point of `update_disk` is to be able to do alternative processing to the one we do in `__getitem__`. So we calling `__getitem__` in `updatecache` defeat this purpose.
Instead we do the equivalent explicitly to preserve the spirit of `update_disk` (that we will actually put to use soon, I promise)
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 10 Mar 2024 05:10:00 +0100] rev 51493
branchcache: explicitly track inheritence "state"
We move from a binary "dirty" flag to a three value "state": "clean", "inherited", "dirty".
The "inherited" means that the branch cache is not only "clean", but it is a
duplicate of its parent filter.
If a branch cache is "inherited", we can non only skip writing its value on
disk, but it is a good idea to delete any stale value on disk, as those will
just waste time (and possibly induce bug) in the future.
We only do this in the update related to transaction or explicit cache update
(e.g `hg debugupdatecache`). Deleting the file when we simply detected a stall
cache during a read only operation seems more dangerous.
We rename `copy` to `inherit_for` to clarify we associate a stronger semantic
to the operation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 10 Mar 2024 04:53:17 +0100] rev 51492
branchcache: stop writing more branchcache file on disk than needed
Before this change, we were unconditionally writing a branchmap file for the
filter level passed to `update_disk`. This is actually counter productive if no
update were needed for this filter level. In many case, the branch cache for a
filter level is identical to its parent "subset" and it is better to simply
keep the subset update and reuse it every time instead of having to do identical
work for similar subset.
So we change the `update_disk` method to only write a file when that filter
level differ from its parent. This removes many cases where identical files were
written, requiring multiple boring update in the test suite.
The only notable changes is the change to `test-strip-branch-cache.t`, this
case was checking a scenario that no longer reproduce the bug as writing less
branchmap file result in less stalled cache on disk.
Strictly speaking, we could create a more convoluted scenario that create a
similar issue. However the next changeset would also cover that scenario so we
directly updated that test case to a "no longer buggy" state.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 16:49:06 +0100] rev 51491
branchcache: do not copy the `_dirty` flag
If the inherited branch cache is dirty, it will be written on disk, and the
super-set did not need to modify it, the on disk value for the subset will be
re-useable as is. So the super set does not needs to write the very same content
itself.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 16:52:08 +0100] rev 51490
branchcache: explicitly assert that copy is always about inheritance
This would catch cases where copy is used for something else if any existed.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 09 Mar 2024 02:07:15 +0100] rev 51489
branchcache: stop using `copy(…)` in `replace(…)`
The `copy` method is mostly used for a filter level to inherit the branchmap
from a subset. So we stop using (abusing) it in "replace" to ensure `copy` is
used only for inheritance purposes.
Since `replace` is a method of the BranchMapCache, it seems fine to do lower
level operation there.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 16:47:32 +0100] rev 51488
branchcache: change the _delayed flag to an explicit `_dirty` flag
This is more consistent with the logic we use for other object and it open the way to a clearer management of the cache state.
Now, cache are created clean, cache update mark them dirty, writing them on
disk mark them clean again.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 15:50:15 +0100] rev 51487
branchcache: write branchmap in subset inheritance order
This way, we can guarantee a valid subset has been written before touching the
branchmap of another filter.
This is especially useful as we are bout to start deleting outdated branchmap
file.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 15:06:54 +0100] rev 51486
branchcache: do not accept "empty update"
This currently does not happens and it will be simpler that is remains that way.
If all update do something, we will be able to simply declare, in a later
changesets, that all update to result in a dirty branchcache.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 11:04:34 +0100] rev 51485
branchcache: avoid created a `None` filter repoview when writing
The repoview class is not intended to be used for unfiltered repository.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:49:55 +0100] rev 51484
stream-clone-tests: stop filtering non existent warning
This filtering was introduced in
74c004a515bc, however there is already no
warning in that changeset. So I guess the warnings existed when we the patch
was created but the problem was solved in another changeset that
74c004a515bc,
rebased on.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:46:12 +0100] rev 51483
stream-clone-test: simplify case testing obsolescence
There is only two important things in this test:
- the number of file we send, to show we picked the obsstore.
- the resulting state, to show we did alter things in the process.
The rest are of the number are very fragile and consume a lot of time for little
value when adjusting formats, caches, and protocol.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:43:07 +0100] rev 51482
stream-clone-test: simplify the case testing phases
There is only two important things in this test:
- the number of file we send, to show we picked the phase roots.
- the resulting phases, to show we did not modified them.
The rest are of the number are very fragile and consume a lot of time for little
value when adjusting formats, caches, and protocol.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:39:10 +0100] rev 51481
stream-clone-test: simplify bookmark clone
The important things to test here is the number of file included (to catch that
the bookmark file was sent). So we keep that part non glob'ed but glob the
rest.
The glob'ed numbers are very fragile and consume a lot of time for little value
when adjusting formats, caches, and protocol.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:31:42 +0100] rev 51480
stream-clone-test: add a verify call to the "clone while changing" case
It seems useful to very that the clone did not result in a corrupted copy.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 10:59:51 +0100] rev 51479
stream-clone-test: add title to various test cases
These case are fine as is, but as we are adding title to all the other as we
simplify them, lets add title for all cases.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:28:07 +0100] rev 51478
stream-clone-test: simplify testing of secret cloning restriction
Here, we just want to check if the streaming clone is allowed and used or not.
We do not care about the details of the clone itself.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:26:27 +0100] rev 51477
stream-clone-test: simplify the background file closing test
Here we just care about the fact the background file closing logic actually ran. We don't need to check the details of the cloning.
The details of the output is very fragile and consume a lot of time for little
value when adjusting formats, caches, and protocol. So we filter it out.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 10:51:01 +0100] rev 51476
stream-clone-test: simplify the --uncompressed alias check
To check that --uncompressed is an alias we just need to check it trigger a
stream clone, we don't need to check anything else.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 08 Mar 2024 10:50:42 +0100] rev 51475
stream-clone-test: drop an automatic pattern replacement
That pattern is nice, but it prevent us to glob the number of bytes when we
don't care about them. We don't care about them more often that what we
currently checks so dropping this pattern will help use to simplify various
tests.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:15:33 +0100] rev 51474
stream-clone-test: simplify the test for getbundle with stream=1
The core of this tests is about checking we receive a stream bundle with such
request. We don't need to look at too much of the details of the stream itself.
Since the content of the stream if shifting overtime, Such check is very
fragile and consume a lot of time for little value when adjusting formats,
caches, and protocol.
So we reduce the size of what we check to focus on "is this a stream clone"
question.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:05:28 +0100] rev 51473
stream-clone-test: factor some piece of basic clone test out
Multiple parts of this case (listing cache, checking error) are common to all
cases and don't need to be in the conditionnal block.
This simplify the test update.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Feb 2024 22:01:09 +0100] rev 51472
stream-clone-test: simplify the case where server disabled it
We have an option to disable it, we don't need to test it with all protocol
variants.
In addition there is little value in looking at the bytes to bytes details of
the reply. Such check is very fragile and consume a lot of time for little
value when adjusting formats, caches, and protocol.
Georges Racinet <georges.racinet@octobus.net> [Mon, 11 Mar 2024 13:36:25 +0100] rev 51471
rust-matchers: raw regular expression builder
Extracting this `re_builder()` from `re_matcher()` makes it reusable
in more general cases than matching `HgPath` instances and would
help reducing code duplication in RHGitaly.
Georges Racinet <georges.racinet@octobus.net> [Mon, 11 Mar 2024 13:23:18 +0100] rev 51470
rust-filepatterns: export glob_to_re function
Making this function public should not risk freezing the internal API,
and it can be useful for all downstream code that needs to perform
glob matching against byte strings, such as RHGitaly where it will
be useful to match on branches and tags.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 11 Mar 2024 01:20:12 +0100] rev 51469
repoview: prevent `None` to be passed as the filtername
We let such instantiation slip in a previous commit, so we add an explicit check
to prevent it to happen in the future.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 07 Mar 2024 11:04:34 +0100] rev 51468
branchcache: avoid created a `None` filter repoview when writing
The repoview class is not intended to be used for unfiltered repository.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 05 Mar 2024 15:07:47 +0100] rev 51467
rust-index: don't use mutable borrow to computed filtered heads
This does not need to mutate the index.
This is the prime suspect for some RuntimeError raised during some pushes.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 05 Mar 2024 15:07:04 +0100] rev 51466
rust-index: don't use mutable borrow for head-diff computation
It does not needs to mutate the index.
This is one of the two suspects of RuntimeError being thrown during push.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:26:08 +0100] rev 51465
branchcache: move head writing in a `_write_headers` method
Same rational: this will help having format variants.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:25:41 +0100] rev 51464
branchcache: move head writing in a `_write_heads` method
Same rational: this will help having format variants.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:23:45 +0100] rev 51463
branchcache: move the header loading in a `_load_header` class method
This will help changing header parsing in format variants.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:15:10 +0100] rev 51462
branchcache: simplify a long line
Gratuitous change to help code readability.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:12:20 +0100] rev 51461
branchcache: rename `load` to `_load_heads`
We are about to have more similar function, we rename the existing one to a more
meaningful name and mark it private in the process.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 25 Feb 2024 20:40:37 +0100] rev 51460
branchcache: move the filename to a class attribute
This prepare the introduction of more variant of cache.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 27 Feb 2024 22:52:00 +0100] rev 51459
test-clonebundles: simplify matching to be less flavor depends
We keep the files and bytes output for the first call, but then we mostly check
that we are being served a stream-clone bundle, not the actual content and size
of the bundle. That aspect being tested by the stream clone test themselves.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 25 Feb 2024 23:05:33 +0100] rev 51458
repoview: fix changelog.__contains__ method
This have been around for ten years, so we can safely that this method have few
callers. However I am about to add one.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 08 Jan 2024 15:11:34 +0100] rev 51457
branchcache: unconditionally write delayed branchmap
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 25 Feb 2024 16:14:15 +0100] rev 51456
branchcache: drop the unused `_verifyclosed`
This code appears dead since its introduction about 5 years ago in this three
consecutive commits:
-
6578654916ae → introduce the method with two calls
-
7c9d4cf23adf → remove first call
-
be5eeaf5c24a → remove second call
o changeset:
be5eeaf5c24a
| user: Pulkit Goyal <pulkit@yandex-team.ru>
| date: Fri Apr 05 15:57:09 2019 +0300
| summary: branchcache: don't verify closed nodes in _branchtip()
|
o changeset:
7c9d4cf23adf
| user: Pulkit Goyal <pulkit@yandex-team.ru>
| date: Fri Apr 05 15:56:33 2019 +0300
| summary: branchcache: don't verify closed nodes in iteropen()
|
o changeset:
6578654916ae
| user: Pulkit Goyal <pulkit@yandex-team.ru>
~ date: Mon Apr 01 13:56:47 2019 +0300
summary: branchcache: lazily validate nodes from the branchmap
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 15:46:24 +0100] rev 51455
branchcache: dispatch the code into the dedicated subclass
The code useful only to the local brancache have now been moved into the
dedicated subclass. This will help improving the branchcache code without subtle
breaking the remote variants.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 25 Feb 2024 14:09:36 +0100] rev 51454
branchcache: introduce a base class for branchmap
This will help define a clear boundary between the two.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Feb 2024 12:09:06 +0100] rev 51453
branchcache: fix the copy code
We copy some internal attribute along too. This should prevent inconsistency in
the resulting branchmap.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Feb 2024 13:11:42 +0100] rev 51452
branchcache: pass a "verify_node" attribut to __init__ instead of hasnode
The hasnode callback cannot be inherited and is dropped on copy, which seems
like a bad idea. Instead we pass the actual semantic as a parameter and let the
internal logic deal with it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Feb 2024 11:59:56 +0100] rev 51451
branchcache: stop storing a repository instance on the cache altogether
We did not really needed it and we do not needs it anymore at all. So lets make
things simpler for consistency and garbage collecting and stop storing it
altogether.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Feb 2024 11:43:19 +0100] rev 51450
branchcache: pass the target repository when copying
Branchmap are usually copied to be used on a different repoview using a
different filter level. Passing the repository around means the repository in
`branchcache._repo` will drift from the actual branchmap filter.
This is currently "fine" because the repo is only used to retrieve the `nullid`
value. However, this is a fairly big trap for any extension or future code using
the `_repo` attribute.
The replace logic is now using a copy to ensure the right repository view is
used to initialized the cached value.
We add a couple of assert for make sure this inconsistency does not sneak back.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 19 Jan 2024 11:30:10 +0100] rev 51449
branchcache: have an explicit method to update the on disk cache
Explicit is better and will give use more flexibility for future evolution of
the storage.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 29 Feb 2024 14:13:21 -0800] rev 51448
crecord: drop calls to `curses.endwin()`
We got a bug report where `curses.endwin()` failed with `_curses.error: endwin()
returned ERR`. Looking at
e306d552dfb12, it seems like we should be able to just
remove these calls.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 04 Mar 2024 04:16:15 +0100] rev 51447
config: move the option to mmap rev branch cache in the storage section
See previous commit for rational.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 04 Mar 2024 04:13:33 +0100] rev 51446
config: document the storage and format sections
This should help people to put configuration in the right section.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 12:59:57 +0100] rev 51445
rust-index: drop offset_override
The inline `offsets` value diverge from the one on disk for added value, so the
offset_override tricks is not going to work well once we start having the full
revlog logic in Rust.
We remove it beforehand and align the Rust logic to the Python one (adjusting
the segment offset at read time for inline revlog).
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 26 Feb 2024 13:41:02 +0100] rev 51444
rust-index: stop calling `with_offset` in the tests
We are not adding any data, so why are we setting any offset?
Raphaël Gomès <rgomes@octobus.net> [Fri, 23 Feb 2024 15:57:50 +0100] rev 51443
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Fri, 23 Feb 2024 15:55:53 +0100] rev 51442
Added signature for changeset
d1d48d18db37
Raphaël Gomès <rgomes@octobus.net> [Fri, 23 Feb 2024 15:55:49 +0100] rev 51441
Added tag 6.7rc0 for changeset
d1d48d18db37
Raphaël Gomès <rgomes@octobus.net> [Fri, 23 Feb 2024 15:18:29 +0100] rev 51440
relnotes: add 6.7rc0
Raphaël Gomès <rgomes@octobus.net> [Fri, 23 Feb 2024 15:18:17 +0100] rev 51439
relnotes: remove outdated message from `next`
Raphaël Gomès <rgomes@octobus.net> [Fri, 23 Feb 2024 15:10:44 +0100] rev 51438
branching: merge default into stable for 6.7rc0
Raphaël Gomès <rgomes@octobus.net> [Fri, 23 Feb 2024 15:09:18 +0100] rev 51437
branching: merge stable into default
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 14:07:33 +0100] rev 51436
perf: add a --as-push option to perf::unbundle
This turned out to make a quite significant difference.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 06:25:09 +0100] rev 51435
chainsaw-update: exit early if one of the intermediate command fails
That will prevent the user to be presented with a start that pretend to be
consistent with the request, but is not.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 03:32:35 +0100] rev 51434
chainsaw-update: lock the repository for the duration of the operation
This should prevent and catch some misusage where something else try to touch
the repository.
Georges Racinet <georges.racinet@octobus.net> [Fri, 23 Feb 2024 11:41:55 +0100] rev 51433
chainsaw-update: taking care of initial cloning
Perhaps we should go just a bit lower level than this `instance()`,
since the main added value in our use-case is full path resolution,
that we need to do anyway for the rmtree cleanup.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 11:30:58 +0100] rev 51432
chainsaw-update: use a graph with branching in graph
This will be relevant for the next improvement of `chainsaw-update`.
Georges Racinet <georges.racinet@octobus.net> [Wed, 17 Jan 2024 14:39:06 +0100] rev 51431
chainsaw-update: log actual locks breaking
Previously, the command would simply state that it was about
to break locks, not if there was actually some to break.
This version is race-free. It would be also possible to display
the content of the lock before hand (not race-free but informative
in almost all cases).
Georges Racinet <georges.racinet@octobus.net> [Wed, 17 Jan 2024 14:26:58 +0100] rev 51430
vfs: have tryunlink tell what it did
It is useful in certain circumstances to know whether vfs.tryunlink()
actually removed something or not, be it for logging purposes.
Georges Racinet <georges.racinet@octobus.net> [Sat, 26 Nov 2022 12:23:56 +0100] rev 51429
chainsaw: new extension for dangerous operations
The first provided command is `chainsaw-update`, whose one and single job is
to make sure that it will pull, update and purge the target repository,
no matter what may be in the way (locks, notably), see docstring for rationale.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 03:45:07 +0100] rev 51428
rust: disable the RustIndex without persistent nodemap
See rational inline.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 03:44:56 +0100] rev 51427
rust: stop claiming the C index is compatible with the rust code
This is no longer the case since the introduction of the pure Rust Index, and
was probably not the case since the MixedIndex itself.
So we fix the dedicated attribute value.
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Feb 2024 15:11:26 +0100] rev 51426
rust-index: remove one collect when converting back
Turns out this is slightly faster. Sending the results back to Python is still
the most costly (like 75% of the time) of the whole method, but it's about
as fast as it can be now.
hg perf::phases on mozilla-try-2023-03-22
before: 0.267114
after: 0.247101
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Feb 2024 15:06:16 +0100] rev 51425
rust-index: improve phase computation speed
While less memory efficient, using an array is *much* faster than using a
HashMap, especially with the default hasher. It even makes the code simpler,
so I'm not really sure what I was thinking in the first place, maybe it's more
obvious now.
This fix a significant performance regression when using the rust version of the
code. (however, the C code still outperform rust on this operation)
hg perf::phases on mozilla-try-2023-03-22
- 6.6.3: 0.451239 seconds
- before: 0.982495 seconds
- after: 0.265347 seconds
- C code: 0.183241 second
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 06:37:25 +0100] rev 51424
phases: directly update the phase sets in advanceboundary
This is similar to what we do in retractboundary. There is no need to invalidate
the cache if we have everything at hand to update it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 05:25:35 +0100] rev 51423
phases: large rework of advance boundary
In a similar spirit as the rework of retractboundary, the new algorithm is doing
an amount of work in the order of magnitude of the amount of changeset that
changes phases. (except to find new roots in impacted higher phases if any may
exists).
This result in a very significant speedup for repository with many old draft
like mozilla try.
runtime of perf:unbundle for a bundle constaining a single changeset (C code):
before 6.7 phase work: 14.497 seconds
before this change: 6.311 seconds (-55%)
with this change: 2.240 seconds (-85%)
Combined with the other patches that fixes the phases computation in the Rust
index, the rust code with a persistent nodemap get back to quite interresting
performances with 2.026 seconds for the same operation, about 10% faster than
the C code.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 22 Feb 2024 19:21:14 +0100] rev 51422
phases: apply similar early filtering to advanceboundary
advanceboundary is called the push's unbundle (but not the other unbundle) so
advanceboundary did not show up the profile I looked at so far.
We start with simple pre-filtering to avoid doing any work if we don't needs
too.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Feb 2024 11:09:25 +0100] rev 51421
phases: filter revision that are already in the right phase
No need to compute new roots if everything is already in order.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Feb 2024 13:05:29 +0100] rev 51420
phases: invalidate the phases set less often on retract boundary
We already have the information to update the phase set, so we do so directly
instead of invalidating the cache.
This show a sizeable speedup in our `perf::unbundle` benchmark on the
many-draft mozilla-try repository.
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.perf.perf-unbundle
# bin-env-vars.hg.flavor = no-rust
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.
issue6528 = disabled
# benchmark.variants.revs = last-10
before: 2.055259 seconds
after: 1.887064 seconds (-8.18%)
# benchmark.variants.revs = last-100
before: 2.409239 seconds
after: 2.222429 seconds (-7.75%)
# benchmark.variants.revs = last-1000
before: 3.945648 seconds
after: 3.762480 seconds (-4.64%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Feb 2024 13:05:23 +0100] rev 51419
phases: incrementally update the phase sets when reasonable
When the amount of manual walking is small, we update the phases set manually
instead of computing them from scratch. This should help small update. The next
changesets will make this used more often by reducing the amount of full
invalidation we do on roots upgrade.
The criteria for using an incremental upgrade are arbitrary, however, it "should
never hurt".
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 23 Feb 2024 00:01:33 +0100] rev 51418
phasees: properly shallow caopy the phase sets dictionary
We are about to increments the set more incrementally in some case, so we need
to make a proper shallow copy of it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Feb 2024 14:42:13 +0100] rev 51417
phases: pass an unfiltered repository to _ensure_phase_sets
It seems better for such a low level function to be able to assume it operate on
a real repository.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Feb 2024 13:01:25 +0100] rev 51416
phases: drop set building in `hasnonpublicphases`
We don't actually use the set, so why do we ensure they are built?
(we should also clean up the use of repository argument but that's a quest for later).
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Feb 2024 11:59:28 +0100] rev 51415
phases: gather the logic for phasesets update in a single method
This logic is duplicated around for no good reason, we gather it in a single
place.
The conditional is the new function are a bit weird as we about going to extend it soon.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 22 Feb 2024 10:58:54 +0100] rev 51414
phases: change the way we warm the phasecache in repocache
Same logic as for the previous chngeset. We are about to rename and change the
method used here.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 22 Feb 2024 10:56:05 +0100] rev 51413
phases: use a more generic way to trigger a phases computation for perf
Querying the tip most revision will require the cache to warm the same as
calling the dedicated method. This avoid using a method that is mostly meant for
internal use and will be renamed in a coming changesets.