Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 18 Jun 2023 00:09:39 +0200] rev 50694
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 20 Jun 2023 02:36:52 +0200] rev 50693
backout: remove db9b6ce803a9
That changeset lost its topic for unknown reason and ended up being picked by
some automation. So I backout it previous incarnation until it can get properly
reviewed.
pierre-yves.david@ens-lyon.org [Mon, 19 Jun 2023 23:27:18 +0200] rev 50692
infinitepush: aggressively deprecated infinite push
We add a large warning and an explicit abort. Both can be disable through
config, but neither are explicitly documented. If any user exists I really want
them to get in touch with the community.
Raphaël Gomès <rgomes@octobus.net> [Tue, 13 Jun 2023 14:35:36 +0200] rev 50691
rust-dependencies: switch from `users` to `whoami`
`users` is unmaintained, with the maintainer apparently MIA. `whoami` is a
popular crate that does simple wrapping of platform-specific calls.
This makes the overhead from using `blackbox` lower. It used to be up 10ms
on slower hardware for both calls, now down to <1ms which is always good.
Raphaël Gomès <rgomes@octobus.net> [Mon, 12 Jun 2023 23:41:28 +0200] rev 50690
rust-hg-core: move from `ouroboros` to `self_cell`
`ouroboros` has a fundamental soundness problem that, while not applicable
today, could become applicable given new compiler optimizations.ยน
`self_cell` is a crate that accomplishes a lot of the same things that
`ouroboros` did while remaining sound (that is, unless a new soundness issue
is discovered) by not assuming as much about the memory layout of the program.
`self_cell` has been scrutinized heavily in the past few months by very
competent people, some from the compiler team and has shown no weaknesses
for a while, with a 1.0 stable release coming out a couple months ago.
Our internal API is exactly the same, this is just an implementation detail.
To reiterate, no actual soundness issue was found with our use of `ouroboros`,
but there might be evolutions of `rustc` (or even a future separate compiler)
that could generate unsound code.
[1] https://github.com/joshua-maros/ouroboros/issues/88
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 02 Jun 2023 15:12:05 +0200] rev 50689
debug-revlog-stats: make it use the new store entry API
This give use free tree manifest support in the process.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 02 Jun 2023 15:10:50 +0200] rev 50688
store: use the revlog type from revlog's constant instead of custom one
We already have a revlog type that everybody use. So we use that one instead.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 12 Jun 2023 23:24:41 +0200] rev 50687
repair: use `is_manifestlog` to recognise manifest revlog
This is simpler that comparing flag, and we want to remove these flags anyway.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 04:56:54 +0200] rev 50686
perf: add a perf::stream-consume
We know how long it take to generate, lets check how long it take to apply now.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 04:09:05 +0200] rev 50685
perf: add a perf::stream-generate command
This record the time we take to generate a bundle.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 12 Jun 2023 18:04:09 +0200] rev 50684
perf: add a new "context" argument to timer
This allow to simple setup/teardown outside of the timed section. Especially
using object that need context manager, like a temporary files.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 01:48:10 +0200] rev 50683
perf: add support for stream-v3 during benchmark
This is getting important as the v3 protocol will diverge from the v2 protocol.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 01:43:48 +0200] rev 50682
perf: add a function to find a stream version generator
The logic is clearer and can be reused for other commands in the future.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 18 May 2023 19:23:59 +0100] rev 50681
treemanifest: make `updatecaches` update the nodemaps for all directories
Without this, if the cache for a nested directory is in a bad state,
it's very hard to repair it.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Wed, 31 May 2023 10:37:55 +0100] rev 50680
stream-clone: avoid opening a revlog in case we do not need it
Opening an revlog has a cost, especially if it is inline as we have to scan the
file and construct an index.
To prevent the associated slowdown, we just do a minimal scan to check that an
inline file is still inline, and simply stream the file without creating a
revlog when we can.
This provides a big boost compared to the previous changeset, even if the full
generation is still penalized by the initial gathering of information.
All benchmarks are run on linux with Python 3.10.7.
# benchmark.name = hg.exchange.stream.generate
# benchmark.variants.version = v2
### Compared to the previous changesets
We get a large win all across the board!
# mercurial-2018-08-01-zstd-sparse-revlog
before: 0.250694 seconds
after: 0.105986 seconds (-57.72%)
# pypy-2018-08-01-zstd-sparse-revlog
before: 3.885657 seconds
after: 1.709748 seconds (-56.00%)
# netbeans-2018-08-01-zstd-sparse-revlog
before: 16.679371 seconds
after: 7.687469 seconds (-53.91%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
before: 38.575482 seconds
after: 17.520316 seconds (-54.58%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
before: 81.160994 seconds
after: 37.073753 seconds (-54.32%)
### Compared to 6.4.3
We are still significantly slower than 6.4.3, the extra time is usually twice
slower than the extra time we observe on the locked section, which is a quite
interesting information.
Except for mercurial-central that is much faster. That discrepancy is not really
explained yet.
# mercurial-2018-08-01-zstd-sparse-revlog
6.4.3: 0.072560 seconds
after: 0.105986 seconds (+46.07%) (- 0.03 seconds)
# pypy-2018-08-01-zstd-sparse-revlog
6.4.3: 1.211193 seconds
after: 1.709748 seconds (+41.16%) (-0.45 seconds)
# netbeans-2018-08-01-zstd-sparse-revlog
6.4.3: 4.932843 seconds
after: 7.687469 seconds (+55.84%) (-2.75 seconds)
# mozilla-central-2018-08-01-zstd-sparse-revlog
6.4.3: 34.012226 seconds
after: 17.520316 seconds (-48.49%) (-16.49 seconds)
# mozilla-try-2019-02-18-zstd-sparse-revlog
6.4.3: 23.850555 seconds
after: 37.073753 seconds (+55.44%) (+13.22 seconds)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 17:43:59 +0100] rev 50679
store: stop relying on a `revlog_type` property
We want to know if a file is related to a revlog, but the rest is dealt with
differently already, so we simplify things further.
as a bonus, this cleanup This provides a small but noticeable speedup.
The number below use `hg perf::stream-locked-section` to measure the time spend
in the locked section of the streaming clone. Number are run on various
repository and compare different steps.:
1) the effect of this patchs,
2) the effect of the cleanup series,
2) current state compared to because large refactoring.
All benchmarks are run on linux with Python 3.10.7.
### Effect of this patch
# mercurial-2018-08-01-zstd-sparse-revlog
# benchmark.name = perf-stream-locked-section
before: 0.030246 seconds
after: 0.029274 seconds (-3.21%)
# pypy-2018-08-01-zstd-sparse-revlog
before: 0.545012 seconds
after: 0.520872 seconds (-4.43%)
# netbeans-2018-08-01-zstd-sparse-revlog
before: 2.719939 seconds
after: 2.626791 seconds (-3.42%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
before: 6.304179 seconds
after: 6.096700 seconds (-3.29%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
before: 14.142687 seconds
after: 13.640779 seconds (-3.55%)
### Effect of this series
A small but sizeable speedup
# mercurial-2018-08-01-zstd-sparse-revlog
before: 0.031122 seconds
after: 0.029274 seconds (-5.94%)
# pypy-2018-08-01-zstd-sparse-revlog
before: 0.589970 seconds
after: 0.520872 seconds (-11.71%)
# netbeans-2018-08-01-zstd-sparse-revlog
before: 2.980300 seconds
after: 2.626791 seconds (-11.86%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
before: 6.863204 seconds
after: 6.096700 seconds (-11.17%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
before: 14.921393 seconds
after: 13.640779 seconds (-8.58%)
### Current state compared to the pre-refactoring state
The refactoring introduced multiple string manipulation and dictionary creation
that seems to induce a signifiant slowdown
Slowdown
# mercurial-2018-08-01-zstd-sparse-revlog
6.4.3: 0.019459 seconds
after: 0.029274 seconds (+50.44%)
## pypy-2018-08-01-zstd-sparse-revlog
6.4.3: 0.290715 seconds
after: 0.520872 seconds (+79.17%)
# netbeans-2018-08-01-zstd-sparse-revlog
6.4.3: 1.403447 seconds
after: 2.626791 seconds (+87.17%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
6.4.3: 3.163549 seconds
after: 6.096700 seconds (+92.72%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
6.4.3: 6.702184 seconds
after: 13.640779 seconds (+103.53%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:38:13 +0100] rev 50678
store: directly pass the filesize in the `details` of revlog
The dictionary only contains 1 (or 0) entries, we can directly store that
information (or None).
Moving to a simpler argument passing result in a noticable speedup (because
Python)
The number below use `hg perf::stream-locked-section` to measure the time spend
in the locked section of the streaming clone. Number are run on various
repository.
### mercurial-2018-08-01-zstd-sparse-revlog
before: 0.031247 seconds
after: 0.030246 seconds (-3.20%)
### mozilla-central-2018-08-01-zstd-sparse-revlog
before: 6.718968 seconds
after: 6.304179 seconds (-6.17%)
### mozilla-try-2019-02-18-zstd-sparse-revlog
before: 14.631343 seconds
after: 14.142687 seconds (-3.34%)
### netbeans-2018-08-01-zstd-sparse-revlog
before: 2.895584 seconds
after: 2.719939 seconds (-6.07%)
### pypy-2018-08-01-zstd-sparse-revlog
before: 0.561843 seconds
after: 0.543034 seconds (-3.35%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:35:10 +0100] rev 50677
store: explicitly pass file_size when creating StoreFile
A small cleanup before large cleanup in the next patch.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:33:28 +0100] rev 50676
store: have the revlog determine which files are volatile itself
This is a first step toward simplifying the walk step.
Mathias De Mare <mathias.de_mare@nokia.com> [Wed, 08 Mar 2023 14:23:43 +0100] rev 50675
clonebundles: add support for inline (streaming) clonebundles
The idea behind inline clonebundles is to send them through
the ssh or https connection to the Mercurial server.
We've been using this specifically for streaming clonebundles,
although it works for 'regular' clonebundles as well
(but is less relevant, since pullbundles exist).
We've had this enabled for around 9 months for a part
of our users.
A few benefits are:
- no need to secure an external system,
since everything goes through the same Mercurial server
- easier scaling (in our case: no risk of inconsistencies
between multiple mercurial-server mirrors and nginx clonebundles hosts)
Remaining topics/questions right now:
- The inline clonebundles don't work for https yet.
This is because httppeer doesn't seem to support sending client
capabilities.
I didn't focus on that as my main goal was to get this working
for ssh.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 31 May 2023 18:08:56 +0100] rev 50674
tree-manifest: allow `debugupgraderepo` to run on tree manifest repo
There does not seems to be anything wrong with running the current logic on
them. So we remove the limitation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 31 May 2023 16:04:16 +0100] rev 50673
stream-clone: update debugcreatestreamclonebundle helps
People to stop using streamv1, so we should point them to alternative in the
place where people might find it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 25 May 2023 00:23:05 +0200] rev 50672
rewrite: simplify the `retained_extras` extra logic
First, we move the definition of value outside of the rebase extensions, as this
apply to all rebase-like operation and some live in other place (like evolve).
Second we make it a simple set, so that it is easy for an extension to add a new
value in it.
Third, we move the associated logic in core too. That make it easily available
to other extensions.
Fourth we simplify it usage, as the verbose version of the filtering is just a
handful on line long, we are just going to test all the value for updates, so
the Projection overlay is not bringing much here.
Note that, we make it a module level set, is a key is worth preserving it is
probably worth preserving in all cases. This was already the behavior prior to
this change.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 18:41:58 +0200] rev 50671
stream-clone: smoothly detect and handle a case were a revlog is split
This detect and handle the most common case for a race condition around stream
and revlog splitting. The one were the revlog is split between the initial
collection of data and the time were we start considering stream that data.
In such case, we repatch an inlined version of that revlog together when this
happens. This is necessary as stream-v2 promised a specific number of bytes and
a specific number of files to the client. In stream-v3, we will have the
opportunity to just send a split revlog instead.
Getting a better version of the protocol for stream-v3 is still useful, but it
is no longer a blocket to fix that race condition.
Note that another, rarer race condition exist, were the revlog is split while
we creating the revlog and extracing content from it. This can be dealt with
later.