Mads Kiilerich <mads@kiilerich.com> [Wed, 22 Mar 2023 16:05:59 +0100] rev 50727
tests: update test-remotefilelog-gc.t for Python 3.11
The test output changed because test coverage changed because normpath changed:
$ python3.10 -c 'import os; print(repr(os.path.normpath("asdas\0das")))'
'asdas\x00das'
$ python3.11 -c 'import os; print(repr(os.path.normpath("asdas\0das")))'
'asdas'
Mads Kiilerich <mads@kiilerich.com> [Mon, 26 Jun 2023 14:54:00 +0200] rev 50726
tests: use grep -F instead of obsolescent fgrep
Testing on Fedora 38 failed with:
fgrep: warning: fgrep is obsolescent; using grep -F
The warning comes from
https://git.savannah.gnu.org/cgit/grep.git/commit/?id=
a9515624709865d480e3142fd959bccd1c9372d1
. For further anecdotal evidence of the change, see
https://www.phoronix.com/news/GNU-Grep-3.8-Stop-egrep-fgrep .
grep -F is POSIX, but there is a risk that it doesn't work the same on all
platforms - especially older Unix versions. It should however always be
possible to put a GNU grep in $PATH before running the tests.
Mads Kiilerich <mads@kiilerich.com> [Mon, 26 Jun 2023 14:34:58 +0200] rev 50725
tests: use grep -E instead of obsolescent egrep
Testing on Fedora 38 failed with:
egrep: warning: egrep is obsolescent; using grep -E
The warning comes from
https://git.savannah.gnu.org/cgit/grep.git/commit/?id=
a9515624709865d480e3142fd959bccd1c9372d1
. For further anecdotal evidence of the change, see
https://www.phoronix.com/news/GNU-Grep-3.8-Stop-egrep-fgrep .
This reverses the code check that goes back to
e7d3b509af8b. grep -E is POSIX,
but there is a risk that it doesn't work the same on all platforms - especially
older Unix versions. It should however always be possible to put a GNU grep in
$PATH before running the tests.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Mon, 26 Jun 2023 11:15:30 +0100] rev 50724
dirstate: avoid leaking disk space in `hg debugrebuilddirstate`
Before this MR running `hg debugrebuilddirstate` simply grows the dirstate
without bound, never shrinking it, because the unused bytes counter stays low,
even though the entirety of the file becomes unused.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Mon, 26 Jun 2023 11:21:43 +0100] rev 50723
dirstate-v2: actually fix the dirstate-v2 upgrade race
It looks like the previous fix for the dirstate-v2 upgrade race didn't work.
The problem is that it only recovers in case the size of the v1 `dirstate` file
is smaller than the `v2` one, whereas in real life it's always larger.
This commit changes the test to be more realistic, which reveals the crash,
and changes the code to fix the crash.
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 14:24:45 +0200] rev 50722
Added signature for changeset
bb42988c7e15
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 14:24:21 +0200] rev 50721
Added tag 6.5rc0 for changeset
bb42988c7e15
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 12:36:49 +0200] rev 50720
relnotes: add 6.5rc0
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 11:36:37 +0200] rev 50719
branching: merge default into stable for 6.5rc0
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 11:28:17 +0200] rev 50718
branching: merge stable into default
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 11:18:47 +0200] rev 50717
Added signature for changeset
271a4ab29605
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 11:18:27 +0200] rev 50716
Added tag 6.4.5 for changeset
271a4ab29605
Raphaël Gomès <rgomes@octobus.net> [Thu, 22 Jun 2023 11:17:40 +0200] rev 50715
relnotes: add 6.4.5
Jason R. Coombs <jaraco@jaraco.com> [Wed, 21 Jun 2023 14:00:50 -0400] rev 50714
narrow: indicated the default of 'Yes' when confirming auto-remove-includes
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Jun 2023 15:55:07 +0200] rev 50713
clone-bundles: check we generate a valid specification
The short bundle-spec are ambiguous and should be expanded to a full one before
clone bundle can use it. We fix the issue and add a test actually using the
generated manifest for a clone bundle operation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 21 Jun 2023 15:53:33 +0200] rev 50712
bundle-spec: add an official method to format a spec into a string
This will be useful for automatic-clonebundles.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Jun 2023 18:15:35 +0200] rev 50711
clonebundles: adds a auto-generate.serve-inline option
This new option disable the "external" serving of auto generated bundle in favor
of using the new "inline bundle" feature that allow serving such bundle through
the mercurial protocol.
This is a less efficient, but very easy to setup way to start using clone bundle.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Jun 2023 18:15:27 +0200] rev 50710
clone-bundle: only add the REQUIRESNI bit for http(s)
we don't needs it for `peer-bundle-cache://` for example
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 23 May 2023 10:24:57 +0200] rev 50709
stream-clone: add a stream-v3 variant of test-persistent-nodemap.t
This will be useful to track and validate future changes to the stream-v3
protocol.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 23 May 2023 10:18:01 +0200] rev 50708
stream-clone: split the stream part out of test-persistent-nodemap.t
We want to test multiple variants, so let's make it an independent test.
Small adaptations and simplifications are needed in the test output as the
tested repository is now shorter.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 23 May 2023 03:09:46 +0200] rev 50707
stream-clone: explicitly split debug test output for application per version
Altering the format may introduce significant changes in output and in order.
It is going to be simpler to deal with explicitly different blocks for the
output.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 01 Jun 2023 17:39:22 +0100] rev 50706
stream-clone: add a v3 version of the protocol
This new version is less rigid regarding the extract number of files and number
of bytes to be actually transfered, it also lays the groundwork for other
improvements.
The format stays experimental, but this is an interesting base to build upon.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 01 Jun 2023 18:20:28 +0100] rev 50705
stream-clone: fix the help for an argument of perf::stream-locked-section
The help for `--stream-version` had a typo and missed a value.
pierre-yves.david@ens-lyon.org [Tue, 20 Jun 2023 02:35:30 +0200] rev 50704
infinitepush: aggressively deprecated infinite push
We add a large warning and an explicit abort. Both can be disable through
config, but neither are explicitly documented. If any user exists I really want
them to get in touch with the community.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Jun 2023 16:13:16 +0200] rev 50703
clone-bundle: double check that inline clone bundle does not break local case
Since there is a large NotImplementedError in the code, I prefer we have a
double check here.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Jun 2023 15:48:03 +0200] rev 50702
clone-bundle: drop the now unused `inlineclonebundles` capabilities
We no longer us the capabilities as a side way to passe argument to the
`clonebundle commands`
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Jun 2023 15:49:35 +0200] rev 50701
clonebundles: introduce a new write protocol command
The old wire command does not take argument so we need a new command. We make
that new command take argument to prepare it for the future (like narrow
support).
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Jun 2023 11:28:45 +0200] rev 50700
clone-bundles: reorder some test section and add titles
This make things clearer around the inline clone bundle tests.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Jun 2023 11:04:27 +0200] rev 50699
clone-bundle: rename the methods and wireprotole command
The current naming is very clone centric, since then, the url scheme was moved
to a more generic wording to make room for other use-cases (e.g. pull bundles).
We apply the same logic to the internal API and, very important prior to the
release, to the wire protocol command.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 18 Jun 2023 00:04:53 +0200] rev 50698
delta-find: remove dead code intended to deal with forced delta reuse
Since the case was dealt with sooner (see XXX), we no longer need to deal with
it in this part of the code.
Jason R. Coombs <jaraco@jaraco.com> [Thu, 20 Apr 2023 15:56:58 -0400] rev 50697
library: incorporate demandimport into runpy invocation
Jason R. Coombs <jaraco@jaraco.com> [Mon, 17 Apr 2023 09:38:52 -0400] rev 50696
library: enable runpy invocation on mercurial package
Raphaël Gomès <rgomes@octobus.net> [Mon, 12 Jun 2023 16:51:08 +0200] rev 50695
match: add `filepath:` pattern to match an exact filepath relative to the root
It's useful in certain automated workflows to make sure we recurse in
directories whose name conflicts with files in other revisions.
In addition it makes it possible to avoid building a potentially costly regex,
improving performance when the set of files to match explicitly is large.
The benchmark below are run in the following configuration :
# data-env-vars.name = mozilla-central-2018-08-01-zstd-sparse-revlog
# benchmark.name = files
# benchmark.variants.rev = tip
# benchmark.variants.files = all-list-filepath-sorted
# bin-env-vars.hg.flavor = no-rust
It also includes timings using the re2 engine (through the `google-re2` module)
to show how much can be saved by just using a better regexp engine.
Pattern time (seconds) time using re2
-----------------------------------------------------------
just "." 0.4 0.4
list of "filepath:…" 1.3 1.3
list of "path:…" 25.7 3.9
list of patterns 29.7 10.4
As you can see, Without re2, using "filepath:" instead of "path:" is a huge
win. With re2, it is still about three times faster to not have to build the
regex.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 17 Jun 2023 23:47:31 +0200] rev 50694
path: use the next `display` argument to deal with boolean
We have a generic mechanism that we can now use for the special case now.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Jun 2023 09:51:29 +0200] rev 50693
path: display proper user facing value for pulled-delta-reuse-policy
The integer was not what the user expect nor what could be provided as a config
value.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Jun 2023 09:50:46 +0200] rev 50692
paths: add an argument to format the suboption display
We will use it in the next function to the delta policy display. It could also
be use to deal with the other special case in the command code, but that is
unnecessary churn for stable so that part will go on default.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 09 Jun 2023 14:32:09 +0100] rev 50691
ui: fix crash in `hg paths` when pulled-delta-reuse-policy is in use
The crash is a result of [value] being an int, but being used
as a bytestring.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 09 Jun 2023 14:31:40 +0100] rev 50690
ui: add a test demonstrating a crash in `hg paths`
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 17 Jun 2023 04:05:53 +0200] rev 50689
re2: fix reporting of availability in `hg debuginstall`
We add and use an official API to check is re2 is available. This prevent the
bug previously in place were debuginstall was considering confusing `None`
(i.e. non-initialized) and `False` (i.e. unavailable).
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 17 Jun 2023 04:05:18 +0200] rev 50688
re: make _checkre2 a static method
There is no need to it to be a method, so we make it at least a static method to
make it easier to call.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 17 Jun 2023 04:02:25 +0200] rev 50687
re2: exit `_checkre2` early if calling it is useless
If we already know if re2 is supported there is no need to run the check again.
Stricly speaking we don't call it twice, but this seems a simple enough
safeguard to include in this series.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 18 Jun 2023 00:09:39 +0200] rev 50686
branching: merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 20 Jun 2023 02:36:52 +0200] rev 50685
backout: remove
db9b6ce803a9
That changeset lost its topic for unknown reason and ended up being picked by
some automation. So I backout it previous incarnation until it can get properly
reviewed.
pierre-yves.david@ens-lyon.org [Mon, 19 Jun 2023 23:27:18 +0200] rev 50684
infinitepush: aggressively deprecated infinite push
We add a large warning and an explicit abort. Both can be disable through
config, but neither are explicitly documented. If any user exists I really want
them to get in touch with the community.
Raphaël Gomès <rgomes@octobus.net> [Tue, 13 Jun 2023 14:35:36 +0200] rev 50683
rust-dependencies: switch from `users` to `whoami`
`users` is unmaintained, with the maintainer apparently MIA. `whoami` is a
popular crate that does simple wrapping of platform-specific calls.
This makes the overhead from using `blackbox` lower. It used to be up 10ms
on slower hardware for both calls, now down to <1ms which is always good.
Raphaël Gomès <rgomes@octobus.net> [Mon, 12 Jun 2023 23:41:28 +0200] rev 50682
rust-hg-core: move from `ouroboros` to `self_cell`
`ouroboros` has a fundamental soundness problem that, while not applicable
today, could become applicable given new compiler optimizations.¹
`self_cell` is a crate that accomplishes a lot of the same things that
`ouroboros` did while remaining sound (that is, unless a new soundness issue
is discovered) by not assuming as much about the memory layout of the program.
`self_cell` has been scrutinized heavily in the past few months by very
competent people, some from the compiler team and has shown no weaknesses
for a while, with a 1.0 stable release coming out a couple months ago.
Our internal API is exactly the same, this is just an implementation detail.
To reiterate, no actual soundness issue was found with our use of `ouroboros`,
but there might be evolutions of `rustc` (or even a future separate compiler)
that could generate unsound code.
[1] https://github.com/joshua-maros/ouroboros/issues/88
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 02 Jun 2023 15:12:05 +0200] rev 50681
debug-revlog-stats: make it use the new store entry API
This give use free tree manifest support in the process.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 02 Jun 2023 15:10:50 +0200] rev 50680
store: use the revlog type from revlog's constant instead of custom one
We already have a revlog type that everybody use. So we use that one instead.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 12 Jun 2023 23:24:41 +0200] rev 50679
repair: use `is_manifestlog` to recognise manifest revlog
This is simpler that comparing flag, and we want to remove these flags anyway.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 04:56:54 +0200] rev 50678
perf: add a perf::stream-consume
We know how long it take to generate, lets check how long it take to apply now.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 04:09:05 +0200] rev 50677
perf: add a perf::stream-generate command
This record the time we take to generate a bundle.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 12 Jun 2023 18:04:09 +0200] rev 50676
perf: add a new "context" argument to timer
This allow to simple setup/teardown outside of the timed section. Especially
using object that need context manager, like a temporary files.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 01:48:10 +0200] rev 50675
perf: add support for stream-v3 during benchmark
This is getting important as the v3 protocol will diverge from the v2 protocol.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jun 2023 01:43:48 +0200] rev 50674
perf: add a function to find a stream version generator
The logic is clearer and can be reused for other commands in the future.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 18 May 2023 19:23:59 +0100] rev 50673
treemanifest: make `updatecaches` update the nodemaps for all directories
Without this, if the cache for a nested directory is in a bad state,
it's very hard to repair it.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Wed, 31 May 2023 10:37:55 +0100] rev 50672
stream-clone: avoid opening a revlog in case we do not need it
Opening an revlog has a cost, especially if it is inline as we have to scan the
file and construct an index.
To prevent the associated slowdown, we just do a minimal scan to check that an
inline file is still inline, and simply stream the file without creating a
revlog when we can.
This provides a big boost compared to the previous changeset, even if the full
generation is still penalized by the initial gathering of information.
All benchmarks are run on linux with Python 3.10.7.
# benchmark.name = hg.exchange.stream.generate
# benchmark.variants.version = v2
### Compared to the previous changesets
We get a large win all across the board!
# mercurial-2018-08-01-zstd-sparse-revlog
before: 0.250694 seconds
after: 0.105986 seconds (-57.72%)
# pypy-2018-08-01-zstd-sparse-revlog
before: 3.885657 seconds
after: 1.709748 seconds (-56.00%)
# netbeans-2018-08-01-zstd-sparse-revlog
before: 16.679371 seconds
after: 7.687469 seconds (-53.91%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
before: 38.575482 seconds
after: 17.520316 seconds (-54.58%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
before: 81.160994 seconds
after: 37.073753 seconds (-54.32%)
### Compared to 6.4.3
We are still significantly slower than 6.4.3, the extra time is usually twice
slower than the extra time we observe on the locked section, which is a quite
interesting information.
Except for mercurial-central that is much faster. That discrepancy is not really
explained yet.
# mercurial-2018-08-01-zstd-sparse-revlog
6.4.3: 0.072560 seconds
after: 0.105986 seconds (+46.07%) (- 0.03 seconds)
# pypy-2018-08-01-zstd-sparse-revlog
6.4.3: 1.211193 seconds
after: 1.709748 seconds (+41.16%) (-0.45 seconds)
# netbeans-2018-08-01-zstd-sparse-revlog
6.4.3: 4.932843 seconds
after: 7.687469 seconds (+55.84%) (-2.75 seconds)
# mozilla-central-2018-08-01-zstd-sparse-revlog
6.4.3: 34.012226 seconds
after: 17.520316 seconds (-48.49%) (-16.49 seconds)
# mozilla-try-2019-02-18-zstd-sparse-revlog
6.4.3: 23.850555 seconds
after: 37.073753 seconds (+55.44%) (+13.22 seconds)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 17:43:59 +0100] rev 50671
store: stop relying on a `revlog_type` property
We want to know if a file is related to a revlog, but the rest is dealt with
differently already, so we simplify things further.
as a bonus, this cleanup This provides a small but noticeable speedup.
The number below use `hg perf::stream-locked-section` to measure the time spend
in the locked section of the streaming clone. Number are run on various
repository and compare different steps.:
1) the effect of this patchs,
2) the effect of the cleanup series,
2) current state compared to because large refactoring.
All benchmarks are run on linux with Python 3.10.7.
### Effect of this patch
# mercurial-2018-08-01-zstd-sparse-revlog
# benchmark.name = perf-stream-locked-section
before: 0.030246 seconds
after: 0.029274 seconds (-3.21%)
# pypy-2018-08-01-zstd-sparse-revlog
before: 0.545012 seconds
after: 0.520872 seconds (-4.43%)
# netbeans-2018-08-01-zstd-sparse-revlog
before: 2.719939 seconds
after: 2.626791 seconds (-3.42%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
before: 6.304179 seconds
after: 6.096700 seconds (-3.29%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
before: 14.142687 seconds
after: 13.640779 seconds (-3.55%)
### Effect of this series
A small but sizeable speedup
# mercurial-2018-08-01-zstd-sparse-revlog
before: 0.031122 seconds
after: 0.029274 seconds (-5.94%)
# pypy-2018-08-01-zstd-sparse-revlog
before: 0.589970 seconds
after: 0.520872 seconds (-11.71%)
# netbeans-2018-08-01-zstd-sparse-revlog
before: 2.980300 seconds
after: 2.626791 seconds (-11.86%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
before: 6.863204 seconds
after: 6.096700 seconds (-11.17%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
before: 14.921393 seconds
after: 13.640779 seconds (-8.58%)
### Current state compared to the pre-refactoring state
The refactoring introduced multiple string manipulation and dictionary creation
that seems to induce a signifiant slowdown
Slowdown
# mercurial-2018-08-01-zstd-sparse-revlog
6.4.3: 0.019459 seconds
after: 0.029274 seconds (+50.44%)
## pypy-2018-08-01-zstd-sparse-revlog
6.4.3: 0.290715 seconds
after: 0.520872 seconds (+79.17%)
# netbeans-2018-08-01-zstd-sparse-revlog
6.4.3: 1.403447 seconds
after: 2.626791 seconds (+87.17%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
6.4.3: 3.163549 seconds
after: 6.096700 seconds (+92.72%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
6.4.3: 6.702184 seconds
after: 13.640779 seconds (+103.53%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:38:13 +0100] rev 50670
store: directly pass the filesize in the `details` of revlog
The dictionary only contains 1 (or 0) entries, we can directly store that
information (or None).
Moving to a simpler argument passing result in a noticable speedup (because
Python)
The number below use `hg perf::stream-locked-section` to measure the time spend
in the locked section of the streaming clone. Number are run on various
repository.
### mercurial-2018-08-01-zstd-sparse-revlog
before: 0.031247 seconds
after: 0.030246 seconds (-3.20%)
### mozilla-central-2018-08-01-zstd-sparse-revlog
before: 6.718968 seconds
after: 6.304179 seconds (-6.17%)
### mozilla-try-2019-02-18-zstd-sparse-revlog
before: 14.631343 seconds
after: 14.142687 seconds (-3.34%)
### netbeans-2018-08-01-zstd-sparse-revlog
before: 2.895584 seconds
after: 2.719939 seconds (-6.07%)
### pypy-2018-08-01-zstd-sparse-revlog
before: 0.561843 seconds
after: 0.543034 seconds (-3.35%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:35:10 +0100] rev 50669
store: explicitly pass file_size when creating StoreFile
A small cleanup before large cleanup in the next patch.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:33:28 +0100] rev 50668
store: have the revlog determine which files are volatile itself
This is a first step toward simplifying the walk step.
Raphaël Gomès <rgomes@octobus.net> [Mon, 12 Jun 2023 10:50:00 +0200] rev 50667
test-dirstate-version-fallback: future-proof the test for a different default
Dirstate-v2 will become the default at some point, which would cause this
test to fail. Let's save someone else the headache later.
Mathias De Mare <mathias.de_mare@nokia.com> [Wed, 08 Mar 2023 14:23:43 +0100] rev 50666
clonebundles: add support for inline (streaming) clonebundles
The idea behind inline clonebundles is to send them through
the ssh or https connection to the Mercurial server.
We've been using this specifically for streaming clonebundles,
although it works for 'regular' clonebundles as well
(but is less relevant, since pullbundles exist).
We've had this enabled for around 9 months for a part
of our users.
A few benefits are:
- no need to secure an external system,
since everything goes through the same Mercurial server
- easier scaling (in our case: no risk of inconsistencies
between multiple mercurial-server mirrors and nginx clonebundles hosts)
Remaining topics/questions right now:
- The inline clonebundles don't work for https yet.
This is because httppeer doesn't seem to support sending client
capabilities.
I didn't focus on that as my main goal was to get this working
for ssh.
Raphaël Gomès <rgomes@octobus.net> [Thu, 08 Jun 2023 17:02:04 +0200] rev 50665
Added signature for changeset
da372c745e0f
Raphaël Gomès <rgomes@octobus.net> [Thu, 08 Jun 2023 17:02:00 +0200] rev 50664
Added tag 6.4.4 for changeset
da372c745e0f
Raphaël Gomès <rgomes@octobus.net> [Thu, 08 Jun 2023 17:01:29 +0200] rev 50663
relnotes: add 6.4.4
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 14:28:21 +0200] rev 50662
revlog: avoid possible collision between directory and temporary index
Since 6.4, we create a temporary index file to write the split data without
overwriting the inline version too early. However, the store encoding does not
prevent these new `.i.s` file to collide with a directory with the same name.
While the odds for such a collision to happens are fairly low, the collision
would prevent Mercurial from working.
The store encoding have a mitigation solution in place to prevent such
collisions from happening for `.i` and `.d` files, but not for other extensions.
We cannot update this encoding scheme to solve the issue since it would diverge
from older version of Mercurial.
Instead, we create an alternative directory tree dedicated to such files.
The use of the `.i` extension combined with store encoding will prevent
collisions there.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 11:08:19 +0200] rev 50661
revlog: move the computation of the split_index path in a property
This is about to become more complex, so we gather the logic in a single place.
Raphaël Gomès <rgomes@octobus.net> [Mon, 05 Jun 2023 16:43:27 +0200] rev 50660
rust-dirstate: fall back to v1 if reading v2 failed
This will help us not fail when a v1 dirstate is present on disk while a v2
was expected (which could happen with a racy/interrupted upgrade).
Raphaël Gomès <rgomes@octobus.net> [Mon, 05 Jun 2023 17:29:52 +0200] rev 50659
dirstate: add test showing dirstate version mismatch causes an error
We should fall back to trying dirstate v1 when v2 fails to read.
Raphaël Gomès <rgomes@octobus.net> [Mon, 05 Jun 2023 16:30:25 +0200] rev 50658
rust-dirstate: rename `has_dirstate_v2` to `use_dirstate_v2`
It is closer to the right semantics. I added a docstring to better explain
the reasonning. In the next patch(es), I will address the underlying issue
of finding the "wrong" version of the dirstate on disk.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Jun 2023 03:11:26 +0200] rev 50657
delta-find: fix pulled-delta-reuse-policy=forced behavior
The code that select delta still has too many oportunity to discard the delta
is has been forcibly asked to reuse. However is is fairly easy to use a
dedicated fastpath for this case. So we do so.
Cleaning other code that tries to enforce that policy will be done on default.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 03:49:44 +0200] rev 50656
delta-find: display more information about the search in some case
This will be useful to access the effect of the delta reuse policy.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 03:05:10 +0200] rev 50655
deltafind: issue debug information when we fast-path rivial case too
More debug options never hurts.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 03:11:51 +0200] rev 50654
delta-find: gather the condition to blindly use a full snapshot together
We are about to make the `if` body bigger, so having only one of them is simpler/
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 02:49:10 +0200] rev 50653
delta-find: initialize the debug information much sooner (when possible)
This help us to record debug information in alternative path.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 02:42:28 +0200] rev 50652
delta-find: fix `parents` round detection
We should compare integer with integer, instead of bytes (node).
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 02:35:03 +0200] rev 50651
delta-find: intrduce a `_one_dbg_data` method
This helps with the initialisation of the expected debug information.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 01:52:35 +0200] rev 50650
delta-find: move final debug processing in a `_dbg_process_data` method
This will help us making sure we can call the necessary debugging code in
alternative codepath.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 01:19:01 +0200] rev 50649
delta-fine: use the `_debug_search` attribute directly
By adjusting the value at creation time, we can avoid the need to use and
intermediate variable.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 01:11:47 +0200] rev 50648
delta-find: move the `gather_debug` logic in a property
This will make it simpler to get be reused in multiple places.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Jun 2023 01:07:00 +0200] rev 50647
tests: check `pulled-delta-reuse-policy=forced` without general delta
If general delta is not available, the amount of delta we can reuse at all will
be more limited. We check that application still work and does not corrupt the
repository.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 31 May 2023 18:08:56 +0100] rev 50646
tree-manifest: allow `debugupgraderepo` to run on tree manifest repo
There does not seems to be anything wrong with running the current logic on
them. So we remove the limitation.
Mathias De Mare <mathias.de_mare@nokia.com> [Tue, 25 Apr 2023 15:28:40 +0200] rev 50645
bundles: clarify streaming v2 bundle usage
This seems useful to clarify through the documentation
(we were still using debugcreatestreamclonebundle, even though
'hg bundle' is already an option...).
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 31 May 2023 16:04:16 +0100] rev 50644
stream-clone: update debugcreatestreamclonebundle helps
People to stop using streamv1, so we should point them to alternative in the
place where people might find it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 25 May 2023 00:23:05 +0200] rev 50643
rewrite: simplify the `retained_extras` extra logic
First, we move the definition of value outside of the rebase extensions, as this
apply to all rebase-like operation and some live in other place (like evolve).
Second we make it a simple set, so that it is easy for an extension to add a new
value in it.
Third, we move the associated logic in core too. That make it easily available
to other extensions.
Fourth we simplify it usage, as the verbose version of the filtering is just a
handful on line long, we are just going to test all the value for updates, so
the Projection overlay is not bringing much here.
Note that, we make it a module level set, is a key is worth preserving it is
probably worth preserving in all cases. This was already the behavior prior to
this change.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Mon, 05 Jun 2023 11:07:08 +0200] rev 50642
revlog: fix a bug in revlog splitting
Specifically, if the .i.s files are not added to the transaction,
then a follow-up modification of such a file records it into transaction
as non-empty, which is incorrect.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 18:41:58 +0200] rev 50641
stream-clone: smoothly detect and handle a case were a revlog is split
This detect and handle the most common case for a race condition around stream
and revlog splitting. The one were the revlog is split between the initial
collection of data and the time were we start considering stream that data.
In such case, we repatch an inlined version of that revlog together when this
happens. This is necessary as stream-v2 promised a specific number of bytes and
a specific number of files to the client. In stream-v3, we will have the
opportunity to just send a split revlog instead.
Getting a better version of the protocol for stream-v3 is still useful, but it
is no longer a blocket to fix that race condition.
Note that another, rarer race condition exist, were the revlog is split while
we creating the revlog and extracing content from it. This can be dealt with
later.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 14:07:58 +0200] rev 50640
stream-clone: implement decidated `get_streams` method for revlog
For revlog, we can do better using the maximum linkrev expected. This approach
open the way to dealing with a much larger set of non-trivial changes, like
splitting of inline revlogs.
We will actually tackle this issue in the next changesets (thanks to this one).
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 05:52:58 +0200] rev 50639
stream-clone: make it the responsability of the store entry to stream content
The store entry has more context, this will especially be true when it comes to
revlogs. So we move the details of how to retrieve binary content to the
StoreEntry. The stream clone code now focus on the protocol bits.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 11:42:16 +0200] rev 50638
store: declare a `files` method on BaseStoreEntry
This will help pytype to type check. We have to move `StoreFile` earlier in the
file to use it in the type declaration.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 05:23:46 +0200] rev 50637
revlog: add a `get_revlog` method
This might seen weird, but I actually thing we have been needing this for a
long time. There is multiple object that kind of pretend being revlogs while
actually wrapping the actual revlog. Since multiple code needs to access the
actuel revlog. See documentation for more details.
Expect cleanup of various places one the current series is done.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:26:39 +0200] rev 50636
stream-clone: drop the _emit_v2 function
It has no user left.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:24:39 +0200] rev 50635
stream-clone: directly use `_entries_walk` to generate stream-v2
This does not requires that much changes and will give us much more
flexibility, like improving revlog handling to gracefully handle race
situation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:12:30 +0200] rev 50634
stream-clone: pre-indent some code
This make the next changeset clearer.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 04:12:10 +0200] rev 50633
local-clone: perform the hardlink/copy based from _entries_walk returns
We previously used `_v2_walk`. However it is not bringing us much. So lets use
the higher level function instead. This will offer us more flexibility with the
`_v2_walk` function… like deleting it eventually.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:24:29 +0200] rev 50632
store: cache the file_size when we get it from disk
The point of caching `files` is to ensure consistency and avoiding redoing
expensive work. So we cache the file_size once retrieved.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 03:46:48 +0200] rev 50631
store: cache the `files()` return for store entries
This make it more efficient to directly use the entries list to retrieve data
in various location. It also make the entry record the file size it previously
promissed to user code, especially the stream clone code.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 27 May 2023 04:22:18 +0200] rev 50630
stream-clone: introduce a richer TempCopyManager object
This replace the previous `copy` callable with a richer object that allow access
to the backup path. This will simplify the user code as they won't need to keep
and pass around the backup path explicitly.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 13:29:01 +0200] rev 50629
store: properly compute the targer_id of manifestlog in no-fncache walk
Creating RevlogStoreEntry is good, but we need to drop the final `00manifest`
part to create something correct.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 13:28:33 +0200] rev 50628
store: do not drop the final `/` when creating manifestlog instance
This bug, inherited from the upgrade code leads to the acces/creation of broken
revlog with name `DIRECTORY00manifest.i` instead of `DIRECTORY/00manifest.i`
We fix it in its own changeset to preserve the "pure code movement" aspect of
the previous changesets.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 27 May 2023 04:01:17 +0200] rev 50627
store: add a `get_revlog_instance` method on revlog entries
The upgrade code needs this a lot, and the stream code is about to needs it
too. So we start by moving the upgrade code in a more generic location.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 02:22:20 +0200] rev 50626
stream-clone: add a test that highlight crash on revlog splitting
This has been a long running problem, we should have a tests for it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 01:38:59 +0200] rev 50625
stream-clone: remove unused code in test-clone-stream.t
We are not using the extension we create inline, we are using
`tests/testlib/ext-stream-clone-steps.py`. So let us deleted the unused version.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 01:38:34 +0200] rev 50624
stream-clone: document the ext-stream-clone-steps.py utility extension
This extension is useful, let us clarify how to use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 12:15:10 +0200] rev 50623
test-treemanifest: cleanup the test to more easily show server side error
This made my life easier debugging.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 01 Jun 2023 22:32:21 +0100] rev 50622
test: make test-contrib-perf.t more robust
Some code path are far too fast now. We raise de number to make sure the run
keep to that floor.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:26:10 +0100] rev 50621
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:51 +0100] rev 50620
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 02:57:49 +0100] rev 50619
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 01:05:41 +0100] rev 50618
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 01:04:22 +0100] rev 50617
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 01:03:45 +0100] rev 50616
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:38:14 +0100] rev 50615
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:38:04 +0100] rev 50614
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:44 +0100] rev 50613
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:37 +0100] rev 50612
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:31 +0100] rev 50611
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:25 +0100] rev 50610
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:11 +0100] rev 50609
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:02 +0100] rev 50608
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.