Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 11:33:47 +0200] rev 51580
phases: use revision number in new_heads
All graph operations will be done using revision numbers, so passing nodes only
means they will eventually get converted to revision numbers internally.
As part of an effort to align the code on using revision number we make the
`phases.newheads` function operated on revision number, taking them as input
and using them in returns, instead of the node-id it used to consume and
produce.
This is part of multiple changesets effort to translate more part of the logic,
but is done step by step to facilitate the identification of issue that might
arise in mercurial core and extensions.
To make the change simpler to handle for third party extensions, we also rename
the function, using a more modern form. This will help detecting the different
between the node-id version and the rev-num version.
I also take this as an opportunity to add some comment about possible
performance improvement for the future. They don't matter too much now, but they
are worse exploring in a while.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 08 Apr 2024 15:11:49 +0200] rev 51579
phases: convert remote phase root to node while reading them
This is currently a bit silly as we will convert them back to node right after,
but that is an intermediate step before doing more disruptive changes.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 11:17:25 +0200] rev 51578
phases: more compact error handling in analyzeremotephases
using an intermediate variable result in more readable code, so let us use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 02:54:12 +0200] rev 51577
push: rework the computation of fallbackheads to be correct
The previous computation tried to be smart but ended up being wrong. This was
caught by phase movement test while reworking the phase discovery logic to be
faster.
The previous logic was failing to catch case where the pushed set was not based
on a common heads (i.e. when the discovery seemed to have "over discovered"
content, outside the pushed set)
In the following graph, `e` is a common head and we `hg push -r f`. We need to
detect `c` as a fallback heads and we previous failed to do so::
e
|
d f
|/
c
|
b
|
a
The performance impact of the change seems minimal. On the most impacted
repository at hand (mozilla-try), the slowdown seems mostly mixed in the
overall noise `hg push` but seems to be in the hundred of milliseconds order of
magnitude. When using rust, we seems to be a bit faster, probably because we
leverage more accelaratd internals.
I added a couple of performance related common for further investigation later
on.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 11:05:54 +0200] rev 51576
revset: stop serializing node when using "%ln"
Turning hundred of thousand of node from node to hex and back can be slow… what
about we stop doing it?
In many case were we are using node id we should be using revision id. However
this is not a good reason to have a stupidly slow implementation of "%ln".
This caught my attention again because the phase discovery during push make an
extensive use of "%ln" or huge set. In absolute, that phase discovery probably
should use "%ld" and need to improves its algorithmic complexity, but improving
"%ln" seems simple and long overdue. This greatly speeds up `hg push` on
repository with many drafts.
Here are some relevant poulpe benchmarks:
### data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
# benchmark.name = hg.command.push
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.explicit-rev = all-out-heads
# benchmark.variants.
issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = default
## benchmark.variants.revs = any-1-extra-rev
before: 44.235070
after: 20.416329 (-53.85%, -23.82)
## benchmark.variants.revs = any-100-extra-rev
before: 49.234697
after: 26.519829 (-46.14%, -22.71)
### benchmark.name = hg.command.bundle
# bin-env-vars.hg.flavor = default
# bin-env-vars.hg.py-re2-module = default
# benchmark.variants.revs = all
# benchmark.variants.type = none-streamv2
## data-env-vars.name = heptapod-public-2024-03-25-zstd-sparse-revlog
before: 10.138396
after: 7.750458 (-23.55%, -2.39)
## data-env-vars.name = mercurial-public-2024-03-22-zstd-sparse-revlog
before: 1.263859
after: 0.700229 (-44.60%, -0.56)
## data-env-vars.name = mozilla-try-2023-03-22-zstd-sparse-revlog
before: 399.484481
after: 346.5089 (-13.26%, -52.98)
## data-env-vars.name = pypy-2024-03-22-zstd-sparse-revlog
before: 4.540080
after: 3.401700 (-25.07%, -1.14)
## data-env-vars.name = tryton-public-2024-03-22-zstd-sparse-revlog
before: 2.975765
after: 1.870798 (-37.13%, -1.10)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 14:41:48 +0200] rev 51575
bundlespec: drop unused _bundlespecvariants dictionary
Why do we have a `_bundlespecvariants`?
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 14:37:24 +0200] rev 51574
bundlespec: type the _bundlespeccontentopts dictionary
If only we had a tool to detect the kind of stupid error we just fixed… ho wait.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 09 Apr 2024 14:36:01 +0200] rev 51573
bundlespec: fix the "streamv2" and "streamv3-exp" variant
In
c4aab3661f25, we broken this feature by adding unicode instead of bytes to
the dictionary.
On the other hand, this feature was never tested, so augment the tests to tests
this.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 04 Apr 2024 14:15:32 +0100] rev 51572
wireprotoserver: ensure that output stream gets flushed on exception
Previously flush was happening due to Python finalizer being run on
`BufferedWriter`. With upgrade to Python 3.11 this started randomly
failing.
My guess is that the finalizer on the raw `FileIO` object may
be running before the finalizer of `BufferedWriter` has a chance to run.
At any rate, since we're not relying on finalizers in the happy case
we should also not rely on them in case of exception.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Mon, 15 Apr 2024 16:33:37 +0100] rev 51571
match: strengthen visit_children_set invariant, Recursive means "all files"
My previous interpretation of "Recursive" was too relaxed: I thought it
instructed the caller to do something like this:
> you can stop calling `visit_children_set` because you'll need to descend into
> every directory recursively, but you should still check every file if it
> matches or not
Whereas the real instruction seems to be:
> I guarantee that everything in this subtree matches, you can stop
> querying the matcher for all files and dirs altogether.
The evidence to support this:
- the test actually passes with the stronger invariant, revealing no
exceptions from this rule
- the implementation of `visit_children_set` for `DifferenceMatcher`
clearly relies on this requirement, so it must hold for that not to
lead to bugs.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 16:09:45 +0100] rev 51570
match: fix the rust-side bug in visit_children_set for rootfilesin matchers
The fix is checked by `test_pattern_matcher_visit_children_set` test,
which is what caught the bug in the first place, but also by an end-to-end
test that I made for this purpose.
Accept the new results of Cargo tests
Many of these were already annotated with "FIXME", which is a good sign.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 15:39:21 +0100] rev 51569
match: fix the "visitdir" method on "rootfilesin" matchers
This fixes just the Python side, the fix for the rust side will follow shortly.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 14:21:14 +0100] rev 51568
match: rename RootFiles to RootFilesIn for more consistency
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 14:17:10 +0100] rev 51567
match: small tweak to PatternMatcher.visit_children_set
This makes it a bit more efficient (avoid a computation in case of early
return), and in my opinion clearer.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 14:09:55 +0100] rev 51566
matchers: fix the bug in rust PatternMatcher that made it cut off early
This brings the rust output in line with the Python output.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 12 Apr 2024 13:48:38 +0100] rev 51565
tests: add an end-to-end test to show a bug in `visit_children_set`
Concretely, `rootfilesin` is completely broken with respect to
`visit_children_set` optimization.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 11 Apr 2024 19:57:36 +0100] rev 51564
tests: add tests and document expectations from visit_children_set in rust
The tests this patch are adding have the form of formal spec in
invariants::visit_children_set::holds,
and then a series of checks that all examples must satisfy this
formal spec.
I tried to make the spec consistent with how this function is used
and how it was originally conceived. This is in conflict with how it's
documented in Rust. Some of the implementations also fail to implement
this spec, which leads to bugs, in particular when complicated patterns
are used with `hg status`.
Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 11 Apr 2024 15:53:23 +0100] rev 51563
tests: add a test that demonstrates a bug in rhg status pattern handling
The bug is in [visit_children_set], will be elaborated on in
follow-up changes.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 05 Apr 2024 01:07:46 +0200] rev 51562
bundle-spec: properly parse boolean configuration as boolean
Before this changesets "v2;revbranchcache=no" would actually request the
addition for a revbranchcache part as the non-empty string `"0"` is `True`
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 04 Apr 2024 16:41:43 +0200] rev 51561
bundle-spec: properly identify changegroup-less bundle
It is possible to produce a bundle without changegroup. For example if we want
to only send phases or obsolescence information. However that lead to crash for
command that identifies bundle content. So we fix that.
The test will come in the next changesets, when we fix another bug preventing to
generate such bundle by hand.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 03 Apr 2024 15:33:25 +0200] rev 51560
perf: create the temporary target next to the source in stream-consume
See inline comment for rational.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 03 Apr 2024 16:00:37 +0200] rev 51559
setup: display return code information about failed `hg` call
This help to understand what is going wrong when things goes wrong.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 02 Apr 2024 21:53:17 +0200] rev 51558
bundlespec: rationalize the way we specify stream bundle version
Instead of having weird dedicated option for each version (v2, v3, etc) we
reuse the same "stream" parameters. This is consistent with the ability to
request a stream clone using "none-v2;stream=v2".
This changeset introduce no user visible change, this is pure internal cleaning.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 02 Apr 2024 17:02:39 +0200] rev 51557
bundle: do no check the changegroup version if no changegroup is included
We don't need to check the compatibility of something we will not use.
In practice this was getting in the was of `streamv2` bundles on a narrow
repository as the 'cg.version=02' value was rejected by this checks.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 18:51:33 +0000] rev 51556
perf-stream-consume: use the source repository config when applying
This might contains critical configuration for the benchmark, like enabling of
extensions like narrow.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 17:46:23 +0000] rev 51555
unbundle: move most of the logic on cmdutil to help debug::unbundle reuse
This make sure `hg debug::unbundle` focus on the core logic.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 17:29:48 +0000] rev 51554
postincoming: move to cmdutil
This looks like a good place for it to live.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 27 Mar 2024 17:21:46 +0000] rev 51553
postincoming: avoid computing branchhead if no report will be posted
This otherwise defeat some of the branch v3 optimization.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 13:46:44 +0000] rev 51552
streamclone: stop listing files for entries that have no volatile files
This will save a lot of python related time.
This significantly boost performance. The following number comes from a large
private repository using perf::stream-locked-section:
base-line: 35.04 seconds
prev-change: 24.51 seconds (-30%)
prev-change: 20.88 seconds (-40%)
prev-change: 14.22 seconds (-60%)
this-change: 11.58 seconds (-67% from baseline; -18% from prev)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 26 Mar 2024 13:34:05 +0000] rev 51551
stream-clone: disable gc for the initial section for the v3 format
The number of small container created turn Python in a gc-frenzy that seriously
impact performance.
This significantly boost performance. The following number comes from a large
private repository using perf::stream-locked-section:
base-line: 35.04 seconds
prev-change: 24.51 seconds (-30%)
prev-change: 20.88 seconds (-40%)
this-change: 14.22 seconds (-60% from baseline; -31% from prev)