Martin von Zweigbergk <martinvonz@google.com> [Tue, 11 Sep 2018 10:50:46 -0700] rev 39546
narrow: move wire proto capabilities to narrowwirepeer
These are not bundle2 capabilities (they just happened to share the
name "narrow"), so they seem to belong with the wirepeer overrides.
Differential Revision: https://phab.mercurial-scm.org/D4528
Martin von Zweigbergk <martinvonz@google.com> [Tue, 11 Sep 2018 11:03:15 -0700] rev 39545
narrow: check "narrow" wire protocol capability, not bundle2 capability
It seems like the new "narrow" wire protocol capability should be what
determines if the server supports the "narrow" and
"{,old}{in,ex}cludepats" arguments to the getbundle request.
Differential Revision: https://phab.mercurial-scm.org/D4527
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 21 Aug 2018 16:11:17 +0300] rev 39544
sparse: add local files to temporaryfiles if they exist out of sparse
We get the f1 from args if it's merge and check that whether that exists in
sparse checkout or not. If that does not, we add that for merging.
The error comes from very low-level where we try to read data of a
working-filectx which does not exists in the working directory. It will be
extremely ugly to plug in logic to update sparse copy with new file at such a
low level.
We already have logic related to updating the checkout with required files in
calculateupdates() and let's handle this case there only. calculateupdates()
call sparse.filterupdatesactions() and the logic is added into the latter
function.
To get the exact traceback, this patch can be backed out and
test-sparse-merges.t can be run with ui.traceback=True.
Differential Revision: https://phab.mercurial-scm.org/D4341
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 21 Aug 2018 15:09:22 +0300] rev 39543
tests: show that merging with sparse is broken when rename is involved
This patch adds test to show that merging with sparse is broken when you have a
rename on one side and just modification on another side.
Differential Revision: https://phab.mercurial-scm.org/D4340
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 25 Aug 2018 22:19:42 +0300] rev 39542
narrowspec: fix a typoed 'supported'
Spotted by martinvonz.
Differential Revision: https://phab.mercurial-scm.org/D4374
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 10 Sep 2018 16:29:22 +0300] rev 39541
narrow: build the known set of nodes only when ellipses is enabled
We don't need to build the known set in non-ellipses case because we don't have
a shallow repo. In this patch, this checks whether the server has ellipses
enabled or not using the server capability and then build the known set of
nodes.
Building the known set of nodes can take ~3-4 minutes on repositories with
millions of csets so this patch speeds up extending a non-shallow narrow clone
on large repositories.
In future, we should first check whether local repository is an ellipses repo
using a new ellipses repo requirement and then control all the combinations
between local repo requirement and server capability.
Differential Revision: https://phab.mercurial-scm.org/D4520
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 10 Sep 2018 15:55:14 +0300] rev 39540
narrow: add narrow and ellipses as server capabilities
Right now we don't have a way to differentiate between whether a server can
serve ellipsis or not. The way we check whether a server is narrow enabled is by
checking bundle2 capability which does not scale outside of bundle2 world.
The goal is to use have wireprotocol commands just like remotefilelog for
widening the narrow clone, atleast in non-ellipses cases. Having a server
capability will help there as we can't rely on bundle2 capability there.
Also having a server capability is neat than having a bundle2 capability.
There are lot of things we can optimize locally on the client side by knowing
before hand that whether the server supports ellipses or not.
This will also help us in making sure that a client ellipses repo does not
communicate with a server repo without ellipses.
Differential Revision: https://phab.mercurial-scm.org/D4521
Kyle Lippincott <spectral@google.com> [Thu, 23 Aug 2018 18:26:14 -0700] rev 39539
treemanifest: use visitchildrenset when doing a walk
For this series, starting at 'introduce lazy loading of subdirs' and ending with
this commit, we get the following timing numbers, using roughly the same
methodology and setup that we did in an earlier commit.
"before" is 6268fed3.
diff --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.585 s +- 0.034 s | 1.574 s +- 0.045 s | 99.3%
m-u | | x | 1.591 s +- 0.024 s | 1.601 s +- 0.034 s | 100.6%
m-u | x | | 1.579 s +- 0.032 s | 1.603 s +- 0.029 s | 101.5%
m-u | x | x | 109.8 ms +- 2.1 ms | 108.8 ms +- 2.2 ms | 99.1%
l-d-r | | | 234.6 ms +- 5.1 ms | 240.1 ms +- 7.9 ms | 102.3%
l-d-r | | x | 238.3 ms +- 8.1 ms | 232.2 ms +- 6.3 ms | 97.4%
l-d-r | x | | 120.0 ms +- 2.7 ms | 119.6 ms +- 2.1 ms | 99.7%
l-d-r | x | x | 120.2 ms +- 1.6 ms | 119.1 ms +- 1.6 ms | 99.1%
diff -c . --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 356.9 ms +- 5.6 ms | 355.9 ms +- 5.6 ms | 99.7%
m-u | | x | 209.1 ms +- 3.3 ms | 205.8 ms +- 2.6 ms | 98.4%
m-u | x | | 418.3 ms +- 14.0 ms | 350.8 ms +- 7.7 ms | 83.9% <--
m-u | x | x | 168.3 ms +- 1.5 ms | 168.1 ms +- 2.7 ms | 99.9%
l-d-r | | | 100.5 ms +- 2.1 ms | 99.1 ms +- 1.6 ms | 98.6%
l-d-r | | x | 5.672 s +- 0.133 s | 4.335 s +- 0.051 s | 76.4% <--
l-d-r | x | | 102.8 ms +- 2.8 ms | 100.4 ms +- 1.7 ms | 97.7%
l-d-r | x | x | 1.025 s +- 0.028 s | 875.9 ms +- 12.1 ms | 85.5% <--
rebase -r . --keep -d .^^:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 6.654 s +- 0.137 s | 6.718 s +- 0.046 s | 101.0%
m-u | | x | 6.672 s +- 0.104 s | 6.716 s +- 0.143 s | 100.7%
m-u | x | | 6.661 s +- 0.072 s | 6.658 s +- 0.066 s | 100.0%
m-u | x | x | 696.0 ms +- 49.0 ms | 696.2 ms +- 54.8 ms | 100.0%
l-d-r | | | 789.1 ms +- 12.5 ms | 793.5 ms +- 17.3 ms | 100.6%
l-d-r | | x | 7.621 s +- 0.194 s | 7.144 s +- 0.116 s | 93.7% <--
l-d-r | x | | 335.4 ms +- 9.7 ms | 339.6 ms +- 10.8 ms | 101.3%
l-d-r | x | x | 6.315 s +- 0.085 s | 5.633 s +- 0.105 s | 89.2% <--
status --change . --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 331.2 ms +- 7.0 ms | 335.1 ms +- 6.5 ms | 101.2%
m-u | | x | 184.0 ms +- 3.8 ms | 183.1 ms +- 3.4 ms | 99.5%
m-u | x | | 330.5 ms +- 7.2 ms | 332.7 ms +- 6.9 ms | 100.7%
m-u | x | x | 148.7 ms +- 3.5 ms | 147.5 ms +- 2.2 ms | 99.2%
l-d-r | | | 97.3 ms +- 1.3 ms | 98.2 ms +- 2.5 ms | 100.9%
l-d-r | | x | 5.196 s +- 0.030 s | 4.281 s +- 0.025 s | 82.4% <--
l-d-r | x | | 101.4 ms +- 3.1 ms | 100.9 ms +- 1.8 ms | 99.5%
l-d-r | x | x | 1.015 s +- 0.018 s | 861.2 ms +- 11.2 ms | 84.8% <--
status --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 2.356 s +- 0.032 s | 2.369 s +- 0.027 s | 100.6%
m-u | | x | 2.371 s +- 0.041 s | 2.385 s +- 0.028 s | 100.6%
m-u | x | | 2.366 s +- 0.036 s | 2.379 s +- 0.030 s | 100.5%
m-u | x | x | 121.8 ms +- 3.8 ms | 121.1 ms +- 2.7 ms | 99.4%
l-d-r | | | 726.2 ms +- 12.9 ms | 723.1 ms +- 12.7 ms | 99.6%
l-d-r | | x | 738.7 ms +- 11.2 ms | 741.3 ms +- 11.1 ms | 100.4%
l-d-r | x | | 215.9 ms +- 3.6 ms | 214.7 ms +- 5.9 ms | 99.4%
l-d-r | x | x | 213.1 ms +- 2.5 ms | 212.8 ms +- 4.5 ms | 99.9%
update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 3.922 s +- 0.045 s | 3.937 s +- 0.034 s | 100.4%
m-u | | x | 3.673 s +- 0.028 s | 3.660 s +- 0.035 s | 99.6%
m-u | x | | 3.959 s +- 0.073 s | 3.904 s +- 0.051 s | 98.6%
m-u | x | x | 417.0 ms +- 13.0 ms | 406.2 ms +- 6.5 ms | 97.4%
l-d-r | | | 534.3 ms +- 12.2 ms | 537.9 ms +- 8.8 ms | 100.7%
l-d-r | | x | 10.671 s +- 0.275 s | 10.111 s +- 0.114 s | 94.8% <--
l-d-r | x | | 309.7 ms +- 3.5 ms | 307.9 ms +- 4.0 ms | 99.4%
l-d-r | x | x | 1.837 s +- 0.014 s | 1.887 s +- 0.035 s | 102.7%
Differential Revision: https://phab.mercurial-scm.org/D4371
Kyle Lippincott <spectral@google.com> [Thu, 23 Aug 2018 00:44:10 -0700] rev 39538
treemanifest: use visitchildrenset when filtering a manifest to a matcher
Differential Revision: https://phab.mercurial-scm.org/D4370
Matt Harbison <matt_harbison@yahoo.com> [Mon, 10 Sep 2018 21:15:54 -0400] rev 39537
tests: stabilize test-no-symlink
This goes with 89630d0b3e23.
Boris Feld <boris.feld@octobus.net> [Tue, 29 May 2018 12:12:18 +0200] rev 39536
shelve: use the internal phase when possible
If the repository support it, use the internal phase for all changesets
created by shelve.
Kyle Lippincott <spectral@google.com> [Thu, 23 Aug 2018 00:41:20 -0700] rev 39535
treemanifest: avoid loading everything just to get their nodeid
Differential Revision: https://phab.mercurial-scm.org/D4369
Kyle Lippincott <spectral@google.com> [Thu, 23 Aug 2018 00:34:25 -0700] rev 39534
treemanifest: avoid unnecessary copies/processing when using alwaysmatcher
Differential Revision: https://phab.mercurial-scm.org/D4368
Kyle Lippincott <spectral@google.com> [Thu, 23 Aug 2018 00:33:06 -0700] rev 39533
treemanifest: attempt to avoid loading all lazily-loaded subdirs in _isempty
Differential Revision: https://phab.mercurial-scm.org/D4367
spectral <spectral@google.com> [Thu, 16 Aug 2018 12:31:52 -0700] rev 39532
treemanifest: introduce lazy loading of subdirs
An earlier patch series made it so that what to load was up to the calling code,
which works fine until manifests are copied - when they're copied, they're
loaded completely and thus we lose the entire benefit.
By lazy loading everything, we can avoid having to pass in the matcher to ~every
manifest function, and handle copies correctly as well. This changeset doesn't
go as far as it could with loading only the necessary subsets, that will happen
in later changes in this series; at the moment, except in a few situations, we
just load everything the moment we want to interact with treemanifest._dirs.
This is thus most likely to be a small slowdown if treemanifests is in use
regardless of whether narrow is in use, but hopefully easier to verify
correctness and review.
This is part of a series of speedups, it is not expected to produce any real speed
improvements itself, but the numbers show that it doesn't produce a large speed
penalty in any common case, and for the cases it does provide a penalty in, it
is not a large absolute amount (even if it is a large percentage amount).
Timing numbers according to command:
hyperfine --prepare <preparation_script> 'hg status'
HGRCPATH points to a file with the following contents:
[extensions]
narrow =
strip =
rebase =
mozilla-unified (called m-u below) was at revision #468856.
regular hash: eb39298e432d
treemanifests hash: 0553b7f29eaf
large-dir-repo (called l-d-r below) was generated with the following script:
#!/bin/bash
hg init large-dir-repo
mkdir -p large-dir-repo/third_party/rust/log
touch large-dir-repo/third_party/rust/log/foo.txt
for i in $(seq 1 30000); do
d=$(mktemp -d large-dir-repo/third_party/XXXXXXXXX)
touch $d/file.txt
done
hg -R large-dir-repo ci -Am 'rev0' --user test --date '0 0'
echo hi > large-dir-repo/third_party/rust/log/bar.txt
hg -R large-dir-repo ci -Am 'rev1' --user test --date '0 0'
echo hi > large-dir-repo/third_party/rust/log/baz.txt
hg -R large-dir-repo ci -Am 'rev2' --user test --date '0 0'
for the repos that use narrow, the narrowspec was this:
[include]
rootfilesin:accessible/jsat
rootfilesin:accessible/tests/mochitest/jsat
rootfilesin:mobile/android/chrome/content
rootfilesin:mobile/android/modules/geckoview
rootfilesin:third_party/rust/log
[exclude]
This narrowspec was chosen due to the size of the third_party/rust directory
(this directory was *not* modified in revision #468856 in mozilla-unified),
plus all the directories that *were* modified in revision #468856 of
mozilla-unified.
Importantly, when using narrow, these repos had everything checked out (in the
case of large-dir-repo, that means all 30,001 directories), *before* adding the
narrowspec. This is to simulate the behavior when using a virtual filesystem
that shows everything for the user even if they haven't added it to the
narrowspec yet. This is not a supported configuration, and `hg update` and `hg
rebase` will not really do the "correct" thing if there are mutations outside
of the narrowspec (which is not the case in these tests, due to a carefully
crafted narrowspec), but non-mutating commands should behave correctly.
I'm not claiming anything less than a 5% speed win as improvements due to this
change; these are probably eiter measurement artifacts or constant time
improvements. The numbers that aren't changing are shown primarily to prove that
this doesn't make anything worse in any case I plan on testing during this
series.
'before' is hg from commit 6268fed3
'N' indicates narrow in use
'T' indicates treemanifest in use
Please note that these commands and the narrowspec are a little different than
the ones in a similar table that I made in a3cabe9415e1.
Important: it is my understanding that these numbers below are *not super reliable*,
the large slowdowns may be artifacts of some odd interaction between GC and
python module/code complexity. Another changeset of mine (D4351) had shown large
timing differences when ~empty, uncalled functions were added to match.py,
though only when using --color=never or redirecting to /dev/null. We seem to be
on some cusp of complexity or code size that is causing, at my best guess
(according to linux `perf` benchmarks) GC to alter behavior and cause a
200-400ms difference in timings. I haven't had a chance to replicate these
results on another machine.
diff --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.580 s +- 0.034 s | 1.576 s +- 0.022 s | 99.7%
m-u | | x | 1.568 s +- 0.025 s | 1.584 s +- 0.044 s | 101.0%
m-u | x | | 1.569 s +- 0.031 s | 1.554 s +- 0.025 s | 99.0%
m-u | x | x | 107.3 ms +- 1.6 ms | 106.3 ms +- 1.5 ms | 99.1%
l-d-r | | | 232.5 ms +- 5.9 ms | 233.5 ms +- 5.3 ms | 100.4%
l-d-r | | x | 236.6 ms +- 6.3 ms | 233.6 ms +- 7.0 ms | 98.7%
l-d-r | x | | 118.4 ms +- 2.1 ms | 118.4 ms +- 1.4 ms | 100.0%
l-d-r | x | x | 116.8 ms +- 1.5 ms | 118.9 ms +- 1.6 ms | 101.8%
diff -c . --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 354.4 ms +- 16.6 ms | 351.0 ms +- 6.9 ms | 99.0%
m-u | | x | 207.2 ms +- 3.0 ms | 206.2 ms +- 2.7 ms | 99.5%
m-u | x | | 422.0 ms +- 26.0 ms | 351.2 ms +- 6.4 ms | 83.2% <--
m-u | x | x | 166.7 ms +- 2.1 ms | 169.5 ms +- 4.1 ms | 101.7%
l-d-r | | | 98.4 ms +- 4.5 ms | 98.5 ms +- 2.1 ms | 100.1%
l-d-r | | x | 5.519 s +- 0.060 s | 5.149 s +- 0.042 s | 93.3% <--
l-d-r | x | | 99.1 ms +- 3.2 ms | 102.6 ms +- 9.7 ms | 103.5% <--?
l-d-r | x | x | 994.9 ms +- 10.7 ms | 1.026 s +- 0.012 s | 103.1% <--?
rebase -r . --keep -d .^^:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 6.639 s +- 0.168 s | 6.559 s +- 0.097 s | 98.8%
m-u | | x | 6.601 s +- 0.143 s | 6.640 s +- 0.207 s | 100.6%
m-u | x | | 6.582 s +- 0.098 s | 6.543 s +- 0.098 s | 99.4%
m-u | x | x | 678.4 ms +- 57.7 ms | 703.7 ms +- 52.4 ms | 103.7% <--?
l-d-r | | | 780.0 ms +- 23.9 ms | 776.0 ms +- 12.6 ms | 99.5%
l-d-r | | x | 7.520 s +- 0.255 s | 7.395 s +- 0.044 s | 98.3%
l-d-r | x | | 331.9 ms +- 16.5 ms | 327.0 ms +- 3.4 ms | 98.5%
l-d-r | x | x | 6.228 s +- 0.113 s | 5.924 s +- 0.044 s | 95.1%
status --change . --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 330.8 ms +- 7.2 ms | 329.0 ms +- 7.1 ms | 99.5%
m-u | | x | 182.9 ms +- 2.7 ms | 183.5 ms +- 2.7 ms | 100.3%
m-u | x | | 330.0 ms +- 7.6 ms | 327.1 ms +- 5.4 ms | 99.1%
m-u | x | x | 146.2 ms +- 2.4 ms | 147.1 ms +- 1.3 ms | 100.6%
l-d-r | | | 95.3 ms +- 1.4 ms | 95.9 ms +- 1.5 ms | 100.6%
l-d-r | | x | 5.157 s +- 0.035 s | 5.166 s +- 0.058 s | 100.2%
l-d-r | x | | 99.7 ms +- 3.0 ms | 100.2 ms +- 4.4 ms | 100.5%
l-d-r | x | x | 993.6 ms +- 13.1 ms | 1.025 s +- 0.015 s | 103.2% <--?
status --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 2.348 s +- 0.031 s | 2.329 s +- 0.019 s | 99.2%
m-u | | x | 2.337 s +- 0.026 s | 2.346 s +- 0.034 s | 100.4%
m-u | x | | 2.354 s +- 0.015 s | 2.342 s +- 0.021 s | 99.5%
m-u | x | x | 120.6 ms +- 4.3 ms | 119.2 ms +- 2.1 ms | 98.8%
l-d-r | | | 731.5 ms +- 11.1 ms | 719.6 ms +- 9.8 ms | 98.4%
l-d-r | | x | 729.0 ms +- 15.5 ms | 725.7 ms +- 10.6 ms | 99.5%
l-d-r | x | | 211.0 ms +- 3.9 ms | 212.8 ms +- 3.7 ms | 100.9%
l-d-r | x | x | 211.5 ms +- 4.2 ms | 211.0 ms +- 3.3 ms | 99.8%
update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 3.910 s +- 0.055 s | 3.920 s +- 0.075 s | 100.3%
m-u | | x | 3.613 s +- 0.056 s | 3.630 s +- 0.056 s | 100.5%
m-u | x | | 3.873 s +- 0.055 s | 3.864 s +- 0.049 s | 99.8%
m-u | x | x | 400.4 ms +- 7.4 ms | 403.6 ms +- 5.0 ms | 100.8%
l-d-r | | | 531.6 ms +- 10.0 ms | 528.8 ms +- 9.6 ms | 99.5%
l-d-r | | x | 10.377 s +- 0.049 s | 9.955 s +- 0.046 s | 95.9%
l-d-r | x | | 308.3 ms +- 4.4 ms | 306.8 ms +- 3.7 ms | 99.5%
l-d-r | x | x | 1.805 s +- 0.015 s | 1.834 s +- 0.020 s | 101.6%
Differential Revision: https://phab.mercurial-scm.org/D4366
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 17:54:55 -0400] rev 39531
contrib: use a monotonic timer in catapipe
As spotted by Gregory, we should use a monotonic timer to get better timings.
Differential Revision: https://phab.mercurial-scm.org/D4516
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 17:51:07 -0400] rev 39530
contrib: fix catapipe output argument documentation
Differential Revision: https://phab.mercurial-scm.org/D4515
Boris Feld <boris.feld@octobus.net> [Thu, 06 Sep 2018 16:59:25 -0400] rev 39529
tracing: trace command function execution
Differential Revision: https://phab.mercurial-scm.org/D4514
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 16:51:51 -0400] rev 39528
extension: add a summary of total loading time per extension
Differential Revision: https://phab.mercurial-scm.org/D4513
Boris Feld <boris.feld@octobus.net> [Thu, 06 Sep 2018 16:49:37 -0400] rev 39527
extensions: trace the total time of running all reposetup callbacks
Differential Revision: https://phab.mercurial-scm.org/D4512
Boris Feld <boris.feld@octobus.net> [Thu, 06 Sep 2018 16:21:42 -0400] rev 39526
extensions: trace the total time of running all extsetup callbacks
Differential Revision: https://phab.mercurial-scm.org/D4511
Boris Feld <boris.feld@octobus.net> [Thu, 06 Sep 2018 16:20:05 -0400] rev 39525
extensions: trace the total time of running all uisetup callbacks
Differential Revision: https://phab.mercurial-scm.org/D4510
Boris Feld <boris.feld@octobus.net> [Mon, 10 Sep 2018 19:36:25 -0700] rev 39524
extensions: add timing for extensions reposetup
Differential Revision: https://phab.mercurial-scm.org/D4509
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:18:45 -0400] rev 39523
sparse-revlog: set max delta chain length to on thousand
The new snapshot system used in the sparse-revlog case gave us some small size
benefit so far. However its most important property is to gracefully handle
harder limit on delta chainlength.
Long delta chain has a very detrimental impact on read (and write) performance
in revlog. Being able to shorter them provide a great boost. However, shorting
delta used to result significantly lower compression ratio. The intermediate
snapshots effectively suppress most of this effect (even all in some case).
# Effect on the test repository
The repository we use for test is not "realistic" but can still show this in
action using an unreasonably low chain limit. Limiting the chain length show a
sizeable increase but stay under control: +6% for limit=15; +15% for limit=10.
Without the snapshot system the increase is significantly bigger: +45% for
limit=15; +80% for limit=10. Even slightly larger than without delta chain
limit, the resulting size is still smaller than before we started doing
snapshots.
Here is a table for comparison. *Since the repository is not branchy, the
initial sparse-revlog version does not bring much benefit compare to the
non-sparse one):
chain length limit | none | limit=15 | limit=10 |
without sparse-revlog | 62 818 987 | 112 664 615 | 131 222 574 |
without snapshot | 74 365 490 | 108 211 410 | 133 857 764 |
with snapshot | 59 230 936 | 63 002 924 | 68 415 329 |
# Effect On Real Life Repositories
The series provides significant benefits on all kind of repositories.
Using `hg debugupgraderepo -o redeltaparent --run`, we recomputed delta chain
for various repositories with different settings:
- delta chain length: unlimited or 1000 limit
- sparse-revlog: enabled or disabled
- this series: applied or not applied
We can observe multiple types of effect:
- On very branchy repositories:
* The delta chain limit as low impact on the repo size.
* Intermediate snapshot greatly reduces manifest size:
- pypy: -80%
- netbeans: -95%
* The delta chain limit is effective, without a size impact:
- netbeans average: 613 -> 282
- private #1 average: 1 068 -> 307
- On more linear repository:
* Intermediate snapshot limit the impact of delta chain limit:
- mozilla:
without the series: +360%
with the series: +25%
* The delta chain limit provides large improvement:
- mozilla's average chain length:
unlimited: 15 338
limited: 469
* Despite the chain length limit, the manifest size is reduced:
- mercurial: -25%
- mozilla: -30%
It is clear that the use of chains of intermediate snapshots provide large
benefits both in storage size and delta chains quality. We should now switch our
effort toward making sure the write performance are acceptable. Then,
`sparse-revlog` will be a suitable format for all new repository.
# Raw Statistic
* no-sparse: general delta repository not using sparse-revlog
* no-snapshot: sparse-revlog repository not using this series
* snapshot: sparse-revlog repository using this series
mercurial
Manifest Size:
limit | none | 1000
------------|-------------|------------
no-sparse | 8 021 373 | 8 199 366
no-snapshot | 8 103 561 | 8 259 719
snapshot | 6 137 116 | 6 126 433
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 307 | 1456 || 279 | 1000 ||
no-snapshot || 312 | 1456 || 283 | 1000 ||
snapshot || 248 | 1208 || 241 | 1000 ||
Full Store Size
limit | none | 1000
------------|-------------|------------
no-sparse | 51 013 198 | 51 201 574
no-snapshot | 50 930 795 | 51 141 006
snapshot | 48 072 037 | 48 093 572
pypy
Manifest Size:
limit | none | 1000
------------|-------------|------------
no-sparse | 193 987 784 | 193 987 784
no-snapshot | 163 171 745 | 163 312 229
snapshot | 34 605 900 | 34 600 750
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 101 | 692 || 101 | 692 ||
no-snapshot || 151 | 1307 || 148 | 1000 ||
snapshot || 128 | 1309 || 125 | 1000 ||
Full Store Size
limit | none | 1000
------------|-------------|------------
no-sparse | 495 931 473 | 495 931 473
no-snapshot | 465 441 017 | 465 581 501
snapshot | 355 467 301 | 355 472 451
Mozilla
Manifest Size:
limit | none | 1000
------------|----------------|---------------
no-sparse | 416 757 148 | 1 869 009 668
no-snapshot | 401 592 370 | 1 843 493 795
snapshot | 224 359 521 | 284 615 500
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 15 333 | 58 980 || 468 | 1 000 ||
no-snapshot || 15 336 | 58 980 || 469 | 1 000 ||
snapshot || 15 338 | 58 983 || 469 | 1 000 ||
Full Store Size
limit | none | 1000
------------|----------------|---------------
no-sparse | 2 712 477 887 | 4 164 995 451
no-snapshot | 2 698 887 835 | 4 141 054 304
snapshot | 2 518 130 385 | 2 578 587 596
Netbeans
Manifest Size:
limit | none | 1000
------------|----------------|---------------
no-sparse | 4 766 794 101 | 4 870 642 687
no-snapshot | 4 334 806 082 | 4 428 681 309
snapshot | 232 659 666 | 240 330 665
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 597 | 6802 || 254 | 1 000 ||
no-snapshot || 648 | 6 802 || 305 | 1 000 ||
snapshot || 613 | 6 804 || 282 | 1 000 ||
Full Store Size
limit | none | 1000
------------|----------------|---------------
no-sparse | 5 807 347 998 | 5 911 196 584
no-snapshot | 5 375 398 602 | 5 469 273 829
snapshot | 1 282 519 928 | 1 290 190 927
Private repo #1
Manifest Size:
limit | none | 1000
------------|-----------------|---------------
no-sparse | 41 389 010 840 | 41 398 162 091
no-snapshot | 9 737 319 435 | 10 223 773 150
snapshot | 744 215 807 | 747 961 822
Manifest Chain length data
limit || none || 1000 ||
value || average | max || average | max ||
------------||---------|---------||---------|---------||
no-sparse || 245 | 8 885 || 81 | 1 000 ||
no-snapshot || 1 225 | 8 885 || 336 | 1 000 ||
snapshot || 1 068 | 7 909 || 307 | 1 000 ||
Full Store Size
limit | none | 1000
------------|----------------|---------------
no-sparse | 49 646 065 126 | 49 655 216 377
no-snapshot | 17 924 862 856 | 18 411 316 571
snapshot | 9 009 024 710 | 9 012 770 725
Private repo #2
We currently have less data available for that repository.
* Before is a sparse-revlog repository without this series
* After is a sparse-revlog repository with this series + 1000 chain limit
Manifest Size:
Before: 1 531 485 040 bytes
After: 1 091 422 451 bytes
Manifest Chain:
Before: 2 218 avg; 6 575 Max
After: 442 avg; 1 000 Max
Full Store Size
Before: 15 203 955 615
after: 8 207 180 693
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:18:45 -0400] rev 39522
snapshot: also consider the snapshot chain of one unrelated revision
To maximize the chance of good delta chain reuse, we inject an unrelated delta
chain into our search. To do so, we search for the highest revision unrelated
to the parents of the current revision and use its snapshot chain too.
Adding this extra snapshot into the mix can have a performance impact. We'll
deal with performance impact in a later series.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:37 -0400] rev 39521
snapshot: extract parent chain computation
The final step of this series is to include chain related to "prev" in the
search. Before adding that code we do some simple code movement to clarify the
next diff.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:36 -0400] rev 39520
snapshot: refine candidate snapshot base upward
Once we found a suitable snapshot base it is useful to check if it has a
"children" snapshot that would provide a better diff. This is useful when base
not directly related to stored revision are picked. In those case, we "jumped"
to this new chain at an arbitrary point, checking if a higher point is more
appropriate will help to provide better results and increase snapshot reuse.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:35 -0400] rev 39519
snapshot: try to refine new snapshot base down the chain
There are cases where doing a diff against a snapshot's parent will be shorter
than against the snapshot itself. Reusing snapshot not directly related to the
revision we are trying to store increase this odd.
So once we found a possible candidate, we check the snapshots lower in the
chain.
This will involve extra processing, but this extra processing will only happen
when we are doing building a snapshot, a rare situation.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:34 -0400] rev 39518
snapshot: make sure we'll never refine delta base from a reused source
The point of reusing delta from the source is to avoid doing computation when
applying a bundle. Refining such delta would go against that spirit.
We do not have refining logic in place yet. This code needed to be moved out
of the way before we could start adding such logic.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:34 -0400] rev 39517
snapshot: turn _refinedgroups into a coroutine
We are now almost ready to start adding refining logic.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:33 -0400] rev 39516
snapshot: also use None as a stop value for `_refinegroup`
This is yet another small step toward turning `_refinegroups` into a co-routine.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:33 -0400] rev 39515
snapshot: add refining logic at the findeltainfo level
Once we found a delta, we want to have the candidates logic challenge it,
searching for a better candidate.
The logic at the lower level is still missing. We'll introduce it later.
Adding small changes in individual commits make it simpler to explain the code
change.
This is another small step toward turning `_refinegroups` into a co-routine.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:32 -0400] rev 39514
snapshot: use None as a stop value when looking for a good delta
Having clear stop value should help keep clear logic around the co-routine.
The alternative of using a StopIteration exception give a messier result.
This is one small step toward turning `_refinegroups` into a co-routine.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:32 -0400] rev 39513
snapshot: introduce an intermediate `_refinedgroups` generator
This method will be used to improve the search for a good snapshot base. To
keep things simpler, we introduce the necessary function before doing any
delta base logic change. The next handful of commits will focus on refactoring
the code to let that new logic land as clearly as possible.
# General Idea
Right now, the search for a good delta base stop whenever we found a good one.
However, when using sparse-revlog, we should probably try a bit harder.
We do significant effort to increase delta re-use by jumping on "unrelated"
delta chains that provide better results. Moving to another chain for a better
result is good, but we have no guarantee we jump at a reasonable point in that
new chain. When we consider over the chains related to the parents, we start
from the higher-level snapshots. This is a way to consider the snapshot closer
to the current revision that has the best chance to produce a small delta. We
do benefit from this walk order when jumping to a better "unrelated" stack.
To counter-balance this, we'll introduce a way to "refine" the result. After a
good delta have been found, we'll keep searching for a better delta, using the
current best one as a starting point.
# Target Setup
The `finddeltainfo` method is responsible for the general search for a good
delta. It requests candidates base from `_candidategroups` and decides which
one are usable.
The `_candidategroups` generator act as a top-level filter, it does not care
about how we pick candidates, it just does basic filtering, excluding
revisions that have been tested already or that are an obvious misfit.
The `_rawgroups` generator is the one with the actual ancestors walking logic,
It does not care about what would do a good delta and what was already tested,
it just issues the initial candidates.
We introduce a new `_refinedgroup` function to bridge the gap between
`_candidategroups` and `_rawgroups`. It delegates the initial iteration logic
and then performing relevant refining of the valid base once found. (This
logic is yet to be added to function)
All these logics are fairly independent and easier to understand when standing
alone, not mixed with each other. It also makes it easy to test and try
different approaches for one of those four layers without affecting the other
ones.
# Technical details
To communicate `finddeltainfo` choice of "current best delta base" to the
`_refinegroup` logic, we plan to use python co-routine feature. The
`_candidategroups` and `_refinegroup` generators will become co-routine. This
will allow `_refinegroup` to detect when a good delta have been found and
triggers various refining steps.
For now, `_candidategroups` will just pass the value down the stack.
After poking at various option, the co-routine appears the best to keep each
layers focus on its duty, without the need to spread implementation details
across layers.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:31 -0400] rev 39512
snapshot: consider unrelated snapshots at a similar level first
This new step is inserted before considering using a level-N snapshot as a
base for a level-N+1 snapshot. We first check if existing level-N+1 snapshots
using the same base would be a suitable base for a level-N+2 snapshot.
This increases snapshot reuse and limits the risk of snapshot explosion in
very branchy repositories.
Using a "deeper" snapshot as the base also results in a smaller snapshot since
it builds a level-N+2 intermediate snapshot instead of an N+1 one.
This logic is similar for the one we added in a previous commit. In that
previous commit is only applied to level-0 "siblings".
We can see this effect in the test repository. Snapshots moved from lower
levels to higher levels.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:30 -0400] rev 39511
snapshot: consider all snapshots in the parents' chains
There are no reasons to only consider full snapshot as a possible base for an
intermediate snapshot. Now that the basic principles have been set, we can
start adding more levels of snapshots.
We now consider all snapshots in the parent's chains (full or intermediate).
This creates a chain of intermediate snapshots, each smaller than the previous
one.
# Effect On The Test Repository
In the test repository, we can see a decrease in the revlog size and slightly
shorter delta chain.
However, that approach creates snapshots more frequently, increasing the risk
of ending into problematic cases in very branchy repositories (not triggered
by the test repository). The next changesets will remove that risk by adding
logic that increases deltas reuse.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:30 -0400] rev 39510
snapshot: search for unrelated but reusable full-snapshot
# New Strategy Step: Reusing Snapshot Outside Of Parents' Chain.
If no suitable bases were found in the parent's chains, see if we could reuse
a full snapshot not directly related to the current revision. Such search can
be expensive, so we only search for snapshots appended to the revlog *after*
the bases used by the parents of the current revision (the one we just
tested). We assume the parent's bases were created because the previous
snapshots were unsuitable, so there are low odds they would be useful now.
This search gives a chance to reuse a delta chain unrelated to the current
revision. Without this re-use, topological branches would keep reopening new
full chains. Creating more and more snapshots as the repository grow.
In repositories with many topological branches, the lack of delta reuse can
create too many snapshots reducing overall compression to nothing. This
results in a very large repository and other usability issues.
For now, we still focus on creating level-1 snapshots. However, this principle
will play a large part in how we avoid snapshot explosion once we have more
snapshot levels.
# Effects On The Test Repository
In the test repository we created, we can see the beneficial effect of such
reuse. We need very few level-0 snapshots and the overall revlog size has
decreased.
The `hg debugrevlog` call, show a "lvl-2" snapshot. It comes from the existing
delta logic using the `prev` revision (revlog's tip) as the base. In this
specific case, it turns out the tip was a level-1 snapshot. This is a
coincidence that can be ignored.
Finding and testing against all these unrelated snapshots can have a
performance impact at write time. We currently focus on building good deltas
chain we build. Performance concern will be dealt with later in another
series.
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:17:29 -0400] rev 39509
snapshot: try intermediate snapshot against parents' base
# Regarding The Series Started By This Changeset
This is the first changesets of a group adjusting delta chain strategy to
build a useful chain of intermediate snapshots. The series will introduce a
full strategy to produce chains of multiple snapshots on top of which a
"usual" delta chain will be built.
That strategy will have multiple steps to maximize snapshot reuse, avoiding
pathological cases and improving overall compression in very branchy
repositories. An important property of sparse-revlog using such snapshot-chain
is that they can use very short delta chain without problematic impact on the
resulting compression. Shorter delta chains are important to achieve good
performance.
To make each step clear, we'll introduce them one by one.
See the end of this series for full details.
# Regarding This Changeset
Before this change, if we cannot store the current revision as a delta against
a "simple" candidate (p1, p2, prev), we created a new level-0 snapshot (also
called full snapshot).
As the first step, we introduce a simple strategy: try an intermediate level-1
snapshot against the chain base of the "current revision" parents.
The "current revision" is the one we are currently trying to store in the
revlog, triggering this search for a good delta base.
The first item in the chain is always a level-0 snapshot.
# Effect On The Test Repository
We can already see the effect on the test-repository. Most of the snapshots
have shifted from level 0 to level 1. The overall size has slightly decreased.
(However, keep in mind that this repository only emulates real data)
# Regarding Statistic
The current series focuses on improving the chain built. Improving the
performance of this logic will be done as a second step. Sparse-revlog is
still experimental and disabled by default.
We'll provide more statistic about resulting size and delta chain at the end
of this series.
Boris Feld <boris.feld@octobus.net> [Mon, 10 Sep 2018 09:08:24 -0700] rev 39508
sparse-revlog: add a test checking revlog deltas for a churning file
The test repository contains 5000 revisions and is therefore slow to build:
five minutes with CHG, over fifteen minutes without. It is too slow to build
during the test. Bundling all content produce a sizeable result, 20BM, too
large to be committed. Instead, we commit a script to build the expected
bundle and the test checks if the bundle is available. Any run of the script
will produce the same repository content, using resulting in the same hashes.
Using smaller repositories was tried, however, it misses most of the cases we
are planning to improve. Having them in a 5000 repository is already nice, we
usually see these case in repositories in the order of magnitude of one
million revisions.
This test will be very useful to check various changes strategy for building
delta to store in a sparse-revlog.
In this series we will focus our attention on the following metrics:
The ones that will impact the final storage performance (size, space):
* size of the revlog data file (".hg/store/data/*.d")
* chain length info
The ones that describe the deltas patterns:
* number of snapshot revision (and their level)
* size taken by snapshot revision (and their level)
Boris Feld <boris.feld@octobus.net> [Sat, 18 Aug 2018 12:45:44 +0200] rev 39507
tests: add a `tests/artifacts/` directory
That directory is meant to cache large items used by tests that are slow to
generate. See 'PURPOSE' file for details and next changesets for a first user.
Meirambek Omyrzak <meirambek77@gmail.com> [Wed, 05 Sep 2018 01:19:48 +0300] rev 39506
verify: make output less confusing (issue5924)
output before: "500 files, 2035 changesets, 2622 total revisions"
output after: "checked 2035 changesets with 2622 changes to 500 files"
new one was suggested in the comments inside the issue.
Differential Revision: https://phab.mercurial-scm.org/D4476
Boris Feld <boris.feld@octobus.net> [Tue, 04 Sep 2018 21:28:28 +0200] rev 39505
revlog: clarify the comment attached to delta reuse
The previous version was a bit complicated and referred to a deprecated
configuration option.
Boris Feld <boris.feld@octobus.net> [Tue, 04 Sep 2018 21:05:21 +0200] rev 39504
revlog: drop duplicated code
This code probably got duplicated by a rebase/evolve conflict. We drop the
extra copy, the other copy is right below.
This had no real effects since other logic ensure that we never test the same
revision twice.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 05 Sep 2018 09:04:40 -0700] rev 39503
wireprotov2peer: properly format errors
formatrichmessage() expects an iterable containing dicts with
well-defined keys. We were passing in something else. This caused
an exception.
Change the code to call formatrichmessage() with the proper argument.
And add a TODO to potentially emit the proper data structure from
the server in the first place.
Differential Revision: https://phab.mercurial-scm.org/D4441
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 23 Aug 2018 13:50:47 -0700] rev 39502
wireprotov2peer: report exceptions in frame handling against request future
Otherwise the future may never resolve, which could cause deadlock.
Differential Revision: https://phab.mercurial-scm.org/D4440
Anton Shestakov <av6@dwimlabs.net> [Sat, 08 Sep 2018 21:58:51 +0800] rev 39501
httppeer: use util.readexactly() to abort on incomplete responses
Plain resp.read(n) may not return exactly n bytes when we need, and to detect
such cases before trying to interpret whatever has been read, we can use
util.readexactly(), which raises an Abort when stream ends unexpectedly. In the
first case here, readexactly() prevents a traceback with struct.error, in the
second it avoids looking for invalid compression engines.
In this test case, _wraphttpresponse doesn't catch the problem (presumably
because it doesn't know transfer encoding), and the code continues reading the
response until it gets to compression engine data. Maybe there should be checks
before the execution gets there, but I'm not sure where (httplib?)
Anton Shestakov <av6@dwimlabs.net> [Sat, 08 Sep 2018 23:57:07 +0800] rev 39500
httppeer: calculate total expected bytes correctly
User-facing error messages that handled httplib.IncompleteRead errors in
Mercurial used to look like this:
abort: HTTP request error (incomplete response; expected 3 bytes got 1)
But the errors that are being handled underneath the UI look like this:
IncompleteRead(1 bytes read, 3 more expected)
I.e. the error actually counts total number of expected bytes minus bytes
already received.
Before, users could see weird messages like "expected 10 bytes got 10", while
in reality httplib expected 10 _more_ bytes (20 in total).
Martin von Zweigbergk <martinvonz@google.com> [Fri, 07 Sep 2018 23:36:09 -0700] rev 39499
lazyancestors: reuse __iter__ implementation in __contains__
There was a comment in the code that said "Trying to do both __iter__
and __contains__ using the same visit heap and seen set is complex
enough that it slows down both. Keep them separate.". However, it
seems easy and efficient to make __contains__ keep an iterator across
calls.
I couldn't measure any slowdown from `hg bundle --all` (which seem to
call lazyancestors.__contains__ frequently).
Differential Revision: https://phab.mercurial-scm.org/D4508
Martin von Zweigbergk <martinvonz@google.com> [Sun, 09 Sep 2018 23:16:55 -0700] rev 39498
lazyancestors: extract __iter__ to free function
The next patch will keep a reference to the returned iterator in a
field, which would otherwise result in a reference cycle.
Differential Revision: https://phab.mercurial-scm.org/D4517
Boris Feld <boris.feld@octobus.net> [Thu, 30 Aug 2018 01:53:21 +0200] rev 39497
phase: report number of non-public changeset alongside the new range
When interacting with non-publishing repository or bundle, it is useful to
have some information about the phase of the changeset we just pulled.
This changeset updates the "new changesets MIN:MAX" output to also includes
phases information for non-public changesets. Displaying extra data about
non-public changesets means the output for exchange with publishing repository
(the default) is unaffected.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 07 Sep 2018 23:54:42 -0400] rev 39496
tests: disable test-nointerrupt on Windows
Per the followup discussion[1]. proc.send_signal(INT) in timeout.py raises a
ValueError because of an unsupported signal. I don't like missing test coverage
for this on Windows. But this is the last test failing on Windows, and red all
the time hides new failures.
[1] https://phab.mercurial-scm.org/D3716
Matt Harbison <matt_harbison@yahoo.com> [Fri, 07 Sep 2018 23:39:49 -0400] rev 39495
tests: conditionalize an error message about unlinking a non empty directory
The message on Windows comes from win32.unlink(). It looks like os.unlink() on
posix platforms is a simple call to unlink(3), which turns into unlinkat(2).
Since there's a comment in one of the tests that the message should be improved,
I don't think it's worth adding a check in win32.unlink() to see if it's empty,
if that function is always going to fail on a directory. (It seems like the
POSIX spec allows unlinking directories though.)
Martin von Zweigbergk <martinvonz@google.com> [Fri, 07 Sep 2018 14:48:38 -0700] rev 39494
ancestors: add nullrev to set from the beginning
Differential Revision: https://phab.mercurial-scm.org/D4507
Yuya Nishihara <yuya@tcha.org> [Sat, 08 Sep 2018 10:59:24 +0900] rev 39493
ancestor: filter out initial revisions lower than stoprev
Yuya Nishihara <yuya@tcha.org> [Sat, 08 Sep 2018 10:48:42 +0900] rev 39492
ancestor: add test showing inconsistency between __iter__ and __contains__
Boris Feld <boris.feld@octobus.net> [Thu, 06 Sep 2018 19:37:38 -0400] rev 39491
ancestors: ensure a consistent order even in the "inclusive" case
It seems odds to first issue the "source" revs and then the other ancestors.
In addition, doing so can break the other contract of always issuing a child
before its parent. We update the code to apply the same logic to all yielded
revision. No tests break so we seem in the clear except where we explicitly
test the order.
Boris Feld <boris.feld@octobus.net> [Thu, 06 Sep 2018 17:00:28 -0400] rev 39490
ancestors: actually iterate over ancestors in topological order (issue5979)
This code previously used a dequeue logic, the first ancestors seen were the
first ancestors to be emitted. In branching/merging situations, it can result
in ancestors being yielded before their descendants, breaking the object
contract.
We got affected by this issue while working on the copy tracing code. At about
the same time, Axel Hecht <axel@mozilla.com> reported the issue and provided
the test case used in this changeset. Thanks Axel.
Running `hg perfancestors` does not show a significant difference between the
old and the new version.
Yuya Nishihara <yuya@tcha.org> [Thu, 06 Sep 2018 22:12:21 +0900] rev 39489
doc: use modern import style in runrst
Yuya Nishihara <yuya@tcha.org> [Sun, 26 Aug 2018 22:18:09 +0900] rev 39488
hgweb: do not audit URL path as working-directory path
Since hgweb is an interface to repository data, we don't need to prohibit
any paths conflicting within the filesystem. Still an access to working
files is audited by filectx.
Yuya Nishihara <yuya@tcha.org> [Sun, 26 Aug 2018 22:23:25 +0900] rev 39487
hgweb: map Abort to 403 error to report inaccessible path for example
Abort is so common in our codebase. We could instead introduce a dedicated
type for path auditing errors, but we'll probably have to catch error.Abort
anyway.
As you can see, an abort message may include a full path to the repository,
which might be considered information leak. If that matters, we should hide
the message and send it to the server log instead.
Yuya Nishihara <yuya@tcha.org> [Fri, 07 Sep 2018 22:19:28 +0900] rev 39486
hgweb: add error template to json so it won't crash
Yuya Nishihara <yuya@tcha.org> [Fri, 07 Sep 2018 22:12:46 +0900] rev 39485
hgweb: show shortlog by default in json output (issue5978)
Augie Fackler <augie@google.com> [Fri, 07 Sep 2018 11:35:43 -0400] rev 39484
merge with stable
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 15:16:22 +0300] rev 39483
tests: improve the widening testing in test-narrow-widen*
Before this patch, we are testing `hg tracked --addinclude` by adding a command
which is not introduced in the changesets till now.
If you closely look at the tests, wider/f was introduced on the server after the
narrow clone was done and extending the existing clone to include wider/f does
not make sense. We should test extending a file which exists.
Differential Revision: https://phab.mercurial-scm.org/D4452
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 19:26:50 +0300] rev 39482
narrow: use util.readfile() and improve error message using --narrowspec
This patch improves the error message and uses util.readfile() for reading
narrowspecs file while cloning.
Differential Revision: https://phab.mercurial-scm.org/D4462
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 04 Sep 2018 15:55:23 -0700] rev 39481
merge: use vfs methods for I/O
All I/O is supposed to be performed via vfs instances so filesystems
can be abstracted. The previous commit ported the old code in purge,
which didn't go through the vfs layer. This commit ports the purge
code to use the vfs layer.
The vfs layer didn't have a method to remove a single directory, so
it was added as part of implementing this.
Differential Revision: https://phab.mercurial-scm.org/D4478
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 Sep 2018 18:30:12 -0700] rev 39480
merge: move purge logic from extension
Working directory purging feels like functionality that should be
in core rather than in an extension.
This commit effectively moves the core purging logic from the
purge extension to merge.py.
Code was refactored slightly. Rather than deal with printing in
this function, the function is a generator of paths and the caller
can worry about printing.
Differential Revision: https://phab.mercurial-scm.org/D4477
Matt Harbison <matt_harbison@yahoo.com> [Thu, 06 Sep 2018 23:37:24 -0400] rev 39479
tests: stabilize test-removeemptydirs.t on Windows
Yuya Nishihara <yuya@tcha.org> [Thu, 06 Sep 2018 21:55:30 +0900] rev 39478
help: add internals.wireprotocolv2 to the table, and remove redundant header
Kyle Lippincott <spectral@google.com> [Fri, 17 Aug 2018 19:18:53 -0700] rev 39477
match: improve includematcher.visitchildrenset to be much faster and cached
This improves the speed of visitchildrenset considerably, especially when there
are complicated matchers involved that may have many entries in _dirs or
_parents.
Unfortunately the benchmark isn't easily upstreamed due to its reliance on
https://github.com/vstinner/perf (primarily due to the conflict when importing
it if I were to contribute the benchmark as contrib/matcherbenchmarks.py)
instead of asv or some other perf measurement system.
To describe the benchmark briefly: I generated an includematcher of either 5 or
3500 "rootfilesin:prefix1/prefix2/prefix3/<randomsubdirs, 1-8 levels deep>"
items in the 'setup' function, and then called
`im.visitchildrenset('prefix1/prefix2')` in the 'stmt' function in perf.timeit.
For the set of 5:
- before: 15.3 us +- 2.9 us
- after: 1.59 us +- 0.02 us
For the set of 3500:
- before: 3.90 ms +- 0.10 ms
- after: 3.15 us +- 0.09 us (note the m->u change)
Differential Revision: https://phab.mercurial-scm.org/D4351
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 06 Sep 2018 03:21:05 +0530] rev 39476
py3: add new passing tests spotted by the buildbot
Differential Revision: https://phab.mercurial-scm.org/D4495
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 06 Sep 2018 03:24:27 +0530] rev 39475
tests: order the imports in test-fastannotate-hg.t
The wrong ordering breaks test-check-module-imports.t on Python 3. I am not sure
why that test is so much active on py3.
Differential Revision: https://phab.mercurial-scm.org/D4496
Matt Harbison <matt_harbison@yahoo.com> [Thu, 06 Sep 2018 00:51:21 -0400] rev 39474
lfs: ensure the blob is linked to the remote store on skipped uploads
I noticed a "missing" blob when pushing two repositories with common blobs to a
fresh server, and then running `hg verify` as a user different from the one
running the web server. When pushing the second repo, several of the blobs
already existed in the user cache, so the server indicated to the client that it
doesn't need to upload the blobs. That's good enough for the web server process
to serve up in the future. But a different user has a different cache by
default, so verify complains that `lfs.url` needs to be set, because it wants to
fetch the missing blobs.
Aside from that corner case, it's better to keep all of the blobs in the repo
whenever possible. Especially since the largefiles wiki says the user cache can
be deleted at any time to reclaim disk space- users switching over may have the
same expectations.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 04 Sep 2018 22:29:38 -0400] rev 39473
cbor: teach the encoder to handle python `long` type for Windows
The tests for 2**32 and -7000000000 were blowing up, complaining about not
knowing how to encode type 'long'. sys.maxint tops out at 2**31-1 on Windows,
but I guess is 2^63-1 on Linux? I *think* we're OK on the decode side, as there
is an assertion that the decoded value is equal to the original primitive value.
I opted for the pycompat alias instead of swallowing the NameError because the
vendored cbor package uses an alias, and I see at least pywatchman and
templatefilters open codes their own aliases.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 04 Sep 2018 20:48:22 -0400] rev 39472
absorb: clarify the reason for not finding changesets to modify
I'm used to pushing to non-publishing repos, so this was the last thing I
thought to check.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 05 Sep 2018 09:36:31 -0700] rev 39471
merge with stable
Includes an extra bump of the version number for parsers because the
merge produces a new, distinct version of the code.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 04 Sep 2018 10:22:42 -0700] rev 39470
state: use our CBOR module
This was the last consumer of the vendored CBOR package in
core.
Differential Revision: https://phab.mercurial-scm.org/D4471
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:41:09 -0700] rev 39469
wireprotov2peer: use our CBOR decoder
Behavior should be the same in order to preserve backwards
compatibility. We obviously want to stream values in this code.
We'll do that in subsequent commits.
Differential Revision: https://phab.mercurial-scm.org/D4470
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:37:55 -0700] rev 39468
debugcommands: use our CBOR decoder
It implements the set of CBOR needed for the wire protocol.
Differential Revision: https://phab.mercurial-scm.org/D4469
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 18:12:04 -0700] rev 39467
wireprotov2server: use our CBOR encoder
Again, test output changed slightly because of map key ordering
differences. This shouldn't matter.
I could have called oncommandresponsereadygen() with the raw output
from the stream encoder. However, this changed test output further.
I left a TODO to follow up on that later.
Differential Revision: https://phab.mercurial-scm.org/D4468
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 18:05:48 -0700] rev 39466
wireprotoserver: use our CBOR encoder
Moving away from the vendored package.
Again, our encoder normalizes keys differently from the vendored
package (for now), hence the test changes.
Differential Revision: https://phab.mercurial-scm.org/D4467
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 18:05:08 -0700] rev 39465
wireprotoframing: use our CBOR module
Tests changed because our CBOR encoder appears to sort map keys
differently from the vendored CBOR package. The CBOR specification
does define canonical sorting rules for keys based on the
byte values. I'm guessing our implementation doesn't follow them.
But our encoder doesn't guarantee that it conforms with the canonical
specification. Right now, we just care that output is deterministic.
And our encoder does guarantee that.
Differential Revision: https://phab.mercurial-scm.org/D4466
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:10:56 -0700] rev 39464
httppeer: use our CBOR decoder
We just implemented our own CBOR decoder. Let's use it in
httppeer.
Differential Revision: https://phab.mercurial-scm.org/D4465
Anton Shestakov <av6@dwimlabs.net> [Wed, 22 Aug 2018 09:02:07 +0800] rev 39463
zsh_completion: complete shelve and unshelve
Differential Revision: https://phab.mercurial-scm.org/D4428
Anton Shestakov <av6@dwimlabs.net> [Tue, 21 Aug 2018 20:30:53 +0800] rev 39462
zsh_completion: use revsets to exclude this rev from suggestions to hg merge
One of the most important aspects of a completion system is its speed, so 1
call to hg is definitely better than 4.
Sorting by rev (descending) is to preserve the same order as in `hg heads`
output.
While at it, declare branches as an array too.
Differential Revision: https://phab.mercurial-scm.org/D4426
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 Sep 2018 10:36:34 -0700] rev 39461
drawdag: correctly pass repo to super constructor
This was caught by someone who was looking for the
super(cls, self).__init__(self, ...)
pattern in all code in our Google-internal repo.
Differential Revision: https://phab.mercurial-scm.org/D4464
Matt Harbison <matt_harbison@yahoo.com> [Tue, 04 Sep 2018 20:54:41 -0400] rev 39460
help: correct a typo in tweakdefaults
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 06 Aug 2018 14:06:19 +0300] rev 39459
narrow: add '--import-rules' flag to tracked command
This patch adds a `--import-rules` flag to tracked command provided by narrow
extension. Using the --import-rules flag, you can pass a filename from which
narrowspecs should be read and added to main narrowspec.
A lot of times, in automation or manually also, when you are working with big
repo, specifying each path name on commandline using '--addinclude' and
'--addexclude' is tedious and something which can scale. So we needed something
where we can pass a file to extend the narrowspecs.
Nice thing about this is that the automations which reads some file to change
the sparse profile, can now read the same file for changing narrowspecs too.
Tests are added for the new feature.
Differential Revision: https://phab.mercurial-scm.org/D4125
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 23 Aug 2018 13:11:13 -0700] rev 39458
internals: extract wire protocol version 2 commands to standalone doc
wireprotocol.txt is a bit long and unwieldy. I think splitting it up
will help aid comprehension.
Let's start by extracting wire protocol version 2 commands to a
standalone document.
As part of the port, I munged with the section titles a bit and
expanded the TODO around node namespaces.
Differential Revision: https://phab.mercurial-scm.org/D4442
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 23 Aug 2018 13:46:39 -0700] rev 39457
wireprotov2peer: split responsedata handling into separate function
So we don't have so much logic inside an if/elif block.
Differential Revision: https://phab.mercurial-scm.org/D4439
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 22 Aug 2018 10:25:47 -0700] rev 39456
httppeer: add TODO about delayed handling of ^C
When testing wire protocol v2 with large responses, I noticed that
^C doesn't result in immediate termination. I'm not sure what the
root cause is. Let's add a TODO so we don't forget about it.
Differential Revision: https://phab.mercurial-scm.org/D4438
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 31 Aug 2018 08:57:14 -0700] rev 39455
httppeer: log commands for version 2 peer
This will aid testing and debugging.
Differential Revision: https://phab.mercurial-scm.org/D4437
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 22 Aug 2018 10:29:30 -0700] rev 39454
wireprotov2peer: add TODO about streaming responses
This is a pretty big deficiency in the current implementation.
We don't want to forget about this.
Differential Revision: https://phab.mercurial-scm.org/D4435
Augie Fackler <augie@google.com> [Tue, 04 Sep 2018 12:36:33 -0400] rev 39453
tests: update relnotes test for actual output
I'm unclear why I didn't see this before but I consistently see it
now. In any event, it doesn't appear to be a regression, so let's just
bake it into the test.
Differential Revision: https://phab.mercurial-scm.org/D4463
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:57:49 +0300] rev 39452
py3: use print as a function in tests/test-hgweb.t
Differential Revision: https://phab.mercurial-scm.org/D4461
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:58:39 +0300] rev 39451
py3: bytestr a bytes val to make sure we get bytechr while iterating
Differential Revision: https://phab.mercurial-scm.org/D4460