Mercurial > hg
view tests/test-audit-path.t @ 49658:523cacdfd324
delta-find: set the default candidate chunk size to 10
I ran performance and storage tests on repositories of various sizes and shapes
for the following values of the config : 5, 10, 20, 50, 100, no-chunking
The performance tests do not show any statistical impact on computation
times for large pushes and pulls.
For searching for an individual delta, this can provide a significant
performance improvement with a minor degradation of space-quality on the
result. (see data at the end of the commit).
For overall store size, the change :
- does not have any impact on many small repositories,
- has an observable, but very negligible impact on most larger repositories.
- One private repository we use for testing sees a small increase in size
(1%) in the narrower version.
We will try to get more numbers on a larger version of that repository to
make sure nothing pathological happens.
We pick "10" as the limit as "5" seems a bit more risky.
There are room to improve the current code, by using more aggressive filtering
and better (i.e any) sorting of the candidates. However this is already a large
improvement for pathological cases, with little impact in the common
situations.
The initial motivation for this change is to fix performance of delta
computation for a file where the previous code ended up testing 20 000 possible
candidate-bases in one go, which is… slow. This affected about ½ of the file
revisions leading to atrocious performance, especially during some push/pull
operations.
Details about individual delta finding timing:
----------------------------------------------
The vast majority of benchmark cases are unchanged but the three below. The first
two do not see any impact on the final delta. The last one sees a change in
delta-size that is negligible compared to the full text size.
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
∞: 5.844783
5: 4.473523 (-23.46%)
10: 4.970053 (-14.97%)
20: 5.770386 (-1.27%)
50 5.821358
100: 5.834887
MANIFESTLOG: rev = 756096: (no-limit)
delta-base = 301840
search-rounds = 6
try-count = 60
delta-type = snapshot
snap-depth = 7
delta-size = 179
MANIFESTLOG: rev=756096: (limit = 10)
delta-base=301840
search-rounds=9
try-count=51
delta-type=snapshot
snap-depth=7
delta-size=179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
∞: 5.017663
5: 3.655931 (-27.14%)
10: 4.095436 (-18.38%)
20: 4.828949 (-3.76%)
50 4.987574
100: 4.994889
MANIFESTLOG: rev=754060: (no limit)
delta-base=301840
search-rounds=5
try-count=53
delta-type=snapshot
snap-depth=7
delta-size = 179
MANIFESTLOG: rev=754060: (limite = 10)
delta-base=301840
search-rounds=8
try-count=45
delta-type=snapshot
snap-depth=7
delta-size = 179
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = perf-delta-find
# bin-env-vars.hg.flavor = rust
# benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
∞: 4.869282
5: 2.039732 (-58.11%)
10: 2.413537 (-50.43%)
20: 4.449639 (-8.62%)
50 4.865863
100: 4.882649
MANIFESTLOG: rev=693368:
delta-base=693336
search-rounds=6
try-count=53
delta-type=snapshot
snap-depth=6
full-test-size=131065
delta-size=199
MANIFESTLOG: rev=693368:
delta-base=278023
search-rounds=5
try-count=21
delta-type=snapshot
snap-depth=4
full-test-size=131065
delta-size=278
Raw data for store size (in bytes) for various chunk size value below:
----------------------------------------------------------------------
440 134 384 5 pypy/.hg/store/
440 134 384 10 pypy/.hg/store/
440 134 384 20 pypy/.hg/store/
440 134 384 50 pypy/.hg/store/
440 134 384 100 pypy/.hg/store/
440 134 384 ... pypy/.hg/store/
666 987 471 5 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 10 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 20 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 50 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 100 netbsd-xsrc-2022-11-15/.hg/store/
666 987 471 ... netbsd-xsrc-2022-11-15/.hg/store/
852 844 884 5 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 10 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 20 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 50 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 100 netbsd-pkgsrc-2022-11-15/.hg/store/
852 844 884 ... netbsd-pkgsrc-2022-11-15/.hg/store/
1 504 227 981 5 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 871 10 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 20 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 50 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 100 netbeans-2018-08-01-sparse-zstd/.hg/store/
1 504 227 813 ... netbeans-2018-08-01-sparse-zstd/.hg/store/
3 875 801 068 5 netbsd-src-2022-11-15/.hg/store/
3 875 696 767 10 netbsd-src-2022-11-15/.hg/store/
3 875 696 757 20 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 50 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 100 netbsd-src-2022-11-15/.hg/store/
3 875 696 653 ... netbsd-src-2022-11-15/.hg/store/
4 531 441 314 5 mozilla-central/.hg/store/
4 531 435 157 10 mozilla-central/.hg/store/
4 531 432 045 20 mozilla-central/.hg/store/
4 531 429 119 50 mozilla-central/.hg/store/
4 531 429 119 100 mozilla-central/.hg/store/
4 531 429 119 ... mozilla-central/.hg/store/
4 875 861 390 5 mozilla-unified/.hg/store/
4 875 855 155 10 mozilla-unified/.hg/store/
4 875 852 027 20 mozilla-unified/.hg/store/
4 875 848 851 50 mozilla-unified/.hg/store/
4 875 848 851 100 mozilla-unified/.hg/store/
4 875 848 851 ... mozilla-unified/.hg/store/
11 498 764 601 5 mozilla-try/.hg/store/
11 497 968 858 10 mozilla-try/.hg/store/
11 497 958 730 20 mozilla-try/.hg/store/
11 497 927 156 50 mozilla-try/.hg/store/
11 497 925 963 100 mozilla-try/.hg/store/
11 497 923 428 ... mozilla-try/.hg/store/
10 047 914 031 5 private-repo
9 969 132 101 10 private-repo
9 944 745 015 20 private-repo
9 939 756 703 50 private-repo
9 939 833 016 100 private-repo
9 939 822 035 ... private-repo
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Wed, 23 Nov 2022 19:08:27 +0100 |
parents | 55c6ebd11cb9 |
children | 9b14a8cf4f78 |
line wrap: on
line source
The simple store doesn't escape paths robustly and can't store paths with periods, etc. So much of this test fails with it. #require no-reposimplestore $ hg init repo $ cd repo audit of .hg $ hg add .hg/00changelog.i abort: path contains illegal component: .hg/00changelog.i [10] #if symlink Symlinks $ mkdir a $ echo a > a/a $ hg ci -Ama adding a/a $ ln -s a b $ echo b > a/b $ hg add b/b abort: path 'b/b' traverses symbolic link 'b' [255] $ hg add b should still fail - maybe $ hg add b/b abort: path 'b/b' traverses symbolic link 'b' [255] $ hg commit -m 'add symlink b' Test symlink traversing when accessing history: ----------------------------------------------- (build a changeset where the path exists as a directory) $ hg up 0 0 files updated, 0 files merged, 1 files removed, 0 files unresolved $ mkdir b $ echo c > b/a $ hg add b/a $ hg ci -m 'add directory b' created new head Test that hg cat does not do anything wrong the working copy has 'b' as directory $ hg cat b/a c $ hg cat -r "desc(directory)" b/a c $ hg cat -r "desc(symlink)" b/a b/a: no such file in rev bc151a1f53bd [1] Test that hg cat does not do anything wrong the working copy has 'b' as a symlink (issue4749) $ hg up 'desc(symlink)' 1 files updated, 0 files merged, 1 files removed, 0 files unresolved $ hg cat b/a b/a: no such file in rev bc151a1f53bd [1] $ hg cat -r "desc(directory)" b/a c $ hg cat -r "desc(symlink)" b/a b/a: no such file in rev bc151a1f53bd [1] #endif unbundle tampered bundle $ hg init target $ cd target $ hg unbundle "$TESTDIR/bundles/tampered.hg" adding changesets adding manifests adding file changes added 5 changesets with 6 changes to 6 files (+4 heads) new changesets b7da9bf6b037:fc1393d727bc (5 drafts) (run 'hg heads' to see heads, 'hg merge' to merge) attack .hg/test $ hg manifest -r0 .hg/test $ hg update -Cr0 abort: path contains illegal component: .hg/test [10] attack foo/.hg/test $ hg manifest -r1 foo/.hg/test $ hg update -Cr1 abort: path 'foo/.hg/test' is inside nested repo 'foo' [10] attack back/test where back symlinks to .. $ hg manifest -r2 back back/test #if symlink $ hg update -Cr2 abort: path 'back/test' traverses symbolic link 'back' [255] #else ('back' will be a file and cause some other system specific error) $ hg update -Cr2 abort: $TESTTMP/target/back/test: $ENOTDIR$ [255] #endif attack ../test $ hg manifest -r3 ../test $ mkdir ../test $ echo data > ../test/file $ hg update -Cr3 abort: path contains illegal component: ../test [10] $ cat ../test/file data attack /tmp/test $ hg manifest -r4 /tmp/test $ hg update -Cr4 abort: path contains illegal component: /tmp/test [10] $ cd .. Test symlink traversal on merge: -------------------------------- #if symlink set up symlink hell $ mkdir merge-symlink-out $ hg init merge-symlink $ cd merge-symlink $ touch base $ hg commit -qAm base $ ln -s ../merge-symlink-out a $ hg commit -qAm 'symlink a -> ../merge-symlink-out' $ hg up -q 0 $ mkdir a $ touch a/poisoned $ hg commit -qAm 'file a/poisoned' $ hg log -G -T '{rev}: {desc}\n' @ 2: file a/poisoned | | o 1: symlink a -> ../merge-symlink-out |/ o 0: base try trivial merge $ hg up -qC 1 $ hg merge 2 abort: path 'a/poisoned' traverses symbolic link 'a' [255] try rebase onto other revision: cache of audited paths should be discarded, and the rebase should fail (issue5628) $ hg up -qC 2 $ hg rebase -s 2 -d 1 --config extensions.rebase= rebasing 2:e73c21d6b244 tip "file a/poisoned" abort: path 'a/poisoned' traverses symbolic link 'a' [255] $ ls ../merge-symlink-out $ cd .. Test symlink traversal on update: --------------------------------- $ mkdir update-symlink-out $ hg init update-symlink $ cd update-symlink $ ln -s ../update-symlink-out a $ hg commit -qAm 'symlink a -> ../update-symlink-out' $ hg rm a $ mkdir a && touch a/b $ hg ci -qAm 'file a/b' a/b $ hg up -qC 0 $ hg rm a $ mkdir a && touch a/c $ hg ci -qAm 'rm a, file a/c' $ hg log -G -T '{rev}: {desc}\n' @ 2: rm a, file a/c | | o 1: file a/b |/ o 0: symlink a -> ../update-symlink-out try linear update where symlink already exists: $ hg up -qC 0 $ hg up 1 abort: path 'a/b' traverses symbolic link 'a' [255] try linear update including symlinked directory and its content: paths are audited first by calculateupdates(), where no symlink is created so both 'a' and 'a/b' are taken as good paths. still applyupdates() should fail. $ hg up -qC null $ hg up 1 abort: path 'a/b' traverses symbolic link 'a' [255] $ ls ../update-symlink-out try branch update replacing directory with symlink, and its content: the path 'a' is audited as a directory first, which should be audited again as a symlink. $ rm -f a $ hg up -qC 2 $ hg up 1 abort: path 'a/b' traverses symbolic link 'a' [255] $ ls ../update-symlink-out $ cd .. #endif