Mercurial > hg
view tests/test-audit-path.t @ 51681:522b4d729e89
mmap: populate the mapping by default
Without pre-population, accessing all data through a mmap can result in many
pagefault, reducing performance significantly. If the mmap is prepopulated, the
performance can no longer get slower than a full read.
(See benchmark number below)
In some cases were very few data is read, prepopulating can be overkill and
slower than populating on access (through page fault). So that behavior can be
controlled when the caller can pre-determine the best behavior.
(See benchmark number below)
In addition, testing with populating in a secondary thread yield great result
combining the best of each approach. This might be implemented in later
changesets.
In all cases, using mmap has a great effect on memory usage when many processes
run in parallel on the same machine.
### Benchmarks
# What did I run
A couple of month back I ran a large benchmark campaign to assess the impact of
various approach for using mmap with the revlog (and other files), it
highlighted a few benchmarks that capture the impact of the changes well. So to
validate this change I checked the following:
- log command displaying various revisions
(read the changelog index)
- log command displaying the patch of listed revisions
(read the changelog index, the manifest index and a few files indexes)
- unbundling a few revisions
(read and write changelog, manifest and few files indexes, and walk the graph
to update some cache)
- pushing a few revisions
(read and write changelog, manifest and few files indexes, walk the graph to
update some cache, performs various accesses locally and remotely during
discovery)
Benchmarks were run using the default module policy (c+py) and the rust one. No
significant difference were found between the two implementation, so we will
present result using the default policy (unless otherwise specified).
I ran them on a few repositories :
- mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using
zstd compression and sparse-revlog
- pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog
- netbeans: a copy of netbeans from 2018-08-01 using zstd compression and
sparse-revlog
- mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and
sparse-revlog
- mozilla-try persistent-nodemap: Same as the above but with a persistent
nodemap. Used for the log --patch benchmark only
# Results
For the smaller repositories (mercurial, pypy), the impact of mmap is almost
imperceptible, other cost dominating the operation. The impact of prepopulating
is undiscernible in the benchmark we ran.
For larger repositories the benchmark support explanation given above:
On netbeans, the log can be about 1% faster without repopulation (for a
difference < 100ms) but unbundle becomes a bit slower, even when small.
### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog
# benchmark.name = hg.command.unbundle
# benchmark.variants.issue6528 = disabled
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
# benchmark.variants.source = unbundle
# benchmark.variants.verbosity = quiet
with-populate: 0.240157
no-populate: 0.265087 (+10.38%, +0.02)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.459518
no-populate: 1.481290 (+1.49%, +0.02)
## benchmark.name = hg.command.push
# benchmark.variants.explicit-rev = none
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
with-populate: 0.771919
no-populate: 0.792025 (+2.60%, +0.02)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.459518
no-populate: 1.481290 (+1.49%, +0.02)
For mozilla-try, the "slow down" from pre-populate for small `hg log` is more
visible, but still small in absolute time. (using rust value for the persistent
nodemap value to be relevant).
### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm
# benchmark.name = hg.command.log
# bin-env-vars.hg.flavor = rust
# benchmark.variants.patch = yes
# benchmark.variants.limit-rev = 1
with-populate: 0.237813
no-populate: 0.229452 (-3.52%, -0.01)
# benchmark.variants.limit-rev = 10
# benchmark.variants.patch = yes
with-populate: 1.213578
no-populate: 1.205189
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.variants.limit-rev = 1000
# benchmark.variants.patch = no
# benchmark.variants.rev = tip
with-populate: 0.198607
no-populate: 0.195038 (-1.80%, -0.00)
However pre-populating provide a significant boost on more complex operations
like unbundle or push:
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = hg.command.push
# benchmark.variants.explicit-rev = none
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
with-populate: 4.798632
no-populate: 4.953295 (+3.22%, +0.15)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 4.903618
no-populate: 5.014963 (+2.27%, +0.11)
## benchmark.name = hg.command.unbundle
# benchmark.variants.revs = any-1-extra-rev
with-populate: 1.423411
no-populate: 1.585365 (+11.38%, +0.16)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.537909
no-populate: 1.688489 (+9.79%, +0.15)
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Thu, 11 Apr 2024 00:02:07 +0200 |
parents | 55c6ebd11cb9 |
children | 9b14a8cf4f78 |
line wrap: on
line source
The simple store doesn't escape paths robustly and can't store paths with periods, etc. So much of this test fails with it. #require no-reposimplestore $ hg init repo $ cd repo audit of .hg $ hg add .hg/00changelog.i abort: path contains illegal component: .hg/00changelog.i [10] #if symlink Symlinks $ mkdir a $ echo a > a/a $ hg ci -Ama adding a/a $ ln -s a b $ echo b > a/b $ hg add b/b abort: path 'b/b' traverses symbolic link 'b' [255] $ hg add b should still fail - maybe $ hg add b/b abort: path 'b/b' traverses symbolic link 'b' [255] $ hg commit -m 'add symlink b' Test symlink traversing when accessing history: ----------------------------------------------- (build a changeset where the path exists as a directory) $ hg up 0 0 files updated, 0 files merged, 1 files removed, 0 files unresolved $ mkdir b $ echo c > b/a $ hg add b/a $ hg ci -m 'add directory b' created new head Test that hg cat does not do anything wrong the working copy has 'b' as directory $ hg cat b/a c $ hg cat -r "desc(directory)" b/a c $ hg cat -r "desc(symlink)" b/a b/a: no such file in rev bc151a1f53bd [1] Test that hg cat does not do anything wrong the working copy has 'b' as a symlink (issue4749) $ hg up 'desc(symlink)' 1 files updated, 0 files merged, 1 files removed, 0 files unresolved $ hg cat b/a b/a: no such file in rev bc151a1f53bd [1] $ hg cat -r "desc(directory)" b/a c $ hg cat -r "desc(symlink)" b/a b/a: no such file in rev bc151a1f53bd [1] #endif unbundle tampered bundle $ hg init target $ cd target $ hg unbundle "$TESTDIR/bundles/tampered.hg" adding changesets adding manifests adding file changes added 5 changesets with 6 changes to 6 files (+4 heads) new changesets b7da9bf6b037:fc1393d727bc (5 drafts) (run 'hg heads' to see heads, 'hg merge' to merge) attack .hg/test $ hg manifest -r0 .hg/test $ hg update -Cr0 abort: path contains illegal component: .hg/test [10] attack foo/.hg/test $ hg manifest -r1 foo/.hg/test $ hg update -Cr1 abort: path 'foo/.hg/test' is inside nested repo 'foo' [10] attack back/test where back symlinks to .. $ hg manifest -r2 back back/test #if symlink $ hg update -Cr2 abort: path 'back/test' traverses symbolic link 'back' [255] #else ('back' will be a file and cause some other system specific error) $ hg update -Cr2 abort: $TESTTMP/target/back/test: $ENOTDIR$ [255] #endif attack ../test $ hg manifest -r3 ../test $ mkdir ../test $ echo data > ../test/file $ hg update -Cr3 abort: path contains illegal component: ../test [10] $ cat ../test/file data attack /tmp/test $ hg manifest -r4 /tmp/test $ hg update -Cr4 abort: path contains illegal component: /tmp/test [10] $ cd .. Test symlink traversal on merge: -------------------------------- #if symlink set up symlink hell $ mkdir merge-symlink-out $ hg init merge-symlink $ cd merge-symlink $ touch base $ hg commit -qAm base $ ln -s ../merge-symlink-out a $ hg commit -qAm 'symlink a -> ../merge-symlink-out' $ hg up -q 0 $ mkdir a $ touch a/poisoned $ hg commit -qAm 'file a/poisoned' $ hg log -G -T '{rev}: {desc}\n' @ 2: file a/poisoned | | o 1: symlink a -> ../merge-symlink-out |/ o 0: base try trivial merge $ hg up -qC 1 $ hg merge 2 abort: path 'a/poisoned' traverses symbolic link 'a' [255] try rebase onto other revision: cache of audited paths should be discarded, and the rebase should fail (issue5628) $ hg up -qC 2 $ hg rebase -s 2 -d 1 --config extensions.rebase= rebasing 2:e73c21d6b244 tip "file a/poisoned" abort: path 'a/poisoned' traverses symbolic link 'a' [255] $ ls ../merge-symlink-out $ cd .. Test symlink traversal on update: --------------------------------- $ mkdir update-symlink-out $ hg init update-symlink $ cd update-symlink $ ln -s ../update-symlink-out a $ hg commit -qAm 'symlink a -> ../update-symlink-out' $ hg rm a $ mkdir a && touch a/b $ hg ci -qAm 'file a/b' a/b $ hg up -qC 0 $ hg rm a $ mkdir a && touch a/c $ hg ci -qAm 'rm a, file a/c' $ hg log -G -T '{rev}: {desc}\n' @ 2: rm a, file a/c | | o 1: file a/b |/ o 0: symlink a -> ../update-symlink-out try linear update where symlink already exists: $ hg up -qC 0 $ hg up 1 abort: path 'a/b' traverses symbolic link 'a' [255] try linear update including symlinked directory and its content: paths are audited first by calculateupdates(), where no symlink is created so both 'a' and 'a/b' are taken as good paths. still applyupdates() should fail. $ hg up -qC null $ hg up 1 abort: path 'a/b' traverses symbolic link 'a' [255] $ ls ../update-symlink-out try branch update replacing directory with symlink, and its content: the path 'a' is audited as a directory first, which should be audited again as a symlink. $ rm -f a $ hg up -qC 2 $ hg up 1 abort: path 'a/b' traverses symbolic link 'a' [255] $ ls ../update-symlink-out $ cd .. #endif