Mercurial > hg
view tests/test-rename.t @ 51681:522b4d729e89
mmap: populate the mapping by default
Without pre-population, accessing all data through a mmap can result in many
pagefault, reducing performance significantly. If the mmap is prepopulated, the
performance can no longer get slower than a full read.
(See benchmark number below)
In some cases were very few data is read, prepopulating can be overkill and
slower than populating on access (through page fault). So that behavior can be
controlled when the caller can pre-determine the best behavior.
(See benchmark number below)
In addition, testing with populating in a secondary thread yield great result
combining the best of each approach. This might be implemented in later
changesets.
In all cases, using mmap has a great effect on memory usage when many processes
run in parallel on the same machine.
### Benchmarks
# What did I run
A couple of month back I ran a large benchmark campaign to assess the impact of
various approach for using mmap with the revlog (and other files), it
highlighted a few benchmarks that capture the impact of the changes well. So to
validate this change I checked the following:
- log command displaying various revisions
(read the changelog index)
- log command displaying the patch of listed revisions
(read the changelog index, the manifest index and a few files indexes)
- unbundling a few revisions
(read and write changelog, manifest and few files indexes, and walk the graph
to update some cache)
- pushing a few revisions
(read and write changelog, manifest and few files indexes, walk the graph to
update some cache, performs various accesses locally and remotely during
discovery)
Benchmarks were run using the default module policy (c+py) and the rust one. No
significant difference were found between the two implementation, so we will
present result using the default policy (unless otherwise specified).
I ran them on a few repositories :
- mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using
zstd compression and sparse-revlog
- pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog
- netbeans: a copy of netbeans from 2018-08-01 using zstd compression and
sparse-revlog
- mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and
sparse-revlog
- mozilla-try persistent-nodemap: Same as the above but with a persistent
nodemap. Used for the log --patch benchmark only
# Results
For the smaller repositories (mercurial, pypy), the impact of mmap is almost
imperceptible, other cost dominating the operation. The impact of prepopulating
is undiscernible in the benchmark we ran.
For larger repositories the benchmark support explanation given above:
On netbeans, the log can be about 1% faster without repopulation (for a
difference < 100ms) but unbundle becomes a bit slower, even when small.
### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog
# benchmark.name = hg.command.unbundle
# benchmark.variants.issue6528 = disabled
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
# benchmark.variants.source = unbundle
# benchmark.variants.verbosity = quiet
with-populate: 0.240157
no-populate: 0.265087 (+10.38%, +0.02)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.459518
no-populate: 1.481290 (+1.49%, +0.02)
## benchmark.name = hg.command.push
# benchmark.variants.explicit-rev = none
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
with-populate: 0.771919
no-populate: 0.792025 (+2.60%, +0.02)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.459518
no-populate: 1.481290 (+1.49%, +0.02)
For mozilla-try, the "slow down" from pre-populate for small `hg log` is more
visible, but still small in absolute time. (using rust value for the persistent
nodemap value to be relevant).
### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm
# benchmark.name = hg.command.log
# bin-env-vars.hg.flavor = rust
# benchmark.variants.patch = yes
# benchmark.variants.limit-rev = 1
with-populate: 0.237813
no-populate: 0.229452 (-3.52%, -0.01)
# benchmark.variants.limit-rev = 10
# benchmark.variants.patch = yes
with-populate: 1.213578
no-populate: 1.205189
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.variants.limit-rev = 1000
# benchmark.variants.patch = no
# benchmark.variants.rev = tip
with-populate: 0.198607
no-populate: 0.195038 (-1.80%, -0.00)
However pre-populating provide a significant boost on more complex operations
like unbundle or push:
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = hg.command.push
# benchmark.variants.explicit-rev = none
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
with-populate: 4.798632
no-populate: 4.953295 (+3.22%, +0.15)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 4.903618
no-populate: 5.014963 (+2.27%, +0.11)
## benchmark.name = hg.command.unbundle
# benchmark.variants.revs = any-1-extra-rev
with-populate: 1.423411
no-populate: 1.585365 (+11.38%, +0.16)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.537909
no-populate: 1.688489 (+9.79%, +0.15)
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Thu, 11 Apr 2024 00:02:07 +0200 |
parents | 55c6ebd11cb9 |
children |
line wrap: on
line source
$ hg init repo $ cd repo $ mkdir d1 d1/d11 d2 $ echo d1/a > d1/a $ echo d1/ba > d1/ba $ echo d1/a1 > d1/d11/a1 $ echo d1/b > d1/b $ echo d2/b > d2/b $ hg add d1/a d1/b d1/ba d1/d11/a1 d2/b $ hg commit -m "1" rename a single file $ hg rename d1/d11/a1 d2/c $ hg --config ui.portablefilenames=abort rename d1/a d1/con.xml abort: filename contains 'con', which is reserved on Windows: d1/con.xml [10] $ hg sum parent: 0:9b4b6e7b2c26 tip 1 branch: default commit: 1 renamed update: (current) phases: 1 draft $ hg status -C A d2/c d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d2/c rename a single file using absolute paths $ hg rename `pwd`/d1/d11/a1 `pwd`/d2/c $ hg status -C A d2/c d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d2/c rename --after a single file $ mv d1/d11/a1 d2/c $ hg rename --after d1/d11/a1 d2/c $ hg status -C A d2/c d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d2/c rename --after a single file when src and tgt already tracked $ mv d1/d11/a1 d2/c $ hg addrem -s 0 removing d1/d11/a1 adding d2/c $ hg rename --after d1/d11/a1 d2/c $ hg status -C A d2/c d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d2/c rename --after a single file to a nonexistent target filename $ hg rename --after d1/a dummy d1/a: not recording move - dummy does not exist [1] move a single file to an existing directory $ hg rename d1/d11/a1 d2 $ hg status -C A d2/a1 d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d2/a1 move --after a single file to an existing directory $ mv d1/d11/a1 d2 $ hg rename --after d1/d11/a1 d2 $ hg status -C A d2/a1 d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d2/a1 rename a file using a relative path $ (cd d1/d11; hg rename ../../d2/b e) $ hg status -C A d1/d11/e d2/b R d2/b $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/d11/e rename --after a file using a relative path $ (cd d1/d11; mv ../../d2/b e; hg rename --after ../../d2/b e) $ hg status -C A d1/d11/e d2/b R d2/b $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/d11/e rename directory d1 as d3 $ hg rename d1/ d3 moving d1/a to d3/a moving d1/b to d3/b moving d1/ba to d3/ba moving d1/d11/a1 to d3/d11/a1 $ hg status -C A d3/a d1/a A d3/b d1/b A d3/ba d1/ba A d3/d11/a1 d1/d11/a1 R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 rename --after directory d1 as d3 $ mv d1 d3 $ hg rename --after d1 d3 moving d1/a to d3/a moving d1/b to d3/b moving d1/ba to d3/ba moving d1/d11/a1 to d3/d11/a1 $ hg status -C A d3/a d1/a A d3/b d1/b A d3/ba d1/ba A d3/d11/a1 d1/d11/a1 R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 move a directory using a relative path $ (cd d2; mkdir d3; hg rename ../d1/d11 d3) moving ../d1/d11/a1 to d3/d11/a1 $ hg status -C A d2/d3/d11/a1 d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d2/d3 move --after a directory using a relative path $ (cd d2; mkdir d3; mv ../d1/d11 d3; hg rename --after ../d1/d11 d3) moving ../d1/d11/a1 to d3/d11/a1 $ hg status -C A d2/d3/d11/a1 d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d2/d3 move directory d1/d11 to an existing directory d2 (removes empty d1) $ hg rename d1/d11/ d2 moving d1/d11/a1 to d2/d11/a1 $ hg status -C A d2/d11/a1 d1/d11/a1 R d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d2/d11 move directories d1 and d2 to a new directory d3 $ mkdir d3 $ hg rename d1 d2 d3 moving d1/a to d3/d1/a moving d1/b to d3/d1/b moving d1/ba to d3/d1/ba moving d1/d11/a1 to d3/d1/d11/a1 moving d2/b to d3/d2/b $ hg status -C A d3/d1/a d1/a A d3/d1/b d1/b A d3/d1/ba d1/ba A d3/d1/d11/a1 d1/d11/a1 A d3/d2/b d2/b R d1/a R d1/b R d1/ba R d1/d11/a1 R d2/b $ hg update -C 5 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 move --after directories d1 and d2 to a new directory d3 $ mkdir d3 $ mv d1 d2 d3 $ hg rename --after d1 d2 d3 moving d1/a to d3/d1/a moving d1/b to d3/d1/b moving d1/ba to d3/d1/ba moving d1/d11/a1 to d3/d1/d11/a1 moving d2/b to d3/d2/b $ hg status -C A d3/d1/a d1/a A d3/d1/b d1/b A d3/d1/ba d1/ba A d3/d1/d11/a1 d1/d11/a1 A d3/d2/b d2/b R d1/a R d1/b R d1/ba R d1/d11/a1 R d2/b $ hg update -C 5 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 move everything under directory d1 to existing directory d2, do not overwrite existing files (d2/b) $ hg rename d1/* d2 d2/b: not overwriting - file already committed ('hg rename --force' to replace the file by recording a rename) moving d1/d11/a1 to d2/d11/a1 [1] $ hg status -C A d2/a d1/a A d2/ba d1/ba A d2/d11/a1 d1/d11/a1 R d1/a R d1/ba R d1/d11/a1 $ diff -u d1/b d2/b --- d1/b * (glob) +++ d2/b * (glob) @@ * (glob) -d1/b +d2/b [1] $ hg update -C 3 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d2/a d2/ba d2/d11/a1 attempt to move one file into a non-existent directory $ hg rename d1/a dx/ abort: destination dx/ is not a directory [10] $ hg status -C $ hg update -C 0 files updated, 0 files merged, 0 files removed, 0 files unresolved attempt to move potentially more than one file into a non-existent directory $ hg rename 'glob:d1/**' dx abort: with multiple sources, destination must be an existing directory [10] move every file under d1 to d2/d21 $ mkdir d2/d21 $ hg rename 'glob:d1/**' d2/d21 moving d1/a to d2/d21/a moving d1/b to d2/d21/b moving d1/ba to d2/d21/ba moving d1/d11/a1 to d2/d21/a1 $ hg status -C A d2/d21/a d1/a A d2/d21/a1 d1/d11/a1 A d2/d21/b d1/b A d2/d21/ba d1/ba R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d2/d21 move --after some files under d1 to d2/d21 $ mkdir d2/d21 $ mv d1/a d1/d11/a1 d2/d21 $ hg rename --after 'glob:d1/**' d2/d21 moving d1/a to d2/d21/a d1/b: not recording move - d2/d21/b does not exist d1/ba: not recording move - d2/d21/ba does not exist moving d1/d11/a1 to d2/d21/a1 [1] $ hg status -C A d2/d21/a d1/a A d2/d21/a1 d1/d11/a1 R d1/a R d1/d11/a1 $ hg update -C 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d2/d21 move every file under d1 starting with an 'a' to d2/d21 (regexp) $ mkdir d2/d21 $ hg rename 're:d1/([^a][^/]*/)*a.*' d2/d21 moving d1/a to d2/d21/a moving d1/d11/a1 to d2/d21/a1 $ hg status -C A d2/d21/a d1/a A d2/d21/a1 d1/d11/a1 R d1/a R d1/d11/a1 $ hg update -C 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d2/d21 attempt to overwrite an existing file $ echo "ca" > d1/ca $ hg rename d1/ba d1/ca d1/ca: not overwriting - file exists ('hg rename --after' to record the rename) [1] $ hg status -C ? d1/ca $ hg update -C 0 files updated, 0 files merged, 0 files removed, 0 files unresolved forced overwrite of an existing file $ echo "ca" > d1/ca $ hg rename --force d1/ba d1/ca $ hg status -C A d1/ca d1/ba R d1/ba $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/ca attempt to overwrite an existing broken symlink #if symlink $ ln -s ba d1/ca $ hg rename --traceback d1/ba d1/ca d1/ca: not overwriting - file exists ('hg rename --after' to record the rename) [1] $ hg status -C ? d1/ca $ hg update -C 0 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/ca replace a symlink with a file $ ln -s ba d1/ca $ hg rename --force d1/ba d1/ca $ hg status -C A d1/ca d1/ba R d1/ba $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/ca #endif do not copy more than one source file to the same destination file $ mkdir d3 $ hg rename d1/* d2/* d3 moving d1/d11/a1 to d3/d11/a1 d3/b: not overwriting - d2/b collides with d1/b [1] $ hg status -C A d3/a d1/a A d3/b d1/b A d3/ba d1/ba A d3/d11/a1 d1/d11/a1 R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 move a whole subtree with "hg rename ." $ mkdir d3 $ (cd d1; hg rename . ../d3) moving a to ../d3/d1/a moving b to ../d3/d1/b moving ba to ../d3/d1/ba moving d11/a1 to ../d3/d1/d11/a1 $ hg status -C A d3/d1/a d1/a A d3/d1/b d1/b A d3/d1/ba d1/ba A d3/d1/d11/a1 d1/d11/a1 R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 move a whole subtree with "hg rename --after ." $ mkdir d3 $ mv d1/* d3 $ (cd d1; hg rename --after . ../d3) moving a to ../d3/a moving b to ../d3/b moving ba to ../d3/ba moving d11/a1 to ../d3/d11/a1 $ hg status -C A d3/a d1/a A d3/b d1/b A d3/ba d1/ba A d3/d11/a1 d1/d11/a1 R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 move the parent tree with "hg rename .." $ (cd d1/d11; hg rename .. ../../d3) moving ../a to ../../d3/a moving ../b to ../../d3/b moving ../ba to ../../d3/ba moving a1 to ../../d3/d11/a1 $ hg status -C A d3/a d1/a A d3/b d1/b A d3/ba d1/ba A d3/d11/a1 d1/d11/a1 R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 skip removed files $ hg remove d1/b $ hg rename d1 d3 moving d1/a to d3/a moving d1/ba to d3/ba moving d1/d11/a1 to d3/d11/a1 $ hg status -C A d3/a d1/a A d3/ba d1/ba A d3/d11/a1 d1/d11/a1 R d1/a R d1/b R d1/ba R d1/d11/a1 $ hg update -C 4 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm -rf d3 transitive rename $ hg rename d1/b d1/bb $ hg rename d1/bb d1/bc $ hg status -C A d1/bc d1/b R d1/b $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/bc transitive rename --after $ hg rename d1/b d1/bb $ mv d1/bb d1/bc $ hg rename --after d1/bb d1/bc $ hg status -C A d1/bc d1/b R d1/b $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/bc $ echo "# idempotent renames (d1/b -> d1/bb followed by d1/bb -> d1/b)" # idempotent renames (d1/b -> d1/bb followed by d1/bb -> d1/b) $ hg rename d1/b d1/bb $ echo "some stuff added to d1/bb" >> d1/bb $ hg rename d1/bb d1/b $ hg status -C M d1/b $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved overwriting with renames (issue1959) $ hg rename d1/a d1/c $ hg rename d1/b d1/a $ hg status -C M d1/a d1/b A d1/c d1/a R d1/b $ hg diff --git diff --git a/d1/a b/d1/a --- a/d1/a +++ b/d1/a @@ -1,1 +1,1 @@ -d1/a +d1/b diff --git a/d1/b b/d1/b deleted file mode 100644 --- a/d1/b +++ /dev/null @@ -1,1 +0,0 @@ -d1/b diff --git a/d1/a b/d1/c copy from d1/a copy to d1/c $ hg update -C 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm d1/c # The file was marked as added, so 'hg update' action was 'forget' check illegal path components $ hg rename d1/d11/a1 .hg/foo abort: path contains illegal component: .hg/foo [10] $ hg status -C $ hg rename d1/d11/a1 ../foo abort: ../foo not under root '$TESTTMP/repo' [255] $ hg status -C $ mv d1/d11/a1 .hg/foo $ hg rename --after d1/d11/a1 .hg/foo abort: path contains illegal component: .hg/foo [10] $ hg status -C ! d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm .hg/foo $ hg rename d1/d11/a1 .hg abort: path contains illegal component: .hg/a1 [10] $ hg --config extensions.largefiles= rename d1/d11/a1 .hg The fsmonitor extension is incompatible with the largefiles extension and has been disabled. (fsmonitor !) abort: path contains illegal component: .hg/a1 [10] $ hg status -C $ hg rename d1/d11/a1 .. abort: ../a1 not under root '$TESTTMP/repo' [255] $ hg --config extensions.largefiles= rename d1/d11/a1 .. The fsmonitor extension is incompatible with the largefiles extension and has been disabled. (fsmonitor !) abort: ../a1 not under root '$TESTTMP/repo' [255] $ hg status -C $ mv d1/d11/a1 .hg $ hg rename --after d1/d11/a1 .hg abort: path contains illegal component: .hg/a1 [10] $ hg status -C ! d1/d11/a1 $ hg update -C 1 files updated, 0 files merged, 0 files removed, 0 files unresolved $ rm .hg/a1 $ (cd d1/d11; hg rename ../../d2/b ../../.hg/foo) abort: path contains illegal component: .hg/foo [10] $ hg status -C $ (cd d1/d11; hg rename ../../d2/b ../../../foo) abort: ../../../foo not under root '$TESTTMP/repo' [255] $ hg status -C check that stat information such as mtime is preserved on rename - it's unclear whether the `touch` and `stat` commands are portable, so we mimic them using python. Not all platforms support precision of even one-second granularity, so we allow a rather generous fudge factor here; 1234567890 is 2009, and the primary thing we care about is that it's not the machine's current time; hopefully it's really unlikely for a machine to have such a broken clock that this test fails. :) $ mkdir mtime Create the file (as empty), then update its mtime and atime to be 1234567890. >>> import os >>> filename = "mtime/f" >>> mtime = 1234567890 >>> open(filename, "w").close() >>> os.utime(filename, (mtime, mtime)) $ hg ci -qAm 'add mtime dir' "hg cp" does not preserve the mtime, so it should be newer than the 2009 timestamp. $ hg cp -q mtime mtime_cp >>> import os >>> filename = "mtime_cp/f" >>> print(os.stat(filename).st_mtime < 1234567999) False "hg mv" preserves the mtime, so it should be ~equal to the 2009 timestamp (modulo some fudge factor due to not every system supporting 1s-level precision). $ hg mv -q mtime mtime_mv >>> import os >>> filename = "mtime_mv/f" >>> print(os.stat(filename).st_mtime < 1234567999) True