Mercurial > hg
view tests/test-share-safe.t @ 51681:522b4d729e89
mmap: populate the mapping by default
Without pre-population, accessing all data through a mmap can result in many
pagefault, reducing performance significantly. If the mmap is prepopulated, the
performance can no longer get slower than a full read.
(See benchmark number below)
In some cases were very few data is read, prepopulating can be overkill and
slower than populating on access (through page fault). So that behavior can be
controlled when the caller can pre-determine the best behavior.
(See benchmark number below)
In addition, testing with populating in a secondary thread yield great result
combining the best of each approach. This might be implemented in later
changesets.
In all cases, using mmap has a great effect on memory usage when many processes
run in parallel on the same machine.
### Benchmarks
# What did I run
A couple of month back I ran a large benchmark campaign to assess the impact of
various approach for using mmap with the revlog (and other files), it
highlighted a few benchmarks that capture the impact of the changes well. So to
validate this change I checked the following:
- log command displaying various revisions
(read the changelog index)
- log command displaying the patch of listed revisions
(read the changelog index, the manifest index and a few files indexes)
- unbundling a few revisions
(read and write changelog, manifest and few files indexes, and walk the graph
to update some cache)
- pushing a few revisions
(read and write changelog, manifest and few files indexes, walk the graph to
update some cache, performs various accesses locally and remotely during
discovery)
Benchmarks were run using the default module policy (c+py) and the rust one. No
significant difference were found between the two implementation, so we will
present result using the default policy (unless otherwise specified).
I ran them on a few repositories :
- mercurial: a "public changeset only" copy of mercurial from 2018-08-01 using
zstd compression and sparse-revlog
- pypy: a copy of pypy from 2018-08-01 using zstd compression and sparse-revlog
- netbeans: a copy of netbeans from 2018-08-01 using zstd compression and
sparse-revlog
- mozilla-try: a copy of mozilla-try from 2019-02-18 using zstd compression and
sparse-revlog
- mozilla-try persistent-nodemap: Same as the above but with a persistent
nodemap. Used for the log --patch benchmark only
# Results
For the smaller repositories (mercurial, pypy), the impact of mmap is almost
imperceptible, other cost dominating the operation. The impact of prepopulating
is undiscernible in the benchmark we ran.
For larger repositories the benchmark support explanation given above:
On netbeans, the log can be about 1% faster without repopulation (for a
difference < 100ms) but unbundle becomes a bit slower, even when small.
### data-env-vars.name = netbeans-2018-08-01-zstd-sparse-revlog
# benchmark.name = hg.command.unbundle
# benchmark.variants.issue6528 = disabled
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
# benchmark.variants.source = unbundle
# benchmark.variants.verbosity = quiet
with-populate: 0.240157
no-populate: 0.265087 (+10.38%, +0.02)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.459518
no-populate: 1.481290 (+1.49%, +0.02)
## benchmark.name = hg.command.push
# benchmark.variants.explicit-rev = none
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
with-populate: 0.771919
no-populate: 0.792025 (+2.60%, +0.02)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.459518
no-populate: 1.481290 (+1.49%, +0.02)
For mozilla-try, the "slow down" from pre-populate for small `hg log` is more
visible, but still small in absolute time. (using rust value for the persistent
nodemap value to be relevant).
### data-env-vars.name = mozilla-try-2019-02-18-ds2-pnm
# benchmark.name = hg.command.log
# bin-env-vars.hg.flavor = rust
# benchmark.variants.patch = yes
# benchmark.variants.limit-rev = 1
with-populate: 0.237813
no-populate: 0.229452 (-3.52%, -0.01)
# benchmark.variants.limit-rev = 10
# benchmark.variants.patch = yes
with-populate: 1.213578
no-populate: 1.205189
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.variants.limit-rev = 1000
# benchmark.variants.patch = no
# benchmark.variants.rev = tip
with-populate: 0.198607
no-populate: 0.195038 (-1.80%, -0.00)
However pre-populating provide a significant boost on more complex operations
like unbundle or push:
### data-env-vars.name = mozilla-try-2019-02-18-zstd-sparse-revlog
# benchmark.name = hg.command.push
# benchmark.variants.explicit-rev = none
# benchmark.variants.issue6528 = disabled
# benchmark.variants.protocol = ssh
# benchmark.variants.reuse-external-delta-parent = yes
# benchmark.variants.revs = any-1-extra-rev
with-populate: 4.798632
no-populate: 4.953295 (+3.22%, +0.15)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 4.903618
no-populate: 5.014963 (+2.27%, +0.11)
## benchmark.name = hg.command.unbundle
# benchmark.variants.revs = any-1-extra-rev
with-populate: 1.423411
no-populate: 1.585365 (+11.38%, +0.16)
# benchmark.variants.revs = any-100-extra-rev
with-populate: 1.537909
no-populate: 1.688489 (+9.79%, +0.15)
author | Pierre-Yves David <pierre-yves.david@octobus.net> |
---|---|
date | Thu, 11 Apr 2024 00:02:07 +0200 |
parents | 704c3d0878d9 |
children |
line wrap: on
line source
setup $ cat >> $HGRCPATH <<EOF > [extensions] > share = > [format] > use-share-safe = True > [storage] > revlog.persistent-nodemap.slow-path=allow > # enforce zlib to ensure we can upgrade to zstd later > [format] > revlog-compression=zlib > # we want to be able to enable it later > use-persistent-nodemap=no > EOF prepare source repo $ hg init source $ cd source $ cat .hg/requires dirstate-v2 (dirstate-v2 !) share-safe $ cat .hg/store/requires dotencode fncache generaldelta revlogv1 sparserevlog store $ hg debugrequirements dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 share-safe sparserevlog store $ echo a > a $ hg ci -Aqm "added a" $ echo b > b $ hg ci -Aqm "added b" $ HGEDITOR=cat hg config --shared abort: repository is not shared; can't use --shared [10] $ cd .. Create a shared repo and check the requirements are shared and read correctly $ hg share source shared1 updating working directory 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ cd shared1 $ cat .hg/requires dirstate-v2 (dirstate-v2 !) share-safe shared $ hg debugrequirements -R ../source dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 share-safe sparserevlog store $ hg debugrequirements dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 share-safe shared sparserevlog store $ echo c > c $ hg ci -Aqm "added c" Check that config of the source repository is also loaded $ hg showconfig ui.curses [1] $ echo "[ui]" >> ../source/.hg/hgrc $ echo "curses=true" >> ../source/.hg/hgrc $ hg showconfig ui.curses true Test that extensions of source repository are also loaded $ hg debugextensions share $ hg extdiff -p echo hg: unknown command 'extdiff' 'extdiff' is provided by the following extension: extdiff command to allow external programs to compare revisions (use 'hg help extensions' for information on enabling extensions) [10] $ echo "[extensions]" >> ../source/.hg/hgrc $ echo "extdiff=" >> ../source/.hg/hgrc $ hg debugextensions -R ../source extdiff share $ hg extdiff -R ../source -p echo BROKEN: the command below will not work if config of shared source is not loaded on dispatch but debugextensions says that extension is loaded $ hg debugextensions extdiff share $ hg extdiff -p echo However, local .hg/hgrc should override the config set by share source $ echo "[ui]" >> .hg/hgrc $ echo "curses=false" >> .hg/hgrc $ hg showconfig ui.curses false $ HGEDITOR=cat hg config --shared [ui] curses=true [extensions] extdiff= $ HGEDITOR=cat hg config --local [ui] curses=false Testing that hooks set in source repository also runs in shared repo $ cd ../source $ cat <<EOF >> .hg/hgrc > [extensions] > hooklib= > [hooks] > pretxnchangegroup.reject_merge_commits = \ > python:hgext.hooklib.reject_merge_commits.hook > EOF $ cd .. $ hg clone source cloned updating to branch default 3 files updated, 0 files merged, 0 files removed, 0 files unresolved $ cd cloned $ hg up 0 0 files updated, 0 files merged, 2 files removed, 0 files unresolved $ echo bar > bar $ hg ci -Aqm "added bar" $ hg merge 2 files updated, 0 files merged, 0 files removed, 0 files unresolved (branch merge, don't forget to commit) $ hg ci -m "merge commit" $ hg push ../source pushing to ../source searching for changes adding changesets adding manifests adding file changes error: pretxnchangegroup.reject_merge_commits hook failed: bcde3522682d rejected as merge on the same branch. Please consider rebase. transaction abort! rollback completed abort: bcde3522682d rejected as merge on the same branch. Please consider rebase. [255] $ hg push ../shared1 pushing to ../shared1 searching for changes adding changesets adding manifests adding file changes error: pretxnchangegroup.reject_merge_commits hook failed: bcde3522682d rejected as merge on the same branch. Please consider rebase. transaction abort! rollback completed abort: bcde3522682d rejected as merge on the same branch. Please consider rebase. [255] Test that if share source config is untrusted, we dont read it $ cd ../shared1 $ cat << EOF > $TESTTMP/untrusted.py > from mercurial import scmutil, util > def uisetup(ui): > class untrustedui(ui.__class__): > def _trusted(self, fp, f): > if util.normpath(fp.name).endswith(b'source/.hg/hgrc'): > return False > return super(untrustedui, self)._trusted(fp, f) > ui.__class__ = untrustedui > EOF $ hg showconfig hooks hooks.pretxnchangegroup.reject_merge_commits=python:hgext.hooklib.reject_merge_commits.hook $ hg showconfig hooks --config extensions.untrusted=$TESTTMP/untrusted.py [1] Update the source repository format and check that shared repo works $ cd ../source Disable zstd related tests because its not present on pure version #if zstd $ echo "[format]" >> .hg/hgrc $ echo "revlog-compression=zstd" >> .hg/hgrc $ hg debugupgraderepo --run -q upgrade will perform the following actions: requirements preserved: dotencode, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (no-dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (dirstate-v2 !) added: revlog-compression-zstd processed revlogs: - all-filelogs - changelog - manifest $ hg log -r . changeset: 1:5f6d8a4bf34a user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: added b #endif $ echo "[format]" >> .hg/hgrc $ echo "use-persistent-nodemap=True" >> .hg/hgrc $ hg debugupgraderepo --run -q -R ../shared1 abort: cannot use these actions on a share repository: persistent-nodemap (upgrade the main repository directly) [255] $ hg debugupgraderepo --run -q upgrade will perform the following actions: requirements preserved: dotencode, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (no-zstd no-dirstate-v2 !) preserved: dotencode, fncache, generaldelta, revlog-compression-zstd, revlogv1, share-safe, sparserevlog, store (zstd no-dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (no-zstd dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlog-compression-zstd, revlogv1, share-safe, sparserevlog, store (zstd dirstate-v2 !) added: persistent-nodemap processed revlogs: - all-filelogs - changelog - manifest $ hg log -r . changeset: 1:5f6d8a4bf34a user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: added b Shared one should work $ cd ../shared1 $ hg log -r . changeset: 2:155349b645be tag: tip user: test date: Thu Jan 01 00:00:00 1970 +0000 summary: added c Testing that nonsharedrc is loaded for source and not shared $ cd ../source $ touch .hg/hgrc-not-shared $ echo "[ui]" >> .hg/hgrc-not-shared $ echo "traceback=true" >> .hg/hgrc-not-shared $ hg showconfig ui.traceback true $ HGEDITOR=cat hg config --non-shared [ui] traceback=true $ cd ../shared1 $ hg showconfig ui.traceback [1] Unsharing works $ hg unshare Test that source config is added to the shared one after unshare, and the config of current repo is still respected over the config which came from source config $ cd ../cloned $ hg push ../shared1 pushing to ../shared1 searching for changes adding changesets adding manifests adding file changes error: pretxnchangegroup.reject_merge_commits hook failed: bcde3522682d rejected as merge on the same branch. Please consider rebase. transaction abort! rollback completed abort: bcde3522682d rejected as merge on the same branch. Please consider rebase. [255] $ hg showconfig ui.curses -R ../shared1 false $ cd ../ Test that upgrading using debugupgraderepo works ================================================= $ hg init non-share-safe --config format.use-share-safe=false $ cd non-share-safe $ hg debugrequirements dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 sparserevlog store $ echo foo > foo $ hg ci -Aqm 'added foo' $ echo bar > bar $ hg ci -Aqm 'added bar' Create a share before upgrading $ cd .. $ hg share non-share-safe nss-share updating working directory 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg debugrequirements -R nss-share dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 shared sparserevlog store $ cd non-share-safe Upgrade $ hg debugupgraderepo -q requirements preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !) added: share-safe no revlogs to process $ hg debugupgraderepo --run upgrade will perform the following actions: requirements preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !) added: share-safe share-safe Upgrades a repository to share-safe format so that future shares of this repository share its requirements and configs. no revlogs to process beginning upgrade... repository locked and read-only creating temporary repository to stage upgraded data: $TESTTMP/non-share-safe/.hg/upgrade.* (glob) (it is safe to interrupt this process any time before data migration completes) upgrading repository requirements removing temporary repository $TESTTMP/non-share-safe/.hg/upgrade.* (glob) repository upgraded to share safe mode, existing shares will still work in old non-safe mode. Re-share existing shares to use them in safe mode New shares will be created in safe mode. $ hg debugrequirements dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 share-safe sparserevlog store $ cat .hg/requires dirstate-v2 (dirstate-v2 !) share-safe $ cat .hg/store/requires dotencode fncache generaldelta revlogv1 sparserevlog store $ hg log -GT "{node}: {desc}\n" @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo Make sure existing shares dont work with default config $ hg log -GT "{node}: {desc}\n" -R ../nss-share abort: version mismatch: source uses share-safe functionality while the current share does not (see `hg help config.format.use-share-safe` for more information) [255] Create a safe share from upgrade one $ cd .. $ hg share non-share-safe ss-share updating working directory 2 files updated, 0 files merged, 0 files removed, 0 files unresolved $ cd ss-share $ hg log -GT "{node}: {desc}\n" @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo $ cd ../non-share-safe Test that downgrading works too $ cat >> $HGRCPATH <<EOF > [extensions] > share = > [format] > use-share-safe = False > EOF $ hg debugupgraderepo -q requirements preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !) removed: share-safe no revlogs to process $ hg debugupgraderepo --run upgrade will perform the following actions: requirements preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !) removed: share-safe no revlogs to process beginning upgrade... repository locked and read-only creating temporary repository to stage upgraded data: $TESTTMP/non-share-safe/.hg/upgrade.* (glob) (it is safe to interrupt this process any time before data migration completes) upgrading repository requirements removing temporary repository $TESTTMP/non-share-safe/.hg/upgrade.* (glob) repository downgraded to not use share safe mode, existing shares will not work and need to be reshared. $ hg debugrequirements dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 sparserevlog store $ cat .hg/requires dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 sparserevlog store $ test -f .hg/store/requires [1] $ hg log -GT "{node}: {desc}\n" @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo Make sure existing shares still works $ hg log -GT "{node}: {desc}\n" -R ../nss-share @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo $ hg log -GT "{node}: {desc}\n" -R ../ss-share abort: share source does not support share-safe requirement (see `hg help config.format.use-share-safe` for more information) [255] Testing automatic downgrade of shares when config is set $ touch ../ss-share/.hg/wlock $ hg log -GT "{node}: {desc}\n" -R ../ss-share --config share.safe-mismatch.source-not-safe=downgrade-abort abort: failed to downgrade share, got error: Lock held (see `hg help config.format.use-share-safe` for more information) [255] $ rm ../ss-share/.hg/wlock $ cp -R ../ss-share ../ss-share-bck $ hg log -GT "{node}: {desc}\n" -R ../ss-share --config share.safe-mismatch.source-not-safe=downgrade-abort repository downgraded to not use share-safe mode @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo $ rm -rf ../ss-share $ mv ../ss-share-bck ../ss-share $ hg log -GT "{node}: {desc}\n" -R ../ss-share --config share.safe-mismatch.source-not-safe=downgrade-abort --config share.safe-mismatch.source-not-safe:verbose-upgrade=no @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo $ hg log -GT "{node}: {desc}\n" -R ../ss-share @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo Testing automatic upgrade of shares when config is set $ hg debugupgraderepo -q --run --config format.use-share-safe=True upgrade will perform the following actions: requirements preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !) preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !) added: share-safe no revlogs to process repository upgraded to share safe mode, existing shares will still work in old non-safe mode. Re-share existing shares to use them in safe mode New shares will be created in safe mode. $ hg debugrequirements dotencode dirstate-v2 (dirstate-v2 !) fncache generaldelta revlogv1 share-safe sparserevlog store $ hg log -GT "{node}: {desc}\n" -R ../nss-share abort: version mismatch: source uses share-safe functionality while the current share does not (see `hg help config.format.use-share-safe` for more information) [255] Check that if lock is taken, upgrade fails but read operation are successful $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgra abort: share-safe mismatch with source. Unrecognized value 'upgra' of `share.safe-mismatch.source-safe` set. (see `hg help config.format.use-share-safe` for more information) [255] $ touch ../nss-share/.hg/wlock $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-allow failed to upgrade share, got error: Lock held @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-allow --config share.safe-mismatch.source-safe.warn=False @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-abort abort: failed to upgrade share, got error: Lock held (see `hg help config.format.use-share-safe` for more information) [255] $ rm ../nss-share/.hg/wlock $ cp -R ../nss-share ../nss-share-bck $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-abort repository upgraded to use share-safe mode @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo $ rm -rf ../nss-share $ mv ../nss-share-bck ../nss-share $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-abort --config share.safe-mismatch.source-safe:verbose-upgrade=no @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo Test that unshare works $ hg unshare -R ../nss-share $ hg log -GT "{node}: {desc}\n" -R ../nss-share @ f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar | o f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo Test automatique upgrade/downgrade of main-repository ------------------------------------------------------ create an initial repository $ hg init auto-upgrade \ > --config format.use-share-safe=no $ hg debugbuilddag -R auto-upgrade --new-file .+5 $ hg -R auto-upgrade update 6 files updated, 0 files merged, 0 files removed, 0 files unresolved $ hg debugformat -R auto-upgrade | grep share-safe share-safe: no upgrade it to share-safe automatically $ hg status -R auto-upgrade \ > --config format.use-share-safe.automatic-upgrade-of-mismatching-repositories=yes \ > --config format.use-share-safe=yes automatically upgrading repository to the `share-safe` feature (see `hg help config.format.use-share-safe` for details) $ hg debugformat -R auto-upgrade | grep share-safe share-safe: yes downgrade it from share-safe automatically $ hg status -R auto-upgrade \ > --config format.use-share-safe.automatic-upgrade-of-mismatching-repositories=yes \ > --config format.use-share-safe=no automatically downgrading repository from the `share-safe` feature (see `hg help config.format.use-share-safe` for details) $ hg debugformat -R auto-upgrade | grep share-safe share-safe: no