stream-clone: avoid opening a revlog in case we do not need it
Opening an revlog has a cost, especially if it is inline as we have to scan the
file and construct an index.
To prevent the associated slowdown, we just do a minimal scan to check that an
inline file is still inline, and simply stream the file without creating a
revlog when we can.
This provides a big boost compared to the previous changeset, even if the full
generation is still penalized by the initial gathering of information.
All benchmarks are run on linux with Python 3.10.7.
# benchmark.name = hg.exchange.stream.generate
# benchmark.variants.version = v2
### Compared to the previous changesets
We get a large win all across the board!
# mercurial-2018-08-01-zstd-sparse-revlog
before: 0.250694 seconds
after: 0.105986 seconds (-57.72%)
# pypy-2018-08-01-zstd-sparse-revlog
before: 3.885657 seconds
after: 1.709748 seconds (-56.00%)
# netbeans-2018-08-01-zstd-sparse-revlog
before: 16.679371 seconds
after: 7.687469 seconds (-53.91%)
# mozilla-central-2018-08-01-zstd-sparse-revlog
before: 38.575482 seconds
after: 17.520316 seconds (-54.58%)
# mozilla-try-2019-02-18-zstd-sparse-revlog
before: 81.160994 seconds
after: 37.073753 seconds (-54.32%)
### Compared to 6.4.3
We are still significantly slower than 6.4.3, the extra time is usually twice
slower than the extra time we observe on the locked section, which is a quite
interesting information.
Except for mercurial-central that is much faster. That discrepancy is not really
explained yet.
# mercurial-2018-08-01-zstd-sparse-revlog
6.4.3: 0.072560 seconds
after: 0.105986 seconds (+46.07%) (- 0.03 seconds)
# pypy-2018-08-01-zstd-sparse-revlog
6.4.3: 1.211193 seconds
after: 1.709748 seconds (+41.16%) (-0.45 seconds)
# netbeans-2018-08-01-zstd-sparse-revlog
6.4.3: 4.932843 seconds
after: 7.687469 seconds (+55.84%) (-2.75 seconds)
# mozilla-central-2018-08-01-zstd-sparse-revlog
6.4.3: 34.012226 seconds
after: 17.520316 seconds (-48.49%) (-16.49 seconds)
# mozilla-try-2019-02-18-zstd-sparse-revlog
6.4.3: 23.850555 seconds
after: 37.073753 seconds (+55.44%) (+13.22 seconds)
#require no-windows
$ . "$TESTDIR/remotefilelog-library.sh"
$ hg init master
$ cd master
$ echo treemanifest >> .hg/requires
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> server=True
> EOF
# uppercase directory name to test encoding
$ mkdir -p A/B
$ echo x > A/B/x
$ hg commit -qAm x
$ cd ..
# shallow clone from full
$ hgcloneshallow ssh://user@dummy/master shallow --noupdate
streaming all changes
4 files to transfer, 449 bytes of data
transferred 449 bytes in * seconds (*/sec) (glob)
searching for changes
no changes found
$ cd shallow
$ hg debugrequires
dotencode
dirstate-v2 (dirstate-v2 !)
exp-remotefilelog-repo-req-1
fncache
generaldelta
persistent-nodemap (rust !)
revlog-compression-zstd (zstd !)
revlogv1
share-safe
sparserevlog
store
treemanifest
$ find .hg/store/meta | sort
.hg/store/meta
.hg/store/meta/_a
.hg/store/meta/_a/00manifest.i
.hg/store/meta/_a/_b
.hg/store/meta/_a/_b/00manifest.i
$ hg update
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
$ cat A/B/x
x
$ ls .hg/store/data
$ echo foo > A/B/F
$ hg add A/B/F
$ hg ci -m 'local content'
$ ls .hg/store/data
ca31988f085bfb945cb8115b78fabdee40f741aa
$ cd ..
# shallow clone from shallow
$ hgcloneshallow ssh://user@dummy/shallow shallow2 --noupdate
streaming all changes
5 files to transfer, 1008 bytes of data
transferred 1008 bytes in * seconds (*/sec) (glob)
searching for changes
no changes found
$ cd shallow2
$ hg debugrequires
dotencode
dirstate-v2 (dirstate-v2 !)
exp-remotefilelog-repo-req-1
fncache
generaldelta
persistent-nodemap (rust !)
revlog-compression-zstd (zstd !)
revlogv1
share-safe
sparserevlog
store
treemanifest
$ ls .hg/store/data
ca31988f085bfb945cb8115b78fabdee40f741aa
$ hg update
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cat A/B/x
x
$ cd ..
# full clone from shallow
# - send stderr to /dev/null because the order of stdout/err causes
# flakiness here
$ hg clone --noupdate ssh://user@dummy/shallow full 2>/dev/null
streaming all changes
[100]
# getbundle full clone
$ printf '[server]\npreferuncompressed=False\n' >> master/.hg/hgrc
$ hgcloneshallow ssh://user@dummy/master shallow3
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 0 changes to 0 files
new changesets 18d955ee7ba0
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ ls shallow3/.hg/store/data
$ hg debugrequires -R shallow3/
dotencode
dirstate-v2 (dirstate-v2 !)
exp-remotefilelog-repo-req-1
fncache
generaldelta
persistent-nodemap (rust !)
revlog-compression-zstd (zstd !)
revlogv1
share-safe
sparserevlog
store
treemanifest