Mercurial > hg
view tests/test-wireproto-exchangev2-shallow.t @ 42377:0546ead39a7e stable
manifest: avoid corruption by dropping removed files with pure (issue5801)
Previously, removed files would simply be marked by overwriting the first byte
with NUL and dropping their entry in `self.position`. But no effort was made to
ignore them when compacting the dictionary into text form. This allowed them to
slip into the manifest revision, since the code seems to be trying to minimize
the string operations by copying as large a chunk as possible. As part of this,
compact() walks the existing text based on entries in the `positions` list, and
consumed everything up to the next position entry. This typically resulted in
a ValueError complaining about unsorted manifest entries.
Sometimes it seems that files do get dropped in large repos- it seems to
correspond to there being a new entry that would take the same slot. A much
more trivial problem is that if the only changes were removals, `_compact()`
didn't even run because `__delitem__` doesn't add anything to `self.extradata`.
Now there's an explicit variable to flag this, both to allow `_compact()` to
run, and to avoid searching the manifest in cases where there are no removals.
In practice, this behavior was mostly obscured by the check in fastdelta() which
takes a different path that explicitly drops removed files if there are fewer
than 1000 changes. However, timeless has a repo where after rebasing tens of
commits, a totally different path[1] is taken that bypasses the change count
check and hits this problem.
[1] https://www.mercurial-scm.org/repo/hg/file/2338bdea4474/mercurial/manifest.py#l1511
author | Matt Harbison <matt_harbison@yahoo.com> |
---|---|
date | Thu, 23 May 2019 21:54:24 -0400 |
parents | 3ed77780f4a6 |
children | 775224e26d74 |
line wrap: on
line source
#require sqlite Tests for wire protocol version 2 exchange. Tests in this file should be folded into existing tests once protocol v2 has enough features that it can be enabled via #testcase in existing tests. $ . $TESTDIR/wireprotohelpers.sh $ enablehttpv2client $ cat >> $HGRCPATH << EOF > [extensions] > sqlitestore = > pullext = $TESTDIR/pullext.py > [storage] > new-repo-backend=sqlite > EOF Configure a server $ hg init server-basic $ enablehttpv2 server-basic $ cd server-basic $ mkdir dir0 dir1 $ echo a0 > a $ echo b0 > b $ hg -q commit -A -m 'commit 0' $ echo c0 > dir0/c $ echo d0 > dir0/d $ hg -q commit -A -m 'commit 1' $ echo e0 > dir1/e $ echo f0 > dir1/f $ hg -q commit -A -m 'commit 2' $ echo c1 > dir0/c $ echo e1 > dir1/e $ hg commit -m 'commit 3' $ echo c2 > dir0/c $ echo e2 > dir1/e $ echo f1 > dir1/f $ hg commit -m 'commit 4' $ echo a1 > a $ echo b1 > b $ hg commit -m 'commit 5' $ hg log -G -T '{node} {desc}' @ 93a8bd067ed2840d9aa810ad598168383a3a2c3a commit 5 | o dc666cf9ecf3d94e6b830f30e5f1272e2a9164d9 commit 4 | o 97765fc3cd624fd1fa0176932c21ffd16adf432e commit 3 | o 47fe012ab237a8c7fc0c78f9f26d5866eef3f825 commit 2 | o b709380892b193c1091d3a817f706052e346821b commit 1 | o 3390ef850073fbc2f0dfff2244342c8e9229013a commit 0 $ hg serve -p $HGPORT -d --pid-file hg.pid -E error.log $ cat hg.pid > $DAEMON_PIDS $ cd .. Shallow clone pulls down latest revision of every file $ hg --debug clone --depth 1 http://localhost:$HGPORT client-shallow-1 using http://localhost:$HGPORT/ sending capabilities command query 1; heads sending 2 commands sending command heads: {} sending command known: { 'nodes': [] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos) sending 1 commands sending command changesetdata: { 'fields': set([ 'bookmarks', 'parents', 'phase', 'revision' ]), 'revisions': [ { 'heads': [ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:' ], 'roots': [], 'type': 'changesetdagrange' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1170; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) add changeset 3390ef850073 add changeset b709380892b1 add changeset 47fe012ab237 add changeset 97765fc3cd62 add changeset dc666cf9ecf3 add changeset 93a8bd067ed2 checking for updated bookmarks sending 1 commands sending command manifestdata: { 'fields': set([ 'parents', 'revision' ]), 'haveparents': True, 'nodes': [ '\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A', '|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8', '\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb', '113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3', 'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby', '\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83' ], 'tree': '' } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1515; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) sending 1 commands sending command filesdata: { 'fields': set([ 'linknode', 'parents', 'revision' ]), 'haveparents': False, 'revisions': [ { 'nodes': [ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:' ], 'type': 'changesetexplicit' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) updating the branch cache new changesets 3390ef850073:93a8bd067ed2 updating to branch default resolving manifests branchmerge: False, force: False, partial: False ancestor: 000000000000, local: 000000000000+, remote: 93a8bd067ed2 a: remote created -> g getting a b: remote created -> g getting b dir0/c: remote created -> g getting dir0/c dir0/d: remote created -> g getting dir0/d dir1/e: remote created -> g getting dir1/e dir1/f: remote created -> g getting dir1/f 6 files updated, 0 files merged, 0 files removed, 0 files unresolved (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob) $ sqlite3 -line client-shallow-1/.hg/store/db.sqlite << EOF > SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata ORDER BY id ASC; > EOF id = 1 path = a revnum = 0 node = \x9a8\x12)\x97\xb3\xac\x97\xbe*\x9a\xa2\xe5V\x83\x83A\xfd\xf2\xcc (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 2 id = 2 path = b revnum = 0 node = \xb1zk\xd3g=\x9a\xb8\xce\xd5\x81\xa2 \xf6/=\xa5\xccEx (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 2 id = 3 path = dir0/c revnum = 0 node = I\x1d\xa1\xbb\x89\xeax\xc0\xc0\xa2s[\x16\xce}\x93\x1d\xc8\xe2\r (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 2 id = 4 path = dir0/d revnum = 0 node = S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4& (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 0 id = 5 path = dir1/e revnum = 0 node = ]\xf3\xac\xd8\xd0\xc7\xfaP\x98\xd0'\x9a\x044\xc3\x02\x9e+x\xe1 (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 2 id = 6 path = dir1/f revnum = 0 node = (\xc7v\xae\x08\xd0\xd5^\xb4\x06H\xb4\x01\xb9\x0f\xf5DH4\x8e (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 2 Test a shallow clone with only some files $ hg --debug clone --depth 1 --include dir0/ http://localhost:$HGPORT client-shallow-narrow-1 using http://localhost:$HGPORT/ sending capabilities command query 1; heads sending 2 commands sending command heads: {} sending command known: { 'nodes': [] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos) sending 1 commands sending command changesetdata: { 'fields': set([ 'bookmarks', 'parents', 'phase', 'revision' ]), 'revisions': [ { 'heads': [ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:' ], 'roots': [], 'type': 'changesetdagrange' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1170; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) add changeset 3390ef850073 add changeset b709380892b1 add changeset 47fe012ab237 add changeset 97765fc3cd62 add changeset dc666cf9ecf3 add changeset 93a8bd067ed2 checking for updated bookmarks sending 1 commands sending command manifestdata: { 'fields': set([ 'parents', 'revision' ]), 'haveparents': True, 'nodes': [ '\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A', '|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8', '\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb', '113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3', 'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby', '\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83' ], 'tree': '' } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1515; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) sending 1 commands sending command filesdata: { 'fields': set([ 'linknode', 'parents', 'revision' ]), 'haveparents': False, 'pathfilter': { 'include': [ 'path:dir0' ] }, 'revisions': [ { 'nodes': [ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:' ], 'type': 'changesetexplicit' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=355; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) updating the branch cache new changesets 3390ef850073:93a8bd067ed2 updating to branch default resolving manifests branchmerge: False, force: False, partial: False ancestor: 000000000000, local: 000000000000+, remote: 93a8bd067ed2 dir0/c: remote created -> g getting dir0/c dir0/d: remote created -> g getting dir0/d 2 files updated, 0 files merged, 0 files removed, 0 files unresolved (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob) $ sqlite3 -line client-shallow-narrow-1/.hg/store/db.sqlite << EOF > SELECT id, path, revnum, node, p1rev, p2rev, linkrev, flags FROM filedata ORDER BY id ASC; > EOF id = 1 path = dir0/c revnum = 0 node = I\x1d\xa1\xbb\x89\xeax\xc0\xc0\xa2s[\x16\xce}\x93\x1d\xc8\xe2\r (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 2 id = 2 path = dir0/d revnum = 0 node = S\x82\x06\xdc\x97\x1eR\x15@\xd6\x84:\xbf\xe6\xd1`2\xf6\xd4& (esc) p1rev = -1 p2rev = -1 linkrev = 5 flags = 0 Cloning an old revision with depth=1 works $ hg --debug clone --depth 1 -r 97765fc3cd624fd1fa0176932c21ffd16adf432e http://localhost:$HGPORT client-shallow-2 using http://localhost:$HGPORT/ sending capabilities command sending 1 commands sending command lookup: { 'key': '97765fc3cd624fd1fa0176932c21ffd16adf432e' } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=21; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) query 1; heads sending 2 commands sending command heads: {} sending command known: { 'nodes': [] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos) sending 1 commands sending command changesetdata: { 'fields': set([ 'bookmarks', 'parents', 'phase', 'revision' ]), 'revisions': [ { 'heads': [ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.' ], 'roots': [], 'type': 'changesetdagrange' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=783; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) add changeset 3390ef850073 add changeset b709380892b1 add changeset 47fe012ab237 add changeset 97765fc3cd62 checking for updated bookmarks sending 1 commands sending command manifestdata: { 'fields': set([ 'parents', 'revision' ]), 'haveparents': True, 'nodes': [ '\x99/Gy\x02\x9a=\xf8\xd0fm\x00\xbb\x92OicN&A', '|2 \x1a\xa3\xa1R\xa9\xe6\xa9"+?\xa8\xd0\xe3\x0f\xc2V\xe8', '\x8d\xd0W<\x7f\xaf\xe2\x04F\xcc\xea\xac\x05N\xea\xa4x\x91M\xdb', '113\x85\xf2!\x8b\x08^\xb2Z\x821\x1e*\xdd\x0e\xeb\x8c3' ], 'tree': '' } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=967; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) sending 1 commands sending command filesdata: { 'fields': set([ 'linknode', 'parents', 'revision' ]), 'haveparents': False, 'revisions': [ { 'nodes': [ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.' ], 'type': 'changesetexplicit' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1005; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) updating the branch cache new changesets 3390ef850073:97765fc3cd62 updating to branch default resolving manifests branchmerge: False, force: False, partial: False ancestor: 000000000000, local: 000000000000+, remote: 97765fc3cd62 a: remote created -> g getting a b: remote created -> g getting b dir0/c: remote created -> g getting dir0/c dir0/d: remote created -> g getting dir0/d dir1/e: remote created -> g getting dir1/e dir1/f: remote created -> g getting dir1/f 6 files updated, 0 files merged, 0 files removed, 0 files unresolved (sent 6 HTTP requests and * bytes; received * bytes in responses) (glob) Incremental pull of shallow clone fetches new changesets $ hg --cwd client-shallow-2 --debug pull http://localhost:$HGPORT pulling from http://localhost:$HGPORT/ using http://localhost:$HGPORT/ sending capabilities command query 1; heads sending 2 commands sending command heads: {} sending command known: { 'nodes': [ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.' ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=22; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) received frame(size=11; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=2; request=3; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=3; stream=2; streamflags=; type=command-response; flags=eos) searching for changes all local heads known remotely sending 1 commands sending command changesetdata: { 'fields': set([ 'bookmarks', 'parents', 'phase', 'revision' ]), 'revisions': [ { 'heads': [ '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:' ], 'roots': [ '\x97v_\xc3\xcdbO\xd1\xfa\x01v\x93,!\xff\xd1j\xdfC.' ], 'type': 'changesetdagrange' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=400; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) add changeset dc666cf9ecf3 add changeset 93a8bd067ed2 checking for updated bookmarks sending 1 commands sending command manifestdata: { 'fields': set([ 'parents', 'revision' ]), 'haveparents': True, 'nodes': [ 'H]O\xc2`\xef\\\xb9\xc0p6\x88K\x00k\x11\x0ej\xdby', '\xd9;\xc4\x0b\x0e*GMp\xee\xf7}^\x91/f\x7fSd\x83' ], 'tree': '' } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=561; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) sending 1 commands sending command filesdata: { 'fields': set([ 'linknode', 'parents', 'revision' ]), 'haveparents': False, 'revisions': [ { 'nodes': [ '\xdcfl\xf9\xec\xf3\xd9Nk\x83\x0f0\xe5\xf1\'.*\x91d\xd9', '\x93\xa8\xbd\x06~\xd2\x84\r\x9a\xa8\x10\xadY\x81h8::,:' ], 'type': 'changesetexplicit' } ] } received frame(size=9; request=1; stream=2; streamflags=stream-begin; type=stream-settings; flags=eos) received frame(size=11; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=1373; request=1; stream=2; streamflags=encoded; type=command-response; flags=continuation) received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos) updating the branch cache new changesets dc666cf9ecf3:93a8bd067ed2 (run 'hg update' to get a working copy) (sent 5 HTTP requests and * bytes; received * bytes in responses) (glob) $ hg --cwd client-shallow-2 up tip merging dir0/c merging dir1/e 3 files updated, 2 files merged, 0 files removed, 0 files unresolved