Mercurial > hg
annotate tests/test-largefiles-small-disk.t @ 28000:d4247c306d82
copies: optimize forward copy detection logic for rebases
Forward copy detection (i.e. detecting what files have been moved/copied in
commit X since ancestor Y) previously required diff'ing the manifests of both X
and Y. This was expensive since it required reading both entire manifests and
doing a set difference (they weren't already in a set because of the
lazymanifest work). This cost almost 1 second on very large repositories, and
happens N times for a rebase of N commits.
This patch optimizes it for the case of rebase. In a rebase, we are comparing a
commit against it's immediate parent, and therefore we can know what files
changed by looking at ctx.files(). This lets us drastically decrease the size
of the set comparison, and makes it O(# of changes) instead of O(size of
manifest). This makes it take 1ms instead of 1000ms.
author | Durham Goode <durham@fb.com> |
---|---|
date | Fri, 05 Feb 2016 13:23:24 -0800 |
parents | 8a021cd38719 |
children | 7356e6b1f5b8 |
rev | line source |
---|---|
15571
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
1 Test how largefiles abort in case the disk runs full |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
2 |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
3 $ cat > criple.py <<EOF |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
4 > import os, errno, shutil |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
5 > from mercurial import util |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
6 > # |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
7 > # this makes the original largefiles code abort: |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
8 > def copyfileobj(fsrc, fdst, length=16*1024): |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
9 > fdst.write(fsrc.read(4)) |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
10 > raise IOError(errno.ENOSPC, os.strerror(errno.ENOSPC)) |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
11 > shutil.copyfileobj = copyfileobj |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
12 > # |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
13 > # this makes the rewritten code abort: |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
14 > def filechunkiter(f, size=65536, limit=None): |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
15 > yield f.read(4) |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
16 > raise IOError(errno.ENOSPC, os.strerror(errno.ENOSPC)) |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
17 > util.filechunkiter = filechunkiter |
15572
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
18 > # |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
19 > def oslink(src, dest): |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
20 > raise OSError("no hardlinks, try copying instead") |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
21 > util.oslink = oslink |
15571
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
22 > EOF |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
23 |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
24 $ echo "[extensions]" >> $HGRCPATH |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
25 $ echo "largefiles =" >> $HGRCPATH |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
26 |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
27 $ hg init alice |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
28 $ cd alice |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
29 $ echo "this is a very big file" > big |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
30 $ hg add --large big |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
31 $ hg commit --config extensions.criple=$TESTTMP/criple.py -m big |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
32 abort: No space left on device |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
33 [255] |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
34 |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
35 The largefile is not created in .hg/largefiles: |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
36 |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
37 $ ls .hg/largefiles |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
38 dirstate |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
39 |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
40 The user cache is not even created: |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
41 |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
42 >>> import os; os.path.exists("$HOME/.cache/largefiles/") |
809788118aa2
largefiles: write .hg/largefiles/ files atomically
Martin Geisler <mg@aragost.com>
parents:
diff
changeset
|
43 False |
15572
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
44 |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
45 Make the commit with space on the device: |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
46 |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
47 $ hg commit -m big |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
48 |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
49 Now make a clone with a full disk, and make sure lfutil.link function |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
50 makes copies instead of hardlinks: |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
51 |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
52 $ cd .. |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
53 $ hg --config extensions.criple=$TESTTMP/criple.py clone --pull alice bob |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
54 requesting all changes |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
55 adding changesets |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
56 adding manifests |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
57 adding file changes |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
58 added 1 changesets with 1 changes to 1 files |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
59 updating to branch default |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
60 getting changed largefiles |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
61 abort: No space left on device |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
62 [255] |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
63 |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
64 The largefile is not created in .hg/largefiles: |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
65 |
926bc23d0b6a
largefiles: copy files into .hg/largefiles atomically
Martin Geisler <mg@aragost.com>
parents:
15571
diff
changeset
|
66 $ ls bob/.hg/largefiles |
20063
8a021cd38719
largefiles: update in two steps, handle interrupted updates better
Mads Kiilerich <madski@unity3d.com>
parents:
18459
diff
changeset
|
67 dirstate |