view tests/test-largefiles-misc.t @ 42044:bb271ec2fbfb

compression: introduce a `storage.revlog.zstd.level` configuration

This option control the zstd compression level used when compressing revlog
chunk. The usage of zstd for revlog compression has not graduated from
experimental yet, but we intend to fix that soon.

The option name for the compression level is more straight forward to pick, so
this changesets comes first.  Having a dedicated option for each compression
engine is useful because they don't support the same range of values.

I ran the same measurement as for the zlib compression level (in the parent
changesets). The variation in repository size is stay mostly in the same (small)
range. The "read/write" performance see smallish variation, but are overall much
better than zlib. Write performance show the same tend of having better write
performance for when reaching high-end compression.

Again, we don't intend to change the default zstd compression level (currently:
3) in this series. However this is worth investigating in the future.

The Performance comparison of zlib vs zstd is quite impressive. The repository
size stay in the same range, but the performance are much better in all
situations.

Comparison summary
==================

We are looking at:
- performance range for zlib
- performance range for zstd
- comparison of default zstd (level-3) to default zlib (level 6)
- comparison of the slowest zstd time to the fastest zlib time

Read performance:
-----------------
          |           zlib          |           zstd          | cmp | f2s
mercurial |   0.170159 -   0.189219 |   0.144127 -   0.149624 | 80% | 88%
pypy      |   2.679217 -   2.768691 |   1.532317 -   1.705044 | 60% | 63%
netbeans  | 122.477027 - 141.620281 |  72.996346 -  89.731560 | 58% | 73%
mozilla   | 147.867662 - 170.572118 |  91.700995 - 105.853099 | 56% | 71%

Write performance:
------------------
          |           zlib          |           zstd          | cmp | f2s
mercurial |  53.250304 - 56.2936129 |  40.877025 -  45.677286 | 75% | 86%
pypy      | 460.721984 - 476.589918 | 270.545409 - 301.002219 | 63% | 65%
netbeans  | 520.560316 - 715.930400 | 370.356311 - 428.329652 | 55% | 82%
mozilla   | 739.803002 - 987.056093 | 505.152906 - 591.930683 | 57% | 80%

Raw data
--------

repo      alg lvl  .hg/store size  00manifest.d read       write

mercurial zlib  1      49,402,813     5,963,475   0.170159  53.250304
mercurial zlib  6      47,197,397     5,875,730   0.182820  56.264320
mercurial zlib  9      47,121,596     5,849,781   0.189219  56.293612

mercurial zstd  1      49,737,084     5,966,355   0.144127  40.877025
mercurial zstd  3      48,961,867     5,895,208   0.146376  42.268142
mercurial zstd  5      48,200,592     5,938,676   0.149624  43.162875
mercurial zstd 10      47,833,520     5,913,353   0.145185  44.012489
mercurial zstd 15      47,314,604     5,728,679   0.147686  45.677286
mercurial zstd 20      47,330,502     5,830,539   0.145789  45.025407
mercurial zstd 22      47,330,076     5,830,539   0.143996  44.690460


pypy      zlib  1     370,830,572    28,462,425   2.679217 460.721984
pypy      zlib  6     340,112,317    27,648,747   2.768691 467.537158
pypy      zlib  9     338,360,736    27,639,003   2.763495 476.589918

pypy      zstd  1     362,377,479    27,916,214   1.532317 270.545409
pypy      zstd  3     354,137,693    27,905,988   1.686718 294.951509
pypy      zstd  5     342,640,043    27,655,774   1.705044 301.002219
pypy      zstd 10     334,224,327    27,164,493   1.567287 285.186239
pypy      zstd 15     329,000,363    26,645,965   1.637729 299.561332
pypy      zstd 20     324,534,039    26,199,547   1.526813 302.149827
pypy      zstd 22     324,530,595    26,198,932   1.525718 307.821218


netbeans  zlib  1   1,281,847,810   165,495,457 122.477027 520.560316
netbeans  zlib  6   1,205,284,353   159,161,207 139.876147 715.930400
netbeans  zlib  9   1,197,135,671   155,034,586 141.620281 678.297064

netbeans  zstd  1   1,259,581,737   160,840,613  72.996346 370.356311
netbeans  zstd  3   1,232,978,122   157,691,551  81.622317 396.733087
netbeans  zstd  5   1,208,034,075   160,246,880  83.080549 364.342626
netbeans  zstd 10   1,188,624,176   156,083,417  79.323935 403.594602
netbeans  zstd 15   1,176,973,589   153,859,477  89.731560 428.329652
netbeans  zstd 20   1,162,958,258   151,147,535  82.842667 392.335349
netbeans  zstd 22   1,162,707,029   151,150,220  82.565695 402.840655


mozilla   zlib  1   2,775,497,186   298,527,987 147.867662 751.263721
mozilla   zlib  6   2,596,856,420   286,597,671 170.572118 987.056093
mozilla   zlib  9   2,587,542,494   287,018,264 163.622338 739.803002

mozilla   zstd  1   2,723,159,348   286,617,532  91.700995 570.042751
mozilla   zstd  3   2,665,055,001   286,152,013  95.240155 561.412805
mozilla   zstd  5   2,607,819,817   288,060,030 101.978048 505.152906
mozilla   zstd 10   2,558,761,085   283,967,648 104.113481 497.771202
mozilla   zstd 15   2,526,216,060   275,581,300 105.853099 591.930683
mozilla   zstd 20   2,485,114,806   266,478,859  95.268795 576.515389
mozilla   zstd 22   2,484,869,080   266,456,505  94.429282 572.785537

This file contains testcases that tend to be related to special cases or less
common commands affecting largefile.

  $ hg init requirements
  $ cd requirements

# largefiles not loaded by default.

  $ hg config extensions
  [1]

# Adding largefiles to requires file will auto-load largefiles extension.

  $ echo largefiles >> .hg/requires
  $ hg config extensions
  extensions.largefiles=

# But only if there is no config entry for the extension already.

  $ cat > .hg/hgrc << EOF
  > [extensions]
  > largefiles=!
  > EOF

  $ hg config extensions
  abort: repository requires features unknown to this Mercurial: largefiles!
  (see https://mercurial-scm.org/wiki/MissingRequirement for more information)
  [255]

  $ cat > .hg/hgrc << EOF
  > [extensions]
  > largefiles=
  > EOF

  $ hg config extensions
  extensions.largefiles=

  $ cat > .hg/hgrc << EOF
  > [extensions]
  > largefiles = missing.py
  > EOF

  $ hg config extensions
  *** failed to import extension largefiles from missing.py: [Errno 2] $ENOENT$: 'missing.py'
  abort: repository requires features unknown to this Mercurial: largefiles!
  (see https://mercurial-scm.org/wiki/MissingRequirement for more information)
  [255]

  $ cd ..

Each sections should be independent of each others.

  $ USERCACHE="$TESTTMP/cache"; export USERCACHE
  $ mkdir "${USERCACHE}"
  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > largefiles=
  > purge=
  > rebase=
  > transplant=
  > [phases]
  > publish=False
  > [largefiles]
  > minsize=2
  > patterns=glob:**.dat
  > usercache=${USERCACHE}
  > [hooks]
  > precommit=sh -c "echo \\"Invoking status precommit hook\\"; hg status"
  > EOF



Test copies and moves from a directory other than root (issue3516)
=========================================================================

  $ hg init lf_cpmv
  $ cd lf_cpmv
  $ mkdir dira
  $ mkdir dira/dirb
  $ touch dira/dirb/largefile
  $ hg add --large dira/dirb/largefile
  $ hg commit -m "added"
  Invoking status precommit hook
  A dira/dirb/largefile
  $ cd dira
  $ hg cp dirb/largefile foo/largefile

TODO: Ideally, this should mention the largefile, not the standin
  $ hg log -T '{rev}\n' --stat 'set:clean()'
  0
   .hglf/dira/dirb/largefile |  1 +
   1 files changed, 1 insertions(+), 0 deletions(-)
  
  $ hg ci -m "deep copy"
  Invoking status precommit hook
  A dira/foo/largefile
  $ find . | sort
  .
  ./dirb
  ./dirb/largefile
  ./foo
  ./foo/largefile
  $ hg mv foo/largefile baz/largefile
  $ hg ci -m "moved"
  Invoking status precommit hook
  A dira/baz/largefile
  R dira/foo/largefile
  $ find . | sort
  .
  ./baz
  ./baz/largefile
  ./dirb
  ./dirb/largefile
  $ cd ..
  $ hg mv dira dirc
  moving .hglf/dira/baz/largefile to .hglf/dirc/baz/largefile
  moving .hglf/dira/dirb/largefile to .hglf/dirc/dirb/largefile
  $ find * | sort
  dirc
  dirc/baz
  dirc/baz/largefile
  dirc/dirb
  dirc/dirb/largefile

  $ hg clone -q . ../fetch
  $ hg --config extensions.fetch= fetch ../fetch
  abort: uncommitted changes
  [255]
  $ hg up -qC
  $ cd ..

Clone a local repository owned by another user
===================================================

#if unix-permissions

We have to simulate that here by setting $HOME and removing write permissions
  $ ORIGHOME="$HOME"
  $ mkdir alice
  $ HOME="`pwd`/alice"
  $ cd alice
  $ hg init pubrepo
  $ cd pubrepo
  $ dd if=/dev/zero bs=1k count=11k > a-large-file 2> /dev/null
  $ hg add --large a-large-file
  $ hg commit -m "Add a large file"
  Invoking status precommit hook
  A a-large-file
  $ cd ..
  $ chmod -R a-w pubrepo
  $ cd ..
  $ mkdir bob
  $ HOME="`pwd`/bob"
  $ cd bob
  $ hg clone --pull ../alice/pubrepo pubrepo
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets 09a186cfa6da (1 drafts)
  updating to branch default
  getting changed largefiles
  1 largefiles updated, 0 removed
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd ..
  $ chmod -R u+w alice/pubrepo
  $ HOME="$ORIGHOME"

#endif


Symlink to a large largefile should behave the same as a symlink to a normal file
=====================================================================================

#if symlink

  $ hg init largesymlink
  $ cd largesymlink
  $ dd if=/dev/zero bs=1k count=10k of=largefile 2>/dev/null
  $ hg add --large largefile
  $ hg commit -m "commit a large file"
  Invoking status precommit hook
  A largefile
  $ ln -s largefile largelink
  $ hg add largelink
  $ hg commit -m "commit a large symlink"
  Invoking status precommit hook
  A largelink
  $ rm -f largelink
  $ hg up >/dev/null
  $ test -f largelink
  [1]
  $ test -L largelink
  [1]
  $ rm -f largelink # make next part of the test independent of the previous
  $ hg up -C >/dev/null
  $ test -f largelink
  $ test -L largelink
  $ cd ..

#endif


test for pattern matching on 'hg status':
==============================================


to boost performance, largefiles checks whether specified patterns are
related to largefiles in working directory (NOT to STANDIN) or not.

  $ hg init statusmatch
  $ cd statusmatch

  $ mkdir -p a/b/c/d
  $ echo normal > a/b/c/d/e.normal.txt
  $ hg add a/b/c/d/e.normal.txt
  $ echo large > a/b/c/d/e.large.txt
  $ hg add --large a/b/c/d/e.large.txt
  $ mkdir -p a/b/c/x
  $ echo normal > a/b/c/x/y.normal.txt
  $ hg add a/b/c/x/y.normal.txt
  $ hg commit -m 'add files'
  Invoking status precommit hook
  A a/b/c/d/e.large.txt
  A a/b/c/d/e.normal.txt
  A a/b/c/x/y.normal.txt

(1) no pattern: no performance boost
  $ hg status -A
  C a/b/c/d/e.large.txt
  C a/b/c/d/e.normal.txt
  C a/b/c/x/y.normal.txt

(2) pattern not related to largefiles: performance boost
  $ hg status -A a/b/c/x
  C a/b/c/x/y.normal.txt

(3) pattern related to largefiles: no performance boost
  $ hg status -A a/b/c/d
  C a/b/c/d/e.large.txt
  C a/b/c/d/e.normal.txt

(4) pattern related to STANDIN (not to largefiles): performance boost
  $ hg status -A .hglf/a
  C .hglf/a/b/c/d/e.large.txt

(5) mixed case: no performance boost
  $ hg status -A a/b/c/x a/b/c/d
  C a/b/c/d/e.large.txt
  C a/b/c/d/e.normal.txt
  C a/b/c/x/y.normal.txt

verify that largefiles doesn't break filesets

  $ hg log --rev . --exclude "set:binary()"
  changeset:   0:41bd42f10efa
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     add files
  
sharing a largefile repo automatically enables largefiles on the share

  $ hg share --config extensions.share= . ../shared_lfrepo
  updating working directory
  getting changed largefiles
  1 largefiles updated, 0 removed
  3 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ grep largefiles ../shared_lfrepo/.hg/requires
  largefiles

verify that large files in subrepos handled properly
  $ hg init subrepo
  $ echo "subrepo = subrepo" > .hgsub
  $ hg add .hgsub
  $ hg ci -m "add subrepo"
  Invoking status precommit hook
  A .hgsub
  ? .hgsubstate
  $ echo "rev 1" > subrepo/large.txt
  $ hg add --large subrepo/large.txt
  $ hg sum
  parent: 1:8ee150ea2e9c tip
   add subrepo
  branch: default
  commit: 1 subrepos
  update: (current)
  phases: 2 draft
  $ hg st
  $ hg st -S
  A subrepo/large.txt
  $ hg ci -S -m "commit top repo"
  committing subrepository subrepo
  Invoking status precommit hook
  A large.txt
  Invoking status precommit hook
  M .hgsubstate
# No differences
  $ hg st -S
  $ hg sum
  parent: 2:ce4cd0c527a6 tip
   commit top repo
  branch: default
  commit: (clean)
  update: (current)
  phases: 3 draft
  $ echo "rev 2" > subrepo/large.txt
  $ hg st -S
  M subrepo/large.txt
  $ hg sum
  parent: 2:ce4cd0c527a6 tip
   commit top repo
  branch: default
  commit: 1 subrepos
  update: (current)
  phases: 3 draft
  $ hg ci -m "this commit should fail without -S"
  abort: uncommitted changes in subrepository "subrepo"
  (use --subrepos for recursive commit)
  [255]

Add a normal file to the subrepo, then test archiving

  $ echo 'normal file' > subrepo/normal.txt
  $ touch large.dat
  $ mv subrepo/large.txt subrepo/renamed-large.txt
  $ hg addremove -S --dry-run
  adding large.dat as a largefile
  removing subrepo/large.txt
  adding subrepo/normal.txt
  adding subrepo/renamed-large.txt
  $ hg status -S
  ! subrepo/large.txt
  ? large.dat
  ? subrepo/normal.txt
  ? subrepo/renamed-large.txt

  $ hg addremove --dry-run subrepo
  removing subrepo/large.txt
  adding subrepo/normal.txt
  adding subrepo/renamed-large.txt
  $ hg status -S
  ! subrepo/large.txt
  ? large.dat
  ? subrepo/normal.txt
  ? subrepo/renamed-large.txt
  $ cd ..

  $ hg -R statusmatch addremove --dry-run statusmatch/subrepo
  removing statusmatch/subrepo/large.txt
  adding statusmatch/subrepo/normal.txt
  adding statusmatch/subrepo/renamed-large.txt
  $ hg -R statusmatch status -S
  ! subrepo/large.txt
  ? large.dat
  ? subrepo/normal.txt
  ? subrepo/renamed-large.txt

  $ hg -R statusmatch addremove --dry-run -S
  adding large.dat as a largefile
  removing subrepo/large.txt
  adding subrepo/normal.txt
  adding subrepo/renamed-large.txt
  $ cd statusmatch

  $ mv subrepo/renamed-large.txt subrepo/large.txt
  $ hg addremove subrepo
  adding subrepo/normal.txt
  $ hg forget subrepo/normal.txt

  $ hg addremove -S
  adding large.dat as a largefile
  adding subrepo/normal.txt
  $ rm large.dat

  $ hg addremove subrepo
  $ hg addremove -S
  removing large.dat

Lock in subrepo, otherwise the change isn't archived

  $ hg ci -S -m "add normal file to top level"
  committing subrepository subrepo
  Invoking status precommit hook
  M large.txt
  A normal.txt
  Invoking status precommit hook
  M .hgsubstate
  $ hg archive -S ../lf_subrepo_archive
  $ find ../lf_subrepo_archive | sort
  ../lf_subrepo_archive
  ../lf_subrepo_archive/.hg_archival.txt
  ../lf_subrepo_archive/.hgsub
  ../lf_subrepo_archive/.hgsubstate
  ../lf_subrepo_archive/a
  ../lf_subrepo_archive/a/b
  ../lf_subrepo_archive/a/b/c
  ../lf_subrepo_archive/a/b/c/d
  ../lf_subrepo_archive/a/b/c/d/e.large.txt
  ../lf_subrepo_archive/a/b/c/d/e.normal.txt
  ../lf_subrepo_archive/a/b/c/x
  ../lf_subrepo_archive/a/b/c/x/y.normal.txt
  ../lf_subrepo_archive/subrepo
  ../lf_subrepo_archive/subrepo/large.txt
  ../lf_subrepo_archive/subrepo/normal.txt
  $ cat ../lf_subrepo_archive/.hg_archival.txt
  repo: 41bd42f10efa43698cc02052ea0977771cba506d
  node: d56a95e6522858bc08a724c4fe2bdee066d1c30b
  branch: default
  latesttag: null
  latesttagdistance: 4
  changessincelatesttag: 4

Test update with subrepos.

  $ hg update 0
  getting changed largefiles
  0 largefiles updated, 1 removed
  0 files updated, 0 files merged, 2 files removed, 0 files unresolved
  $ hg status -S
  $ hg update tip
  getting changed largefiles
  1 largefiles updated, 0 removed
  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ hg status -S
# modify a large file
  $ echo "modified" > subrepo/large.txt
  $ hg st -S
  M subrepo/large.txt
# update -C should revert the change.
  $ hg update -C
  getting changed largefiles
  1 largefiles updated, 0 removed
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ hg status -S

Forget doesn't change the content of the file
  $ echo 'pre-forget content' > subrepo/large.txt
  $ hg forget -v subrepo/large.txt
  removing subrepo/large.txt
  $ cat subrepo/large.txt
  pre-forget content

Test reverting a forgotten file
  $ hg revert -R subrepo subrepo/large.txt
  $ hg status -SA subrepo/large.txt
  C subrepo/large.txt

  $ hg rm -v subrepo/large.txt
  removing subrepo/large.txt
  $ hg revert -R subrepo subrepo/large.txt
  $ rm subrepo/large.txt
  $ hg addremove -S
  removing subrepo/large.txt
  $ hg st -S
  R subrepo/large.txt

Test archiving a revision that references a subrepo that is not yet
cloned (see test-subrepo-recursion.t):

  $ hg clone -U . ../empty
  $ cd ../empty
  $ hg archive --subrepos -r tip ../archive.tar.gz
  cloning subrepo subrepo from $TESTTMP/statusmatch/subrepo
  $ cd ..






Test addremove, forget and others
==============================================

Test that addremove picks up largefiles prior to the initial commit (issue3541)

  $ hg init addrm2
  $ cd addrm2
  $ touch large.dat
  $ touch large2.dat
  $ touch normal
  $ hg add --large large.dat
  $ hg addremove -v
  adding large2.dat as a largefile
  adding normal

Test that forgetting all largefiles reverts to islfilesrepo() == False
(addremove will add *.dat as normal files now)
  $ hg forget large.dat
  $ hg forget large2.dat
  $ hg addremove -v
  adding large.dat
  adding large2.dat

Test commit's addremove option prior to the first commit
  $ hg forget large.dat
  $ hg forget large2.dat
  $ hg add --large large.dat
  $ hg ci -Am "commit"
  adding large2.dat as a largefile
  Invoking status precommit hook
  A large.dat
  A large2.dat
  A normal
  $ find .hglf | sort
  .hglf
  .hglf/large.dat
  .hglf/large2.dat

Test actions on largefiles using relative paths from subdir

  $ mkdir sub
  $ cd sub
  $ echo anotherlarge > anotherlarge
  $ hg add --large anotherlarge
  $ hg st
  A sub/anotherlarge
  $ hg st anotherlarge
  A anotherlarge
  $ hg commit -m anotherlarge anotherlarge
  Invoking status precommit hook
  A sub/anotherlarge
  $ hg log anotherlarge
  changeset:   1:9627a577c5e9
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     anotherlarge
  
  $ hg --debug log -T '{rev}: {desc}\n' ../sub/anotherlarge
  updated patterns: ../.hglf/sub/../sub/anotherlarge, ../sub/anotherlarge
  1: anotherlarge

  $ hg log -G anotherlarge
  @  changeset:   1:9627a577c5e9
  |  tag:         tip
  ~  user:        test
     date:        Thu Jan 01 00:00:00 1970 +0000
     summary:     anotherlarge
  

  $ hg log glob:another*
  changeset:   1:9627a577c5e9
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     anotherlarge
  
  $ hg --debug log -T '{rev}: {desc}\n' -G glob:another*
  updated patterns: glob:../.hglf/sub/another*, glob:another*
  @  1: anotherlarge
  |
  ~

#if no-msys
  $ hg --debug log -T '{rev}: {desc}\n' 'glob:../.hglf/sub/another*' # no-msys
  updated patterns: glob:../.hglf/sub/another*
  1: anotherlarge

  $ hg --debug log -G -T '{rev}: {desc}\n' 'glob:../.hglf/sub/another*' # no-msys
  updated patterns: glob:../.hglf/sub/another*
  @  1: anotherlarge
  |
  ~
#endif

  $ echo more >> anotherlarge
  $ hg st .
  M anotherlarge
  $ hg cat anotherlarge
  anotherlarge
  $ hg revert anotherlarge
  $ hg st
  ? sub/anotherlarge.orig

Test orig files go where we want them
  $ echo moremore >> anotherlarge
  $ hg revert anotherlarge -v --config 'ui.origbackuppath=.hg/origbackups'
  creating directory: $TESTTMP/addrm2/.hg/origbackups/.hglf/sub
  saving current version of ../.hglf/sub/anotherlarge as ../.hg/origbackups/.hglf/sub/anotherlarge
  reverting ../.hglf/sub/anotherlarge
  creating directory: $TESTTMP/addrm2/.hg/origbackups/sub
  found 90c622cf65cebe75c5842f9136c459333faf392e in store
  found 90c622cf65cebe75c5842f9136c459333faf392e in store
  $ ls ../.hg/origbackups/sub
  anotherlarge
  $ cd ..

Test glob logging from the root dir
  $ hg log glob:**another*
  changeset:   1:9627a577c5e9
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     anotherlarge
  
  $ hg log -G glob:**another*
  @  changeset:   1:9627a577c5e9
  |  tag:         tip
  ~  user:        test
     date:        Thu Jan 01 00:00:00 1970 +0000
     summary:     anotherlarge
  

  $ cd ..

Log from outer space
  $ hg --debug log -R addrm2 -T '{rev}: {desc}\n' 'addrm2/sub/anotherlarge'
  updated patterns: addrm2/.hglf/sub/anotherlarge, addrm2/sub/anotherlarge
  1: anotherlarge
  $ hg --debug log -R addrm2 -T '{rev}: {desc}\n' 'addrm2/.hglf/sub/anotherlarge'
  updated patterns: addrm2/.hglf/sub/anotherlarge
  1: anotherlarge


Check error message while exchange
=========================================================

issue3651: summary/outgoing with largefiles shows "no remote repo"
unexpectedly

  $ mkdir issue3651
  $ cd issue3651

  $ hg init src
  $ echo a > src/a
  $ hg -R src add --large src/a
  $ hg -R src commit -m '#0'
  Invoking status precommit hook
  A a

check messages when no remote repository is specified:
"no remote repo" route for "hg outgoing --large" is not tested here,
because it can't be reproduced easily.

  $ hg init clone1
  $ hg -R clone1 -q pull src
  $ hg -R clone1 -q update
  $ hg -R clone1 paths | grep default
  [1]

  $ hg -R clone1 summary --large
  parent: 0:fc0bd45326d3 tip
   #0
  branch: default
  commit: (clean)
  update: (current)
  phases: 1 draft
  largefiles: (no remote repo)

check messages when there is no files to upload:

  $ hg -q clone src clone2
  $ hg -R clone2 paths | grep default
  default = $TESTTMP/issue3651/src

  $ hg -R clone2 summary --large
  parent: 0:fc0bd45326d3 tip
   #0
  branch: default
  commit: (clean)
  update: (current)
  phases: 1 draft
  largefiles: (no files to upload)
  $ hg -R clone2 outgoing --large
  comparing with $TESTTMP/issue3651/src
  searching for changes
  no changes found
  largefiles: no files to upload
  [1]

  $ hg -R clone2 outgoing --large --graph --template "{rev}"
  comparing with $TESTTMP/issue3651/src
  searching for changes
  no changes found
  largefiles: no files to upload

check messages when there are files to upload:

  $ echo b > clone2/b
  $ hg -R clone2 add --large clone2/b
  $ hg -R clone2 commit -m '#1'
  Invoking status precommit hook
  A b
  $ hg -R clone2 summary --large
  parent: 1:1acbe71ce432 tip
   #1
  branch: default
  commit: (clean)
  update: (current)
  phases: 2 draft
  largefiles: 1 entities for 1 files to upload
  $ hg -R clone2 outgoing --large
  comparing with $TESTTMP/issue3651/src
  searching for changes
  changeset:   1:1acbe71ce432
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     #1
  
  largefiles to upload (1 entities):
  b
  
  $ hg -R clone2 outgoing --large --graph --template "{rev}"
  comparing with $TESTTMP/issue3651/src
  searching for changes
  @  1
  
  largefiles to upload (1 entities):
  b
  

  $ cp clone2/b clone2/b1
  $ cp clone2/b clone2/b2
  $ hg -R clone2 add --large clone2/b1 clone2/b2
  $ hg -R clone2 commit -m '#2: add largefiles referring same entity'
  Invoking status precommit hook
  A b1
  A b2
  $ hg -R clone2 summary --large
  parent: 2:6095d0695d70 tip
   #2: add largefiles referring same entity
  branch: default
  commit: (clean)
  update: (current)
  phases: 3 draft
  largefiles: 1 entities for 3 files to upload
  $ hg -R clone2 outgoing --large -T "{rev}:{node|short}\n"
  comparing with $TESTTMP/issue3651/src
  searching for changes
  1:1acbe71ce432
  2:6095d0695d70
  largefiles to upload (1 entities):
  b
  b1
  b2
  
  $ hg -R clone2 cat -r 1 clone2/.hglf/b
  89e6c98d92887913cadf06b2adb97f26cde4849b
  $ hg -R clone2 outgoing --large -T "{rev}:{node|short}\n" --debug --config progress.debug=true
  comparing with $TESTTMP/issue3651/src
  query 1; heads
  searching for changes
  all remote heads known locally
  1:1acbe71ce432
  2:6095d0695d70
  finding outgoing largefiles: 0/2 revisions (0.00%)
  finding outgoing largefiles: 1/2 revisions (50.00%)
  largefiles to upload (1 entities):
  b
      89e6c98d92887913cadf06b2adb97f26cde4849b
  b1
      89e6c98d92887913cadf06b2adb97f26cde4849b
  b2
      89e6c98d92887913cadf06b2adb97f26cde4849b
  

  $ echo bbb > clone2/b
  $ hg -R clone2 commit -m '#3: add new largefile entity as existing file'
  Invoking status precommit hook
  M b
  $ echo bbbb > clone2/b
  $ hg -R clone2 commit -m '#4: add new largefile entity as existing file'
  Invoking status precommit hook
  M b
  $ cp clone2/b1 clone2/b
  $ hg -R clone2 commit -m '#5: refer existing largefile entity again'
  Invoking status precommit hook
  M b
  $ hg -R clone2 summary --large
  parent: 5:036794ea641c tip
   #5: refer existing largefile entity again
  branch: default
  commit: (clean)
  update: (current)
  phases: 6 draft
  largefiles: 3 entities for 3 files to upload
  $ hg -R clone2 outgoing --large -T "{rev}:{node|short}\n"
  comparing with $TESTTMP/issue3651/src
  searching for changes
  1:1acbe71ce432
  2:6095d0695d70
  3:7983dce246cc
  4:233f12ada4ae
  5:036794ea641c
  largefiles to upload (3 entities):
  b
  b1
  b2
  
  $ hg -R clone2 cat -r 3 clone2/.hglf/b
  c801c9cfe94400963fcb683246217d5db77f9a9a
  $ hg -R clone2 cat -r 4 clone2/.hglf/b
  13f9ed0898e315bf59dc2973fec52037b6f441a2
  $ hg -R clone2 outgoing --large -T "{rev}:{node|short}\n" --debug --config progress.debug=true
  comparing with $TESTTMP/issue3651/src
  query 1; heads
  searching for changes
  all remote heads known locally
  1:1acbe71ce432
  2:6095d0695d70
  3:7983dce246cc
  4:233f12ada4ae
  5:036794ea641c
  finding outgoing largefiles: 0/5 revisions (0.00%)
  finding outgoing largefiles: 1/5 revisions (20.00%)
  finding outgoing largefiles: 2/5 revisions (40.00%)
  finding outgoing largefiles: 3/5 revisions (60.00%)
  finding outgoing largefiles: 4/5 revisions (80.00%)
  largefiles to upload (3 entities):
  b
      13f9ed0898e315bf59dc2973fec52037b6f441a2
      89e6c98d92887913cadf06b2adb97f26cde4849b
      c801c9cfe94400963fcb683246217d5db77f9a9a
  b1
      89e6c98d92887913cadf06b2adb97f26cde4849b
  b2
      89e6c98d92887913cadf06b2adb97f26cde4849b
  

Pushing revision #1 causes uploading entity 89e6c98d9288, which is
shared also by largefiles b1, b2 in revision #2 and b in revision #5.

Then, entity 89e6c98d9288 is not treated as "outgoing entity" at "hg
summary" and "hg outgoing", even though files in outgoing revision #2
and #5 refer it.

  $ hg -R clone2 push -r 1 -q
  $ hg -R clone2 summary --large
  parent: 5:036794ea641c tip
   #5: refer existing largefile entity again
  branch: default
  commit: (clean)
  update: (current)
  phases: 6 draft
  largefiles: 2 entities for 1 files to upload
  $ hg -R clone2 outgoing --large -T "{rev}:{node|short}\n"
  comparing with $TESTTMP/issue3651/src
  searching for changes
  2:6095d0695d70
  3:7983dce246cc
  4:233f12ada4ae
  5:036794ea641c
  largefiles to upload (2 entities):
  b
  
  $ hg -R clone2 outgoing --large -T "{rev}:{node|short}\n" --debug --config progress.debug=true
  comparing with $TESTTMP/issue3651/src
  query 1; heads
  searching for changes
  all remote heads known locally
  2:6095d0695d70
  3:7983dce246cc
  4:233f12ada4ae
  5:036794ea641c
  finding outgoing largefiles: 0/4 revisions (0.00%)
  finding outgoing largefiles: 1/4 revisions (25.00%)
  finding outgoing largefiles: 2/4 revisions (50.00%)
  finding outgoing largefiles: 3/4 revisions (75.00%)
  largefiles to upload (2 entities):
  b
      13f9ed0898e315bf59dc2973fec52037b6f441a2
      c801c9cfe94400963fcb683246217d5db77f9a9a
  

  $ cd ..

merge action 'd' for 'local renamed directory to d2/g' which has no filename
==================================================================================

  $ hg init merge-action
  $ cd merge-action
  $ touch l
  $ hg add --large l
  $ mkdir d1
  $ touch d1/f
  $ hg ci -Aqm0
  Invoking status precommit hook
  A d1/f
  A l
  $ echo > d1/f
  $ touch d1/g
  $ hg ci -Aqm1
  Invoking status precommit hook
  M d1/f
  A d1/g
  $ hg up -qr0
  $ hg mv d1 d2
  moving d1/f to d2/f
  $ hg ci -qm2
  Invoking status precommit hook
  A d2/f
  R d1/f
  $ hg merge
  merging d2/f and d1/f to d2/f
  1 files updated, 1 files merged, 0 files removed, 0 files unresolved
  (branch merge, don't forget to commit)
  $ cd ..


Merge conflicts:
=====================

  $ hg init merge
  $ cd merge
  $ echo 0 > f-different
  $ echo 0 > f-same
  $ echo 0 > f-unchanged-1
  $ echo 0 > f-unchanged-2
  $ hg add --large *
  $ hg ci -m0
  Invoking status precommit hook
  A f-different
  A f-same
  A f-unchanged-1
  A f-unchanged-2
  $ echo tmp1 > f-unchanged-1
  $ echo tmp1 > f-unchanged-2
  $ echo tmp1 > f-same
  $ hg ci -m1
  Invoking status precommit hook
  M f-same
  M f-unchanged-1
  M f-unchanged-2
  $ echo 2 > f-different
  $ echo 0 > f-unchanged-1
  $ echo 1 > f-unchanged-2
  $ echo 1 > f-same
  $ hg ci -m2
  Invoking status precommit hook
  M f-different
  M f-same
  M f-unchanged-1
  M f-unchanged-2
  $ hg up -qr0
  $ echo tmp2 > f-unchanged-1
  $ echo tmp2 > f-unchanged-2
  $ echo tmp2 > f-same
  $ hg ci -m3
  Invoking status precommit hook
  M f-same
  M f-unchanged-1
  M f-unchanged-2
  created new head
  $ echo 1 > f-different
  $ echo 1 > f-unchanged-1
  $ echo 0 > f-unchanged-2
  $ echo 1 > f-same
  $ hg ci -m4
  Invoking status precommit hook
  M f-different
  M f-same
  M f-unchanged-1
  M f-unchanged-2
  $ hg merge
  largefile f-different has a merge conflict
  ancestor was 09d2af8dd22201dd8d48e5dcfcaed281ff9422c7
  keep (l)ocal e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e or
  take (o)ther 7448d8798a4380162d4b56f9b452e2f6f9e24e7a? l
  getting changed largefiles
  1 largefiles updated, 0 removed
  0 files updated, 4 files merged, 0 files removed, 0 files unresolved
  (branch merge, don't forget to commit)
  $ cat f-different
  1
  $ cat f-same
  1
  $ cat f-unchanged-1
  1
  $ cat f-unchanged-2
  1
  $ cd ..

Test largefile insulation (do not enabled a side effect
========================================================

Check whether "largefiles" feature is supported only in repositories
enabling largefiles extension.

  $ mkdir individualenabling
  $ cd individualenabling

  $ hg init enabledlocally
  $ echo large > enabledlocally/large
  $ hg -R enabledlocally add --large enabledlocally/large
  $ hg -R enabledlocally commit -m '#0'
  Invoking status precommit hook
  A large

  $ hg init notenabledlocally
  $ echo large > notenabledlocally/large
  $ hg -R notenabledlocally add --large notenabledlocally/large
  $ hg -R notenabledlocally commit -m '#0'
  Invoking status precommit hook
  A large

  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > # disable globally
  > largefiles=!
  > EOF
  $ cat >> enabledlocally/.hg/hgrc <<EOF
  > [extensions]
  > # enable locally
  > largefiles=
  > EOF
  $ hg -R enabledlocally root
  $TESTTMP/individualenabling/enabledlocally
  $ hg -R notenabledlocally root
  abort: repository requires features unknown to this Mercurial: largefiles!
  (see https://mercurial-scm.org/wiki/MissingRequirement for more information)
  [255]

  $ hg init push-dst
  $ hg -R enabledlocally push push-dst
  pushing to push-dst
  abort: required features are not supported in the destination: largefiles
  [255]

  $ hg init pull-src
  $ hg -R pull-src pull enabledlocally
  pulling from enabledlocally
  abort: required features are not supported in the destination: largefiles
  [255]

  $ hg clone enabledlocally clone-dst
  abort: repository requires features unknown to this Mercurial: largefiles!
  (see https://mercurial-scm.org/wiki/MissingRequirement for more information)
  [255]
  $ test -d clone-dst
  [1]
  $ hg clone --pull enabledlocally clone-pull-dst
  abort: required features are not supported in the destination: largefiles
  [255]
  $ test -d clone-pull-dst
  [1]

#if serve

Test largefiles specific peer setup, when largefiles is enabled
locally (issue4109)

  $ hg showconfig extensions | grep largefiles
  extensions.largefiles=!
  $ mkdir -p $TESTTMP/individualenabling/usercache

  $ hg serve -R enabledlocally -d -p $HGPORT --pid-file hg.pid
  $ cat hg.pid >> $DAEMON_PIDS

  $ hg init pull-dst
  $ cat > pull-dst/.hg/hgrc <<EOF
  > [extensions]
  > # enable locally
  > largefiles=
  > [largefiles]
  > # ignore system cache to force largefiles specific wire proto access
  > usercache=$TESTTMP/individualenabling/usercache
  > EOF
  $ hg -R pull-dst -q pull -u http://localhost:$HGPORT

  $ killdaemons.py
#endif

Test overridden functions work correctly even for repos disabling
largefiles (issue4547)

  $ hg showconfig extensions | grep largefiles
  extensions.largefiles=!

(test updating implied by clone)

  $ hg init enabled-but-no-largefiles
  $ echo normal1 > enabled-but-no-largefiles/normal1
  $ hg -R enabled-but-no-largefiles add enabled-but-no-largefiles/normal1
  $ hg -R enabled-but-no-largefiles commit -m '#0@enabled-but-no-largefiles'
  Invoking status precommit hook
  A normal1
  $ cat >> enabled-but-no-largefiles/.hg/hgrc <<EOF
  > [extensions]
  > # enable locally
  > largefiles=
  > EOF
  $ hg clone -q enabled-but-no-largefiles no-largefiles

  $ echo normal2 > enabled-but-no-largefiles/normal2
  $ hg -R enabled-but-no-largefiles add enabled-but-no-largefiles/normal2
  $ hg -R enabled-but-no-largefiles commit -m '#1@enabled-but-no-largefiles'
  Invoking status precommit hook
  A normal2

  $ echo normal3 > no-largefiles/normal3
  $ hg -R no-largefiles add no-largefiles/normal3
  $ hg -R no-largefiles commit -m '#1@no-largefiles'
  Invoking status precommit hook
  A normal3

  $ hg -R no-largefiles -q pull --rebase
  Invoking status precommit hook
  A normal3

(test reverting)

  $ hg init subrepo-root
  $ cat >> subrepo-root/.hg/hgrc <<EOF
  > [extensions]
  > # enable locally
  > largefiles=
  > EOF
  $ echo large > subrepo-root/large
  $ mkdir -p subrepo-root/dir/subdir
  $ echo large2 > subrepo-root/dir/subdir/large.bin
  $ hg -R subrepo-root add --large subrepo-root/large subrepo-root/dir/subdir/large.bin
  $ hg clone -q no-largefiles subrepo-root/no-largefiles
  $ cat > subrepo-root/.hgsub <<EOF
  > no-largefiles = no-largefiles
  > EOF
  $ hg -R subrepo-root add subrepo-root/.hgsub
  $ hg -R subrepo-root commit -m '#0'
  Invoking status precommit hook
  A .hgsub
  A dir/subdir/large.bin
  A large
  ? .hgsubstate
  $ echo dirty >> subrepo-root/large
  $ echo dirty >> subrepo-root/no-largefiles/normal1
  $ hg -R subrepo-root status -S
  M large
  M no-largefiles/normal1
  $ hg -R subrepo-root extdiff -p echo -S --config extensions.extdiff=
  "*\\no-largefiles\\normal1" "*\\no-largefiles\\normal1" (glob) (windows !)
  */no-largefiles/normal1 */no-largefiles/normal1 (glob) (no-windows !)
  [1]
  $ hg -R subrepo-root revert --all
  reverting subrepo-root/.hglf/large
  reverting subrepo no-largefiles
  reverting subrepo-root/no-largefiles/normal1

Move (and then undo) a directory move with only largefiles.

  $ cd subrepo-root
  $ "$PYTHON" $TESTDIR/list-tree.py .hglf dir* large*
  .hglf/
  .hglf/dir/
  .hglf/dir/subdir/
  .hglf/dir/subdir/large.bin
  .hglf/large
  dir/
  dir/subdir/
  dir/subdir/large.bin
  large
  large.orig

  $ hg mv dir/subdir dir/subdir2
  moving .hglf/dir/subdir/large.bin to .hglf/dir/subdir2/large.bin

  $ "$PYTHON" $TESTDIR/list-tree.py .hglf dir* large*
  .hglf/
  .hglf/dir/
  .hglf/dir/subdir2/
  .hglf/dir/subdir2/large.bin
  .hglf/large
  dir/
  dir/subdir2/
  dir/subdir2/large.bin
  large
  large.orig
  $ hg status -C
  A dir/subdir2/large.bin
    dir/subdir/large.bin
  R dir/subdir/large.bin
  ? large.orig

  $ echo 'modified' > dir/subdir2/large.bin
  $ hg status -C
  A dir/subdir2/large.bin
    dir/subdir/large.bin
  R dir/subdir/large.bin
  ? large.orig

  $ hg revert --all
  forgetting .hglf/dir/subdir2/large.bin
  undeleting .hglf/dir/subdir/large.bin
  reverting subrepo no-largefiles

  $ hg status -C
  ? dir/subdir2/large.bin
  ? large.orig

The content of the forgotten file shouldn't be clobbered

  $ cat dir/subdir2/large.bin
  modified

The standin for subdir2 should be deleted, not just dropped

  $ "$PYTHON" $TESTDIR/list-tree.py .hglf dir* large*
  .hglf/
  .hglf/dir/
  .hglf/dir/subdir/
  .hglf/dir/subdir/large.bin
  .hglf/large
  dir/
  dir/subdir/
  dir/subdir/large.bin
  dir/subdir2/
  dir/subdir2/large.bin
  large
  large.orig

  $ rm -r dir/subdir2

'subdir' should not be in the destination.  It would be if the subdir2 directory
existed under .hglf/.
  $ hg mv dir/subdir dir/subdir2
  moving .hglf/dir/subdir/large.bin to .hglf/dir/subdir2/large.bin

  $ hg status -C
  A dir/subdir2/large.bin
    dir/subdir/large.bin
  R dir/subdir/large.bin
  ? large.orig

  $ "$PYTHON" $TESTDIR/list-tree.py .hglf dir* large*
  .hglf/
  .hglf/dir/
  .hglf/dir/subdir2/
  .hglf/dir/subdir2/large.bin
  .hglf/large
  dir/
  dir/subdir2/
  dir/subdir2/large.bin
  large
  large.orig

Start from scratch, and rename something other than the final path component.

  $ hg up -qC .
  $ hg --config extensions.purge= purge

  $ hg mv dir/subdir dir2/subdir
  moving .hglf/dir/subdir/large.bin to .hglf/dir2/subdir/large.bin

  $ hg status -C
  A dir2/subdir/large.bin
    dir/subdir/large.bin
  R dir/subdir/large.bin

  $ "$PYTHON" $TESTDIR/list-tree.py .hglf dir* large*
  .hglf/
  .hglf/dir2/
  .hglf/dir2/subdir/
  .hglf/dir2/subdir/large.bin
  .hglf/large
  dir2/
  dir2/subdir/
  dir2/subdir/large.bin
  large

  $ hg revert --all
  forgetting .hglf/dir2/subdir/large.bin
  undeleting .hglf/dir/subdir/large.bin
  reverting subrepo no-largefiles

  $ hg status -C
  ? dir2/subdir/large.bin

  $ "$PYTHON" $TESTDIR/list-tree.py .hglf dir* large*
  .hglf/
  .hglf/dir/
  .hglf/dir/subdir/
  .hglf/dir/subdir/large.bin
  .hglf/large
  dir/
  dir/subdir/
  dir/subdir/large.bin
  dir2/
  dir2/subdir/
  dir2/subdir/large.bin
  large

  $ cd ../..

Test "pull --rebase" when rebase is enabled before largefiles (issue3861)
=========================================================================

  $ hg showconfig extensions | grep largefiles
  extensions.largefiles=!

  $ mkdir issue3861
  $ cd issue3861
  $ hg init src
  $ hg clone -q src dst
  $ echo a > src/a
  $ hg -R src commit -Aqm "#0"
  Invoking status precommit hook
  A a

  $ cat >> dst/.hg/hgrc <<EOF
  > [extensions]
  > largefiles=
  > EOF
  $ hg -R dst pull --rebase
  pulling from $TESTTMP/issue3861/src
  requesting all changes
  adding changesets
  adding manifests
  adding file changes
  added 1 changesets with 1 changes to 1 files
  new changesets bf5e395ced2c (1 drafts)
  nothing to rebase - updating instead
  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

  $ cd ..