view tests/test-share-safe.t @ 50400:95acba2c29f6

encoding: avoid quadratic time complexity when json-encoding non-UTF8 strings

Apparently the code uses "+=" with a bytes object, which is linear-time, so the
whole encoding is quadratic-time. This patch makes us use a bytearray object,
instead, which has a(n amortized-)constant-time append operation.

The encoding is still not particularly fast, but at least a 10MB file
takes tens of seconds, not many hours to encode.

setup

  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > share =
  > [format]
  > use-share-safe = True
  > [storage]
  > revlog.persistent-nodemap.slow-path=allow
  > # enforce zlib to ensure we can upgrade to zstd later
  > [format]
  > revlog-compression=zlib
  > # we want to be able to enable it later
  > use-persistent-nodemap=no
  > EOF

prepare source repo

  $ hg init source
  $ cd source
  $ cat .hg/requires
  dirstate-v2 (dirstate-v2 !)
  share-safe
  $ cat .hg/store/requires
  dotencode
  fncache
  generaldelta
  revlogv1
  sparserevlog
  store
  $ hg debugrequirements
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  share-safe
  sparserevlog
  store

  $ echo a > a
  $ hg ci -Aqm "added a"
  $ echo b > b
  $ hg ci -Aqm "added b"

  $ HGEDITOR=cat hg config --shared
  abort: repository is not shared; can't use --shared
  [10]
  $ cd ..

Create a shared repo and check the requirements are shared and read correctly
  $ hg share source shared1
  updating working directory
  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd shared1
  $ cat .hg/requires
  dirstate-v2 (dirstate-v2 !)
  share-safe
  shared

  $ hg debugrequirements -R ../source
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  share-safe
  sparserevlog
  store

  $ hg debugrequirements
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  share-safe
  shared
  sparserevlog
  store

  $ echo c > c
  $ hg ci -Aqm "added c"

Check that config of the source repository is also loaded

  $ hg showconfig ui.curses
  [1]

  $ echo "[ui]" >> ../source/.hg/hgrc
  $ echo "curses=true" >> ../source/.hg/hgrc

  $ hg showconfig ui.curses
  true

Test that extensions of source repository are also loaded

  $ hg debugextensions
  share
  $ hg extdiff -p echo
  hg: unknown command 'extdiff'
  'extdiff' is provided by the following extension:
  
      extdiff       command to allow external programs to compare revisions
  
  (use 'hg help extensions' for information on enabling extensions)
  [10]

  $ echo "[extensions]" >> ../source/.hg/hgrc
  $ echo "extdiff=" >> ../source/.hg/hgrc

  $ hg debugextensions -R ../source
  extdiff
  share
  $ hg extdiff -R ../source -p echo

BROKEN: the command below will not work if config of shared source is not loaded
on dispatch but debugextensions says that extension
is loaded
  $ hg debugextensions
  extdiff
  share

  $ hg extdiff -p echo

However, local .hg/hgrc should override the config set by share source

  $ echo "[ui]" >> .hg/hgrc
  $ echo "curses=false" >> .hg/hgrc

  $ hg showconfig ui.curses
  false

  $ HGEDITOR=cat hg config --shared
  [ui]
  curses=true
  [extensions]
  extdiff=

  $ HGEDITOR=cat hg config --local
  [ui]
  curses=false

Testing that hooks set in source repository also runs in shared repo

  $ cd ../source
  $ cat <<EOF >> .hg/hgrc
  > [extensions]
  > hooklib=
  > [hooks]
  > pretxnchangegroup.reject_merge_commits = \
  >   python:hgext.hooklib.reject_merge_commits.hook
  > EOF

  $ cd ..
  $ hg clone source cloned
  updating to branch default
  3 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd cloned
  $ hg up 0
  0 files updated, 0 files merged, 2 files removed, 0 files unresolved
  $ echo bar > bar
  $ hg ci -Aqm "added bar"
  $ hg merge
  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
  (branch merge, don't forget to commit)
  $ hg ci -m "merge commit"

  $ hg push ../source
  pushing to ../source
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  error: pretxnchangegroup.reject_merge_commits hook failed: bcde3522682d rejected as merge on the same branch. Please consider rebase.
  transaction abort!
  rollback completed
  abort: bcde3522682d rejected as merge on the same branch. Please consider rebase.
  [255]

  $ hg push ../shared1
  pushing to ../shared1
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  error: pretxnchangegroup.reject_merge_commits hook failed: bcde3522682d rejected as merge on the same branch. Please consider rebase.
  transaction abort!
  rollback completed
  abort: bcde3522682d rejected as merge on the same branch. Please consider rebase.
  [255]

Test that if share source config is untrusted, we dont read it

  $ cd ../shared1

  $ cat << EOF > $TESTTMP/untrusted.py
  > from mercurial import scmutil, util
  > def uisetup(ui):
  >     class untrustedui(ui.__class__):
  >         def _trusted(self, fp, f):
  >             if util.normpath(fp.name).endswith(b'source/.hg/hgrc'):
  >                 return False
  >             return super(untrustedui, self)._trusted(fp, f)
  >     ui.__class__ = untrustedui
  > EOF

  $ hg showconfig hooks
  hooks.pretxnchangegroup.reject_merge_commits=python:hgext.hooklib.reject_merge_commits.hook

  $ hg showconfig hooks --config extensions.untrusted=$TESTTMP/untrusted.py
  [1]

Update the source repository format and check that shared repo works

  $ cd ../source

Disable zstd related tests because its not present on pure version
#if zstd
  $ echo "[format]" >> .hg/hgrc
  $ echo "revlog-compression=zstd" >> .hg/hgrc

  $ hg debugupgraderepo --run -q
  upgrade will perform the following actions:
  
  requirements
     preserved: dotencode, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (no-dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (dirstate-v2 !)
     added: revlog-compression-zstd
  
  processed revlogs:
    - all-filelogs
    - changelog
    - manifest
  
  $ hg log -r .
  changeset:   1:5f6d8a4bf34a
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     added b
  
#endif
  $ echo "[format]" >> .hg/hgrc
  $ echo "use-persistent-nodemap=True" >> .hg/hgrc

  $ hg debugupgraderepo --run -q -R ../shared1
  abort: cannot use these actions on a share repository: persistent-nodemap
  (upgrade the main repository directly)
  [255]

  $ hg debugupgraderepo --run -q
  upgrade will perform the following actions:
  
  requirements
     preserved: dotencode, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (no-zstd no-dirstate-v2 !)
     preserved: dotencode, fncache, generaldelta, revlog-compression-zstd, revlogv1, share-safe, sparserevlog, store (zstd no-dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, share-safe, sparserevlog, store (no-zstd dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlog-compression-zstd, revlogv1, share-safe, sparserevlog, store (zstd dirstate-v2 !)
     added: persistent-nodemap
  
  processed revlogs:
    - all-filelogs
    - changelog
    - manifest
  
  $ hg log -r .
  changeset:   1:5f6d8a4bf34a
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     added b
  

Shared one should work
  $ cd ../shared1
  $ hg log -r .
  changeset:   2:155349b645be
  tag:         tip
  user:        test
  date:        Thu Jan 01 00:00:00 1970 +0000
  summary:     added c
  

Testing that nonsharedrc is loaded for source and not shared

  $ cd ../source
  $ touch .hg/hgrc-not-shared
  $ echo "[ui]" >> .hg/hgrc-not-shared
  $ echo "traceback=true" >> .hg/hgrc-not-shared

  $ hg showconfig ui.traceback
  true

  $ HGEDITOR=cat hg config --non-shared
  [ui]
  traceback=true

  $ cd ../shared1
  $ hg showconfig ui.traceback
  [1]

Unsharing works

  $ hg unshare

Test that source config is added to the shared one after unshare, and the config
of current repo is still respected over the config which came from source config
  $ cd ../cloned
  $ hg push ../shared1
  pushing to ../shared1
  searching for changes
  adding changesets
  adding manifests
  adding file changes
  error: pretxnchangegroup.reject_merge_commits hook failed: bcde3522682d rejected as merge on the same branch. Please consider rebase.
  transaction abort!
  rollback completed
  abort: bcde3522682d rejected as merge on the same branch. Please consider rebase.
  [255]
  $ hg showconfig ui.curses -R ../shared1
  false

  $ cd ../

Test that upgrading using debugupgraderepo works
=================================================

  $ hg init non-share-safe --config format.use-share-safe=false
  $ cd non-share-safe
  $ hg debugrequirements
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  sparserevlog
  store
  $ echo foo > foo
  $ hg ci -Aqm 'added foo'
  $ echo bar > bar
  $ hg ci -Aqm 'added bar'

Create a share before upgrading

  $ cd ..
  $ hg share non-share-safe nss-share
  updating working directory
  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ hg debugrequirements -R nss-share
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  shared
  sparserevlog
  store
  $ cd non-share-safe

Upgrade

  $ hg debugupgraderepo -q
  requirements
     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !)
     added: share-safe
  
  no revlogs to process
  
  $ hg debugupgraderepo --run
  upgrade will perform the following actions:
  
  requirements
     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !)
     added: share-safe
  
  share-safe
     Upgrades a repository to share-safe format so that future shares of this repository share its requirements and configs.
  
  no revlogs to process
  
  beginning upgrade...
  repository locked and read-only
  creating temporary repository to stage upgraded data: $TESTTMP/non-share-safe/.hg/upgrade.* (glob)
  (it is safe to interrupt this process any time before data migration completes)
  upgrading repository requirements
  removing temporary repository $TESTTMP/non-share-safe/.hg/upgrade.* (glob)
  repository upgraded to share safe mode, existing shares will still work in old non-safe mode. Re-share existing shares to use them in safe mode New shares will be created in safe mode.

  $ hg debugrequirements
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  share-safe
  sparserevlog
  store

  $ cat .hg/requires
  dirstate-v2 (dirstate-v2 !)
  share-safe

  $ cat .hg/store/requires
  dotencode
  fncache
  generaldelta
  revlogv1
  sparserevlog
  store

  $ hg log -GT "{node}: {desc}\n"
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

Make sure existing shares dont work with default config

  $ hg log -GT "{node}: {desc}\n" -R ../nss-share
  abort: version mismatch: source uses share-safe functionality while the current share does not
  (see `hg help config.format.use-share-safe` for more information)
  [255]


Create a safe share from upgrade one

  $ cd ..
  $ hg share non-share-safe ss-share
  updating working directory
  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ cd ss-share
  $ hg log -GT "{node}: {desc}\n"
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  
  $ cd ../non-share-safe

Test that downgrading works too

  $ cat >> $HGRCPATH <<EOF
  > [extensions]
  > share =
  > [format]
  > use-share-safe = False
  > EOF

  $ hg debugupgraderepo -q
  requirements
     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !)
     removed: share-safe
  
  no revlogs to process
  
  $ hg debugupgraderepo --run
  upgrade will perform the following actions:
  
  requirements
     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !)
     removed: share-safe
  
  no revlogs to process
  
  beginning upgrade...
  repository locked and read-only
  creating temporary repository to stage upgraded data: $TESTTMP/non-share-safe/.hg/upgrade.* (glob)
  (it is safe to interrupt this process any time before data migration completes)
  upgrading repository requirements
  removing temporary repository $TESTTMP/non-share-safe/.hg/upgrade.* (glob)
  repository downgraded to not use share safe mode, existing shares will not work and needs to be reshared.

  $ hg debugrequirements
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  sparserevlog
  store

  $ cat .hg/requires
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  sparserevlog
  store

  $ test -f .hg/store/requires
  [1]

  $ hg log -GT "{node}: {desc}\n"
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

Make sure existing shares still works

  $ hg log -GT "{node}: {desc}\n" -R ../nss-share
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

  $ hg log -GT "{node}: {desc}\n" -R ../ss-share
  abort: share source does not support share-safe requirement
  (see `hg help config.format.use-share-safe` for more information)
  [255]

Testing automatic downgrade of shares when config is set

  $ touch ../ss-share/.hg/wlock
  $ hg log -GT "{node}: {desc}\n" -R ../ss-share --config share.safe-mismatch.source-not-safe=downgrade-abort
  abort: failed to downgrade share, got error: Lock held
  (see `hg help config.format.use-share-safe` for more information)
  [255]
  $ rm ../ss-share/.hg/wlock

  $ cp -R ../ss-share ../ss-share-bck
  $ hg log -GT "{node}: {desc}\n" -R ../ss-share --config share.safe-mismatch.source-not-safe=downgrade-abort
  repository downgraded to not use share-safe mode
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  
  $ rm -rf ../ss-share
  $ mv ../ss-share-bck ../ss-share

  $ hg log -GT "{node}: {desc}\n" -R ../ss-share --config share.safe-mismatch.source-not-safe=downgrade-abort --config share.safe-mismatch.source-not-safe:verbose-upgrade=no
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

  $ hg log -GT "{node}: {desc}\n" -R ../ss-share
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  


Testing automatic upgrade of shares when config is set

  $ hg debugupgraderepo -q --run --config format.use-share-safe=True
  upgrade will perform the following actions:
  
  requirements
     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store (no-dirstate-v2 !)
     preserved: dotencode, use-dirstate-v2, fncache, generaldelta, revlogv1, sparserevlog, store (dirstate-v2 !)
     added: share-safe
  
  no revlogs to process
  
  repository upgraded to share safe mode, existing shares will still work in old non-safe mode. Re-share existing shares to use them in safe mode New shares will be created in safe mode.
  $ hg debugrequirements
  dotencode
  dirstate-v2 (dirstate-v2 !)
  fncache
  generaldelta
  revlogv1
  share-safe
  sparserevlog
  store
  $ hg log -GT "{node}: {desc}\n" -R ../nss-share
  abort: version mismatch: source uses share-safe functionality while the current share does not
  (see `hg help config.format.use-share-safe` for more information)
  [255]

Check that if lock is taken, upgrade fails but read operation are successful
  $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgra
  abort: share-safe mismatch with source.
  Unrecognized value 'upgra' of `share.safe-mismatch.source-safe` set.
  (see `hg help config.format.use-share-safe` for more information)
  [255]
  $ touch ../nss-share/.hg/wlock
  $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-allow
  failed to upgrade share, got error: Lock held
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

  $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-allow --config share.safe-mismatch.source-safe.warn=False
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

  $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-abort
  abort: failed to upgrade share, got error: Lock held
  (see `hg help config.format.use-share-safe` for more information)
  [255]

  $ rm ../nss-share/.hg/wlock
  $ cp -R ../nss-share ../nss-share-bck
  $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-abort
  repository upgraded to use share-safe mode
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  
  $ rm -rf ../nss-share
  $ mv ../nss-share-bck ../nss-share
  $ hg log -GT "{node}: {desc}\n" -R ../nss-share --config share.safe-mismatch.source-safe=upgrade-abort --config share.safe-mismatch.source-safe:verbose-upgrade=no
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

Test that unshare works

  $ hg unshare -R ../nss-share
  $ hg log -GT "{node}: {desc}\n" -R ../nss-share
  @  f63db81e6dde1d9c78814167f77fb1fb49283f4f: added bar
  |
  o  f3ba8b99bb6f897c87bbc1c07b75c6ddf43a4f77: added foo
  

Test automatique upgrade/downgrade of main-repository
------------------------------------------------------

create an initial repository

  $ hg init auto-upgrade \
  > --config format.use-share-safe=no
  $ hg debugbuilddag -R auto-upgrade --new-file .+5
  $ hg -R auto-upgrade update
  6 files updated, 0 files merged, 0 files removed, 0 files unresolved
  $ hg debugformat -R auto-upgrade | grep share-safe
  share-safe:          no

upgrade it to share-safe automatically

  $ hg status -R auto-upgrade \
  >     --config format.use-share-safe.automatic-upgrade-of-mismatching-repositories=yes \
  >     --config format.use-share-safe=yes
  automatically upgrading repository to the `share-safe` feature
  (see `hg help config.format.use-share-safe` for details)
  $ hg debugformat -R auto-upgrade | grep share-safe
  share-safe:         yes

downgrade it from share-safe automatically

  $ hg status -R auto-upgrade \
  >     --config format.use-share-safe.automatic-upgrade-of-mismatching-repositories=yes \
  >     --config format.use-share-safe=no
  automatically downgrading repository from the `share-safe` feature
  (see `hg help config.format.use-share-safe` for details)
  $ hg debugformat -R auto-upgrade | grep share-safe
  share-safe:          no