view tests/test-obsolete-divergent.t @ 25757:4d1382fd96ff

context: write dirstate out explicitly at the end of markcommitted

To detect change of a file without redundant comparison of file
content, dirstate recognizes a file as certainly clean, if:

  (1) it is already known as "normal",
  (2) dirstate entry for it has valid (= not "-1") timestamp, and
  (3) mode, size and timestamp of it on the filesystem are as same as
      ones expected in dirstate

This works as expected in many cases, but doesn't in the corner case
that changing a file keeps mode, size and timestamp of it on the
filesystem.

The timetable below shows steps in one of typical such situations:

  ---- ----------------------------------- ----------------
                                           timestamp of "f"
                                           ----------------
                                           dirstate   file-
  time          action                     mem  file  system
  ---- ----------------------------------- ---- ----- -----
   *                                             ***    ***
       - 'hg transplant REV1 REV2 ...'
         - transplanting REV1
           ....
   N
           - change "f", but keep size                   N
             (via 'patch.patch()')
           - 'dirstate.normal("f")'          N   ***
             (via 'repo.commit()')

         - transplanting REV2
           - change "f", but keep size                   N
             (via 'patch.patch()')
           - aborted while patching
  N+1
         - release wlock
           - 'dirstate.write()'              N    N      N

       - 'hg status' shows "r1" as "clean"   N    N      N
  ---- ----------------------------------- ---- ----- -----

The most important point is that 'dirstate.write()' is executed at N+1
or later. This causes writing dirstate timestamp N of "f" out
successfully. If it is executed at N, 'parsers.pack_dirstate()'
replaces timestamp N with "-1" before actual writing dirstate out.

This issue can occur when 'hg transplant' satisfies conditions below:

  - multiple revisions to be transplanted change the same file
  - those revisions don't change mode and size of the file, and
  - the 2nd or later revision of them fails after changing the file

The root cause of this issue is that files are changed without
flushing in-memory dirstate changes via 'repo.commit()' (even though
omitting 'dirstate.normallookup()' on files changed by 'patch.patch()'
for efficiency also causes this issue).

To detect changes of files correctly, this patch writes in-memory
dirstate changes out explicitly after marking files as clean in
'committablectx.markcommitted()', which is invoked via
'repo.commit()'.

After this change, timetable is changed as below:

  ---- ----------------------------------- ----------------
                                           timestamp of "f"
                                           ----------------
                                           dirstate   file-
  time          action                     mem  file  system
  ---- ----------------------------------- ---- ----- -----
   *                                             ***    ***
       - 'hg transplant REV1 REV2 ...'
         - transplanting REV1
           ....
   N
           - change "f", but keep size                   N
             (via 'patch.patch()')
           - 'dirstate.normal("f")'          N    ***
             (via 'repo.commit()')
       ----------------------------------- ---- ----- -----
           - 'dirsttate.write()'            -1    -1
       ----------------------------------- ---- ----- -----
         - transplanting REV2
           - change "f", but keep size                   N
             (via 'patch.patch()')
           - aborted while patching
  N+1
         - release wlock
           - 'dirstate.write()'              -1   -1     N

       - 'hg status' shows "r1" as "clean"   -1   -1     N
  ---- ----------------------------------- ---- ----- -----

To reproduce this issue in tests certainly, this patch emulates some
timing critical actions as below:

  - change "f" at N

    'patch.patch()' with 'fakepatchtime.py' explicitly changes mtime
    of patched files to "2000-01-01 00:00" (= N).

  - 'dirstate.write()' via 'repo.commit()' at N

    'fakedirstatewritetime.py' forces 'pack_dirstate()' to use
    "2000-01-01 00:00" as "now", only if 'pack_dirstate()' is invoked
    via 'committablectx.markcommitted()'.

  - 'dirstate.write()' via releasing wlock at N+1 (or "not at N")

    'pack_dirstate()' via releasing wlock uses actual timestamp at
    runtime as "now", and it should be different from the "2000-01-01
    00:00" of "f".

BTW, this patch doesn't test cases below, even though 'patch.patch()'
is used similarly in these cases:

  1. failure of 'hg import' or 'hg qpush'

  2. success of 'hg import', 'hg qpush' or 'hg transplant'

Case (1) above doesn't cause this kind of issue, because:

  - if patching is aborted by conflicts, changed files are committed

    changed files are marked as CLEAN, even though they are partially
    patched.

  - otherwise, dirstate are fully restored by 'dirstateguard'

    For example in timetable above, timestamp of "f" in .hg/dirstate
    is restored to -1 (or less than N), and subsequent 'hg status' can
    detect changes correctly.

Case (2) always causes 'repo.status()' invocation via 'repo.commit()'
just after changing files inside same wlock scope.

  ---- ----------------------------------- ----------------
                                           timestamp of "f"
                                           ----------------
                                           dirstate   file-
  time          action                     mem  file  system
  ---- ----------------------------------- ---- ----- -----
  N                                              ***   ***
       - make file "f" clean                            N

       - execute 'hg foobar'
         ....
         - 'dirstate.normal("f")'           N    ***
           (e.g. via dirty check
            or previous 'repo.commit()')

         - change "f", but keep size                    N
         - 'repo.status()' (*1)
           (via 'repo.commit()')
  ---- ----------------------------------- ---- ----- -----

At a glance, 'repo.status()' at (*1) seems to cause similar issue (=
"changed files are treated as clean"), but actually doesn't.

'dirstate._lastnormaltime' should be N at (*1) above, because
'dirstate.normal()' via dirty check is finished at N.

Therefore, "f" changed at N (= 'dirstate._lastnormaltime') is forcibly
treated as "unsure" at (*1), and changes are detected as expected (see
'dirstate.status()' for detail).

If 'hg import' is executed with '--no-commit', 'repo.status()' isn't
invoked just after changing files inside same wlock scope.

But preceding 'dirstate.normal()' is invoked inside another wlock
scope via 'cmdutil.bailifchanged()', and in-memory changes should be
flushed at the end of that scope.

Therefore, timestamp N of clean "f" should be replaced by -1, if
'dirstate.write()' is invoked at N. It means that condition of this
issue isn't satisfied.

Test file dedicated to testing the divergent troubles from obsolete changeset.

This is the most complex troubles from far so we isolate it in a dedicated
file.

Enable obsolete

  $ cat >> $HGRCPATH << EOF
  > [ui]
  > logtemplate = {rev}:{node|short} {desc}\n
  > [experimental]
  > evolution=createmarkers
  > [alias]
  > debugobsolete = debugobsolete -d '0 0'
  > [phases]
  > publish=False
  > EOF


  $ mkcommit() {
  >    echo "$1" > "$1"
  >    hg add "$1"
  >    hg ci -m "$1"
  > }
  $ getid() {
  >    hg log --hidden -r "desc('$1')" -T '{node}\n'
  > }

setup repo

  $ hg init reference
  $ cd reference
  $ mkcommit base
  $ mkcommit A_0
  $ hg up 0
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ mkcommit A_1
  created new head
  $ hg up 0
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ mkcommit A_2
  created new head
  $ hg up 0
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ cd ..


  $ newcase() {
  >    hg clone -u 0 -q reference $1
  >    cd $1
  > }

direct divergence
-----------------

A_1 have two direct and divergent successors A_1 and A_1

  $ newcase direct
  $ hg debugobsolete `getid A_0` `getid A_1`
  $ hg debugobsolete `getid A_0` `getid A_2`
  $ hg log -G --hidden
  o  3:392fd25390da A_2
  |
  | o  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  @  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      82623d38b9ba
      392fd25390da
  82623d38b9ba
      82623d38b9ba
  392fd25390da
      392fd25390da
  $ hg log -r 'divergent()'
  2:82623d38b9ba A_1
  3:392fd25390da A_2

check that mercurial refuse to push

  $ hg init ../other
  $ hg push ../other
  pushing to ../other
  searching for changes
  abort: push includes divergent changeset: 392fd25390da!
  [255]

  $ cd ..


indirect divergence with known changeset
-------------------------------------------

  $ newcase indirect_known
  $ hg debugobsolete `getid A_0` `getid A_1`
  $ hg debugobsolete `getid A_0` `getid A_2`
  $ mkcommit A_3
  created new head
  $ hg debugobsolete `getid A_2` `getid A_3`
  $ hg log -G --hidden
  @  4:01f36c5a8fda A_3
  |
  | x  3:392fd25390da A_2
  |/
  | o  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  o  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      82623d38b9ba
      01f36c5a8fda
  82623d38b9ba
      82623d38b9ba
  392fd25390da
      01f36c5a8fda
  01f36c5a8fda
      01f36c5a8fda
  $ hg log -r 'divergent()'
  2:82623d38b9ba A_1
  4:01f36c5a8fda A_3
  $ cd ..


indirect divergence with known changeset
-------------------------------------------

  $ newcase indirect_unknown
  $ hg debugobsolete `getid A_0` aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  $ hg debugobsolete aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa `getid A_1`
  $ hg debugobsolete `getid A_0` `getid A_2`
  $ hg log -G --hidden
  o  3:392fd25390da A_2
  |
  | o  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  @  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      82623d38b9ba
      392fd25390da
  82623d38b9ba
      82623d38b9ba
  392fd25390da
      392fd25390da
  $ hg log -r 'divergent()'
  2:82623d38b9ba A_1
  3:392fd25390da A_2
  $ cd ..

do not take unknown node in account if they are final
-----------------------------------------------------

  $ newcase final-unknown
  $ hg debugobsolete `getid A_0` `getid A_1`
  $ hg debugobsolete `getid A_1` `getid A_2`
  $ hg debugobsolete `getid A_0` bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
  $ hg debugobsolete bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb cccccccccccccccccccccccccccccccccccccccc
  $ hg debugobsolete `getid A_1` dddddddddddddddddddddddddddddddddddddddd

  $ hg debugsuccessorssets --hidden 'desc('A_0')'
  007dc284c1f8
      392fd25390da

  $ cd ..

divergence that converge again is not divergence anymore
-----------------------------------------------------

  $ newcase converged_divergence
  $ hg debugobsolete `getid A_0` `getid A_1`
  $ hg debugobsolete `getid A_0` `getid A_2`
  $ mkcommit A_3
  created new head
  $ hg debugobsolete `getid A_1` `getid A_3`
  $ hg debugobsolete `getid A_2` `getid A_3`
  $ hg log -G --hidden
  @  4:01f36c5a8fda A_3
  |
  | x  3:392fd25390da A_2
  |/
  | x  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  o  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      01f36c5a8fda
  82623d38b9ba
      01f36c5a8fda
  392fd25390da
      01f36c5a8fda
  01f36c5a8fda
      01f36c5a8fda
  $ hg log -r 'divergent()'
  $ cd ..

split is not divergences
-----------------------------

  $ newcase split
  $ hg debugobsolete `getid A_0` `getid A_1` `getid A_2`
  $ hg log -G --hidden
  o  3:392fd25390da A_2
  |
  | o  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  @  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      82623d38b9ba 392fd25390da
  82623d38b9ba
      82623d38b9ba
  392fd25390da
      392fd25390da
  $ hg log -r 'divergent()'

Even when subsequent rewriting happen

  $ mkcommit A_3
  created new head
  $ hg debugobsolete `getid A_1` `getid A_3`
  $ hg up 0
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ mkcommit A_4
  created new head
  $ hg debugobsolete `getid A_2` `getid A_4`
  $ hg up 0
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ mkcommit A_5
  created new head
  $ hg debugobsolete `getid A_4` `getid A_5`
  $ hg log -G --hidden
  @  6:e442cfc57690 A_5
  |
  | x  5:6a411f0d7a0a A_4
  |/
  | o  4:01f36c5a8fda A_3
  |/
  | x  3:392fd25390da A_2
  |/
  | x  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  o  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      01f36c5a8fda e442cfc57690
  82623d38b9ba
      01f36c5a8fda
  392fd25390da
      e442cfc57690
  01f36c5a8fda
      01f36c5a8fda
  6a411f0d7a0a
      e442cfc57690
  e442cfc57690
      e442cfc57690
  $ hg log -r 'divergent()'

Check more complex obsolescence graft (with divergence)

  $ mkcommit B_0; hg up 0
  0 files updated, 0 files merged, 2 files removed, 0 files unresolved
  $ hg debugobsolete `getid B_0` `getid A_2`
  $ mkcommit A_7; hg up 0
  created new head
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ mkcommit A_8; hg up 0
  created new head
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ hg debugobsolete `getid A_5` `getid A_7` `getid A_8`
  $ mkcommit A_9; hg up 0
  created new head
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ hg debugobsolete `getid A_5` `getid A_9`
  $ hg log -G --hidden
  o  10:bed64f5d2f5a A_9
  |
  | o  9:14608b260df8 A_8
  |/
  | o  8:7ae126973a96 A_7
  |/
  | x  7:3750ebee865d B_0
  | |
  | x  6:e442cfc57690 A_5
  |/
  | x  5:6a411f0d7a0a A_4
  |/
  | o  4:01f36c5a8fda A_3
  |/
  | x  3:392fd25390da A_2
  |/
  | x  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  @  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      01f36c5a8fda bed64f5d2f5a
      01f36c5a8fda 7ae126973a96 14608b260df8
  82623d38b9ba
      01f36c5a8fda
  392fd25390da
      bed64f5d2f5a
      7ae126973a96 14608b260df8
  01f36c5a8fda
      01f36c5a8fda
  6a411f0d7a0a
      bed64f5d2f5a
      7ae126973a96 14608b260df8
  e442cfc57690
      bed64f5d2f5a
      7ae126973a96 14608b260df8
  3750ebee865d
      bed64f5d2f5a
      7ae126973a96 14608b260df8
  7ae126973a96
      7ae126973a96
  14608b260df8
      14608b260df8
  bed64f5d2f5a
      bed64f5d2f5a
  $ hg log -r 'divergent()'
  4:01f36c5a8fda A_3
  8:7ae126973a96 A_7
  9:14608b260df8 A_8
  10:bed64f5d2f5a A_9

fix the divergence

  $ mkcommit A_A; hg up 0
  created new head
  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
  $ hg debugobsolete `getid A_9` `getid A_A`
  $ hg debugobsolete `getid A_7` `getid A_A`
  $ hg debugobsolete `getid A_8` `getid A_A`
  $ hg log -G --hidden
  o  11:a139f71be9da A_A
  |
  | x  10:bed64f5d2f5a A_9
  |/
  | x  9:14608b260df8 A_8
  |/
  | x  8:7ae126973a96 A_7
  |/
  | x  7:3750ebee865d B_0
  | |
  | x  6:e442cfc57690 A_5
  |/
  | x  5:6a411f0d7a0a A_4
  |/
  | o  4:01f36c5a8fda A_3
  |/
  | x  3:392fd25390da A_2
  |/
  | x  2:82623d38b9ba A_1
  |/
  | x  1:007dc284c1f8 A_0
  |/
  @  0:d20a80d4def3 base
  
  $ hg debugsuccessorssets --hidden 'all()'
  d20a80d4def3
      d20a80d4def3
  007dc284c1f8
      01f36c5a8fda a139f71be9da
  82623d38b9ba
      01f36c5a8fda
  392fd25390da
      a139f71be9da
  01f36c5a8fda
      01f36c5a8fda
  6a411f0d7a0a
      a139f71be9da
  e442cfc57690
      a139f71be9da
  3750ebee865d
      a139f71be9da
  7ae126973a96
      a139f71be9da
  14608b260df8
      a139f71be9da
  bed64f5d2f5a
      a139f71be9da
  a139f71be9da
      a139f71be9da
  $ hg log -r 'divergent()'

  $ cd ..


Subset does not diverge
------------------------------

Do not report divergent successors-set if it is a subset of another
successors-set. (report [A,B] not [A] + [A,B])

  $ newcase subset
  $ hg debugobsolete `getid A_0` `getid A_2`
  $ hg debugobsolete `getid A_0` `getid A_1` `getid A_2`
  $ hg debugsuccessorssets --hidden 'desc('A_0')'
  007dc284c1f8
      82623d38b9ba 392fd25390da

  $ cd ..