typing: add basic type hints to localrepo.py There's a lot more that could be done, but this sticks to the obviously correct stuff that is either related to existing imports or primitives. Hopefully this helps smoke out more path related bytes vs str issues in TortoiseHg. I'm avoiding the interfaces for now, because they seem to confuse pytype and/or PyCharm. It might be worth typing the return of `makelocalrepository` to `localrepository`, but that leaks an implementation detail, so that can be revisited later.

check-code: drop the check for whitespace around named parameters This check flags py3 annotations of named parameters, because `black` adds spaces around the assignment in this case. Since the chosen formatter has opinions (and pylint also wants the space in the case of annotations), drop the check so we can use py3 annotations.

rust-status: query fs traversal metadata lazily Currently, any time the status algorithm needs to read a directory from the filesystem (because the stat-only optimization is not available), it also stats each directory entry eagerly. Stat'ing the entries is only needed in a few cases (like when checking the mtime of a directory for caching): this patch creates a wrapper struct `DirEntry` that only stats the directory entry it represents when needed. Excerpt of an `strace` before this change on Mozilla Central: ``` openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=3540, ...}, AT_EMPTY_PATH) = 0 getdents64(3, 0x55dc970bd440 /* 139 entries */, 32768) = 5072 statx(3, ".hg", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=772, ...}) = 0 [... 135 other successful `statx` calls] getdents64(3, 0x55dc970bd440 /* 0 entries */, 32768) = 0 close(3) = 0 ``` After this change: ``` openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=3540, ...}, AT_EMPTY_PATH) = 0 getdents64(3, 0x561567c10190 /* 139 entries */, 32768) = 5072 getdents64(3, 0x561567c10190 /* 0 entries */, 32768) = 0 close(3) = 0 ```

rust-status: make `DirEntry` attributes clearer

tests: remove non-python3 line matching and tests block We don't support Python2 anymore

pywatchman: remove obsolete comments about importing from future See 6000f5b25c9b.

demandimport: remove an obsolete comment about importing from future See 6000f5b25c9b.

mr-template: wrap the instructions inside a comment block At least in preview mode, this hides the text so the user doesn't have to delete it. It's still visible in edit mode, so the user sees it.

revlog: use the user facing filename as the display_id for filelogs I had trouble isolating some LFS blob corruption detected by `hg verify` because the traceback referenced a file, but with the `data/` prefix in the `.hg/store` path, so it couldn't be located with the `file()` revset: ``` Traceback (most recent call last): File "/mnt/d/mercurial/mercurial/revlog.py", line 3209, in verifyintegrity _verify_revision(self, skipflags, state, node) File "/mnt/d/mercurial/hgext/lfs/wrapper.py", line 246, in _verify_revision orig(rl, skipflags, state, node) File "/mnt/d/mercurial/mercurial/revlog.py", line 158, in _verify_revision rl.revision(node) File "/mnt/d/mercurial/mercurial/revlog.py", line 1816, in revision return self._revisiondata(nodeorrev, _df) File "/mnt/d/mercurial/mercurial/revlog.py", line 1870, in _revisiondata self.checkhash(text, node, rev=rev) File "/mnt/d/mercurial/mercurial/revlog.py", line 1996, in checkhash % (self.display_id, pycompat.bytestr(revornode)) mercurial.error.RevlogError: integrity check failed on data/EXE/PPC/shrinksrec.exe:0 ``` (I'm a little surprised it resulted in a stacktrace instead of just a message, but that's a different issue. I'm also not sure how to trigger the simplestore case, since IIUC, it's also a revlog based store.) It's not clear how to handle the changelog and manifest (because the user doesn't interact with them as a file), so those cases are left alone. The other thing that would be nice to improve somehow is to indicate that the ":0" is a revlog revision, not the changeset revision that users are used to. I'm not sure how to handle the "or node" part though.

revlog: drop an unused variable assignment It's assigned again 2 lines later.

lfs: improve an exception message for blob corruption detected on transfer The message about the server crash originated in 0ee0a3f6a990 (after support for serving blobs was added), but was copied from the Facebook repo that forked prior to server side support. Therefore, this message only displayed in their client, so it was safe to assume the server crashed. But that was never the case for vanilla Mercurial, as I saw this in a server log. Also, display the blob reference so that it's easier to figure out where the problem was when a bunch of blobs are transferred at once.

branching: merge stable into default

Added signature for changeset 8830004967ad

Added tag 6.3.3 for changeset 8830004967ad

relnotes: add 6.3.3

typing: add typehints to mercurial/diffutil.py Lack of typehints here caused the fact that TortoiseHg was passing str instead of bytes as the key in `opts` to be missed, resulting in shelf corruption in cases where `diff.git` is required.

patchbomb: respect the `--git` option I *think* this is the only diffopt exposed on the command line. TortoiseHg had a similar issue creating diffopts, and this was caught by type hints in the next commit.

rhg: remember the inode of .hg/dirstate This allows us to detect changes of `.hg/dirstate`, which is either the full dirstate (in dirstate-v1) or the docket file (v2) without relying on data inside the file. It only works on UNIX systems. This fixes a race condition for dirstate-v1 (as demonstrated by the test changes) and adds a confortable layer of sanity for dirstate-v2.

rust-dirstate-v2: don't write dirstate if data file has changed This fixes the following race: - process A reads the dirstate - process B reads and writes the dirstate - process A writes the dirstate This either resulted in losing what process B had just written or a crash because the `uuid` had changed and we were trying to write to a file that doesn't exist. More explanations inside. This doesn't fix the issue for dirstate-v1, a later patch addresses it.

rust-dirstate: remember the data file uuid dirstate was loaded with This will be used in the next patch to fix a race condition.

dirstate: set identity whenever we read the dirstate's v2 docket The docket can be loaded outside of a full read (for exemple when pre-fetching parents), so the current code would read/set the identity after loading the data, opening a race condition: A0: first process docket is read B0: other process appends new data to the dirstate (and changes the docket) A1: first process sets the identity (based on pre-B content, but with post-B identity) A1: first process loads the dirstatemap from the data file A1: first process does not detect the race and overwrites the update from B.

dirstate: factor the identity setting code in the dirstate map We need it in more locations, so let us start factoring thing out first to make sure the same code is called everywhere. This bears some similarity with 85746485a4dd on default, but at a smaller scope and for a different purpose.

dirstate: simplify the dirstate's read race testing Now that most code behaves properly, we can simplify the expected matching.

dirstate: deal with read-race for pure rust code path (rhg) If we cannot read the dirstate data, this is probably because a writing process wrote it under our feet. So refresh the docket and try again a handful of time.

dirstate: deal with read-race for python code using rust object If we cannot read the dirstate data, this is probably because a writing process wrote it under our feet. So refresh the docket and try again a handful of time.

dirstate: deal with read-race for pure python code If we cannot read the dirstate data, this is probably because a writing process wrote it under our feet. So refresh the docket and try again a handful of time.

dirstate: abstract the reading of the data file in v2 in a method We will need more changes to avoid some race conditions during read, so we first isolate the simple logic before making it more complicated.

dirstate: add append/new-file variants in the dirstate's read race tests This covers more ground and finds more bugs. At that point I gave up on making things as `known-bad-output` / `missing-correct-output` as this gets too messy.

dirstate: add a synchronisation point in the middle of the read This will be useful to test some more race conditions around dirstate.

dirstate: add v1-v2 variants to the dirstate's read race tests More cases mean different issues.

dirstate: check dirstate race condition around status More problems to solve.

dirstate: check dirstate race condition around update More problems to solve.

dirstate: check dirstate race condition around commit Once in a while, rhg is the only one to behave right here.

dirstate: initial creation of a test file to check dirstate race read More problems to solve… yeah ! (I guess)

dirstate: add a synchronisation point before doing a full dirstate read This will be useful to test some race conditions around the dirstate.

rust-repo: move dirstate-v2 opening to a separate method The next changeset will make changes to this logic, it helps to have it in order first.

rhg: fix race when an ambiguous file is deleted on disk There are two places in the status code where we handle files whose status we are unsure of based off of metadata alone: this one is the first one to actually disambiguate, and the second one is later in the code (but updated in the previous commit) for files that are actually clean to update the dirstate. Since there is a chance that the contents have changed between those two moments, we need to stat the files again, since re-using the old stat could lie about the clean state of the file.

rhg: fix race when a fixup file is deleted on disk See next changeset for the other race in the same kind of logic and why there are two different places.

dirstate: test a `hg status` raced by a `hg remove` This shows that `rhg` is misbehaving here.

dirstate: tests racing status with both dirstate-v2 append and rewrite The way the racing process touches the dirstate results in different challenges for the raced process. We now test each variant in the `test-dirstate-status-race.t` tests.

dirstate-v2: add devel config option to control write behavior This will help us to write predictable tests checking behavior in each case.

dirstate: use more than a bool to control append behavior When writing dirstate-v2, we might either append to the existing file, or create a new file. We are about to introduce some configuration to control this behavior. As a prelude, we change the current way the behavior was automatically controlled to make the change smaller/clearer.

dirstate: cover each dirstate version when testing for status race Previously we were only testing it with the default (dirstate-v1 currently). Now we explicitly test each variant.

dirstate: test a `hg status` raced by another `hg status` This shows that `rhg` is misbehaving here.

dirstate: test a `hg status` raced by a `hg update` This shows that `rhg` is misbehaving here.

dirstate: test a `hg status` raced by a `hg commit` This shows that `rhg` is misbehaving here.

dirstate: test a `hg status` raced by a `hg add` This shows that `rhg` is misbehaving here.

dirstate: add a way to test races happening during status We add the `devel.sync.status.pre-dirstate-write-file` config option to easily test what happens when other operations happen during the window where `hg status` is done working but has not updated the cache on disk yet. We introduce the framework for testing such races too, actual tests will be added in the next changesets. For now the test is only checking dirstate-v1. We will extend the test coverage later too. Check test documentation for details. Code change from Raphaël Gomès <rgomes@octobus.net> Test change from Pierre-Yves David <pierre-yves.david@octobus.net>

testing: introduce util function to synchronize concurrent commands on files This is an extension of mechanisms that the tests have been using for a while. To be able to also control the execution in Rust, we introduce utility to perform such `wait_on_file` logic based on some configuration value. This will be used in the tests introduced in the next changesets.

dirstate: add some debug output when writing the dirstate This helps debugging Mercurial behavior in the dirstate-v2 case.

run-tests: make it possible to nest conditionals This is not that hard to implement and makes our life easier on a regular basis.

dirstate-v2: don't mmap the data file when on NFS `mmap` on NFS will trigger a SIGBUS when the mmap'ed file is deleted, which wouldn't work in our case. Also, the performance advantage of using mmap on NFS is debatable at best.

rust-dirstate: trace append/no append to help debugging

rust: add debug log about skipping dirstate update

test-dirstate: use more robust method to trigger a data-file append The previous method was fragile and somewhat flaky on fast machines.

transaction: tests we don't overwrite bookmark activation on abort We actually do not! Great.