dirstate: write the `branch` as part of the transaction if any Bypassing the transaction means we could get out of sync with the dirstatemap content. The branch is stil written right away if no transaction is around, but at least it no longer bypass the transaction. Actual caller of this still need to be updated.

dirstate: factor the transaction abort logic We will need it in more occasion if the branch is to be written as part of the transaction.

dirstate: use a context manager to handle the file used for writing the branch This is more modern.

style: rewrap `ui.deprecwarn` declaration This get easier to read, especially with the type annotation.

branching: merge stable into default The clippy god had to be appeased on some aspect.

Added signature for changeset 8830004967ad

Added tag 6.3.3 for changeset 8830004967ad

relnotes: add 6.3.3

narrow: read pending file when applicable Now that this is part of the transaction, this is necessary to make sure we read the right data in hooks (if any).

typing: add typehints to mercurial/diffutil.py Lack of typehints here caused the fact that TortoiseHg was passing str instead of bytes as the key in `opts` to be missed, resulting in shelf corruption in cases where `diff.git` is required.

patchbomb: respect the `--git` option I *think* this is the only diffopt exposed on the command line. TortoiseHg had a similar issue creating diffopts, and this was caught by type hints in the next commit.

rhg: remember the inode of .hg/dirstate This allows us to detect changes of `.hg/dirstate`, which is either the full dirstate (in dirstate-v1) or the docket file (v2) without relying on data inside the file. It only works on UNIX systems. This fixes a race condition for dirstate-v1 (as demonstrated by the test changes) and adds a confortable layer of sanity for dirstate-v2.

rust-dirstate-v2: don't write dirstate if data file has changed This fixes the following race: - process A reads the dirstate - process B reads and writes the dirstate - process A writes the dirstate This either resulted in losing what process B had just written or a crash because the `uuid` had changed and we were trying to write to a file that doesn't exist. More explanations inside. This doesn't fix the issue for dirstate-v1, a later patch addresses it.

rust-dirstate: remember the data file uuid dirstate was loaded with This will be used in the next patch to fix a race condition.

dirstate: set identity whenever we read the dirstate's v2 docket The docket can be loaded outside of a full read (for exemple when pre-fetching parents), so the current code would read/set the identity after loading the data, opening a race condition: A0: first process docket is read B0: other process appends new data to the dirstate (and changes the docket) A1: first process sets the identity (based on pre-B content, but with post-B identity) A1: first process loads the dirstatemap from the data file A1: first process does not detect the race and overwrites the update from B.

dirstate: factor the identity setting code in the dirstate map We need it in more locations, so let us start factoring thing out first to make sure the same code is called everywhere. This bears some similarity with 85746485a4dd on default, but at a smaller scope and for a different purpose.

dirstate: simplify the dirstate's read race testing Now that most code behaves properly, we can simplify the expected matching.

dirstate: deal with read-race for pure rust code path (rhg) If we cannot read the dirstate data, this is probably because a writing process wrote it under our feet. So refresh the docket and try again a handful of time.

dirstate: deal with read-race for python code using rust object If we cannot read the dirstate data, this is probably because a writing process wrote it under our feet. So refresh the docket and try again a handful of time.

dirstate: deal with read-race for pure python code If we cannot read the dirstate data, this is probably because a writing process wrote it under our feet. So refresh the docket and try again a handful of time.

dirstate: abstract the reading of the data file in v2 in a method We will need more changes to avoid some race conditions during read, so we first isolate the simple logic before making it more complicated.

dirstate: add append/new-file variants in the dirstate's read race tests This covers more ground and finds more bugs. At that point I gave up on making things as `known-bad-output` / `missing-correct-output` as this gets too messy.

dirstate: add a synchronisation point in the middle of the read This will be useful to test some more race conditions around dirstate.

dirstate: add v1-v2 variants to the dirstate's read race tests More cases mean different issues.

dirstate: check dirstate race condition around status More problems to solve.

dirstate: check dirstate race condition around update More problems to solve.

dirstate: check dirstate race condition around commit Once in a while, rhg is the only one to behave right here.

dirstate: initial creation of a test file to check dirstate race read More problems to solve… yeah ! (I guess)

dirstate: add a synchronisation point before doing a full dirstate read This will be useful to test some race conditions around the dirstate.

rust-repo: move dirstate-v2 opening to a separate method The next changeset will make changes to this logic, it helps to have it in order first.

rhg: fix race when an ambiguous file is deleted on disk There are two places in the status code where we handle files whose status we are unsure of based off of metadata alone: this one is the first one to actually disambiguate, and the second one is later in the code (but updated in the previous commit) for files that are actually clean to update the dirstate. Since there is a chance that the contents have changed between those two moments, we need to stat the files again, since re-using the old stat could lie about the clean state of the file.

rhg: fix race when a fixup file is deleted on disk See next changeset for the other race in the same kind of logic and why there are two different places.

dirstate: test a `hg status` raced by a `hg remove` This shows that `rhg` is misbehaving here.

dirstate: tests racing status with both dirstate-v2 append and rewrite The way the racing process touches the dirstate results in different challenges for the raced process. We now test each variant in the `test-dirstate-status-race.t` tests.

dirstate-v2: add devel config option to control write behavior This will help us to write predictable tests checking behavior in each case.

dirstate: use more than a bool to control append behavior When writing dirstate-v2, we might either append to the existing file, or create a new file. We are about to introduce some configuration to control this behavior. As a prelude, we change the current way the behavior was automatically controlled to make the change smaller/clearer.

dirstate: cover each dirstate version when testing for status race Previously we were only testing it with the default (dirstate-v1 currently). Now we explicitly test each variant.

dirstate: test a `hg status` raced by another `hg status` This shows that `rhg` is misbehaving here.

dirstate: test a `hg status` raced by a `hg update` This shows that `rhg` is misbehaving here.

dirstate: test a `hg status` raced by a `hg commit` This shows that `rhg` is misbehaving here.

dirstate: test a `hg status` raced by a `hg add` This shows that `rhg` is misbehaving here.

dirstate: add a way to test races happening during status We add the `devel.sync.status.pre-dirstate-write-file` config option to easily test what happens when other operations happen during the window where `hg status` is done working but has not updated the cache on disk yet. We introduce the framework for testing such races too, actual tests will be added in the next changesets. For now the test is only checking dirstate-v1. We will extend the test coverage later too. Check test documentation for details. Code change from Raphaël Gomès <rgomes@octobus.net> Test change from Pierre-Yves David <pierre-yves.david@octobus.net>

testing: introduce util function to synchronize concurrent commands on files This is an extension of mechanisms that the tests have been using for a while. To be able to also control the execution in Rust, we introduce utility to perform such `wait_on_file` logic based on some configuration value. This will be used in the tests introduced in the next changesets.

hghave: drop py27 and py35 support These versions of python are no longer supported by Mercurial.

hghave: byteify a path passed to a core API It looks like this predicate isn't used at all(?)

dirstate: add some debug output when writing the dirstate This helps debugging Mercurial behavior in the dirstate-v2 case.

run-tests: make it possible to nest conditionals This is not that hard to implement and makes our life easier on a regular basis.

rust: box ConfigValueParseError to avoid large result types clippy emits a warning that all the Result types are way too large because of HgError includes ConfigValueParseError as one of the variants, so its size is 136 bytes. By boxing ConfigValueParseError we're hopefully making everything faster "for free".

dirstate: drop `identity` from the public API We no longer needs it.

delta-find: rename `delta-reuse-policy` to `pulled-delta-reuse-policy` This make it clearer which type of delta we are talking about.

config-item: declare undeclared path suboption This should prevent issue like the previous commit.

delta-find: declare the "paths..*:delta-reuse-policy option While looking into renaming the option I realized it was not declared.

delta-find: adjust the default candidate group chunk size We move from 10 to 20 as the default as some usage in the wild saw a small degradation in storage quality when using `10`.

record: extract a closure to the module level This clean up is almost as gratuituous as this closure was.

record: drop a now useless overlay that grab the lock Since 28dfb2df4ab9, commit grab the wlock and the extra layer grabing the lock in record is no longer needed. We clean up the code to make this simpler (and add a small assert for extra security against future change).

bundlerepo: fix string interpolation Matt Harbison is saying we cannot `%s` a type into a byte string and that seems reasonable.

bundlerepo: apply phase data stored in the bundle instead of assuming `draft` The phase information contained in the changegroup part and the explicit `phase-heads` part are now taken in account. Initial changes and test by Matt Harbison, code rework by Pierre-Yves David.

bundlerepo: handle changegroup induced phase movement in the associated method These movement comes from handling the changegroup part, so we keeps the code grouped. This will be important when handling more part (and more changegroup part in the future) This induce a small code duplication, but it does not looks terrible.

bundlerepo: move most attribute declaration earlier in __init__ The expected attribute are clearer this way. The bundle handling code is responsible for setting most of it.

bundlerepo: move the handling of bundl1 in its own method This should make the overall flow simpler to follow.

bundlerepo: expliclty handing cg part from bundle2 We will handle other types of parts soon (phase-heads) so we need some cleanup first.

transaction: use the standard transaction mechanism to backup branch Branch is a bit special : - It currently does not collaborate with the transaction (or any scoping) for writing (this is bad) - It can change without the lock being taken (it is protected by `wlock`) So we rely on the same mechanism as for the backup of the other dirstate file: - we only do a backup if we hold the wlock - we force a backup though the transaction Since "branch" write does not collaborate with the transaction, we cannot back it up "at the last minute" as we do for the dirstate. We have to back it up "upfront". Since we have a backup, the transaction is no longer doing its "quick_abort" and get noisy. Which is quite annoying. To work around this, and to avoid jumping in yet-another-rabbit-hole of "getting branch written properly", I am doing horrible things to the transaction in the meantime. We should be able to get this code go away during the next cycle. In the meantime, I prefer to take this small stop so that we stop abusing the "journal" and "undo" mechanism instead of the proper backup mechanism of the transaction. Also note that this change regress the warning message for the legacy fallback introduced in 2008 when issue902 got fixed in dd5a501cb97f (Mercurial 1.0). I feel like this is fine as issue 902 remains fixed, and this would only affect people deploying a mix of 15 year old Mercurial and modern mercurial, and using branch and rollback extensively.

transaction: no longer explicitly cache bookmarks The transaction file generation is already dealing with the backup for this. So, no need to duplicate such backup.

transaction: no longer explicitly cache phaseroots The transaction file generation is already dealing with the backup for this. So, no need to duplicate such backup.

narrow: enforce that narrow spec is written within a transaction

narrow: write the narrow spec in a transaction during share It will be simpler if all write happens within transaction.

narrow: open the transaction sooner when unbundling That way, the narrow spec changes will be done within a transaction.

narrow: write the narrow spec in a transaction during clone It will be simpler if all write happens within transaction.

narrow: drop the dedicated backup code Now that the transaction manage the writes, we can simply use the transaction for backup. Some extra cleanup to ensure all changes happens within a transaction will be made in the next changesets.

narrow: delegate the dirstate's narrow spec writing to the transaction This make it more transactional and will help us to simplify their backup. The implementation is not great, but it keep the patch simple as this is not the time for a larger refactoring yet.

narrow: delegate the narrow spec writing to the transaction This make it more transactional and will help us to simplify their backup. The implementation is not great, but it keep the patch simple as this is not the time for a larger refactoring yet.

narrow: get the narrow patterns from the repository object instead of disk Relying on disk data make the transactionally of this change complicated, so let us start reading data from other API instead.

narrow: widden the lock context in `tracking` The tracking configuration we modify must be read under lock. So we grab the lock sooner.

narrow: move `only_show` handling sooner in `tracked` This will help us to improve the locking scope in the `tracked` command.

tests: in test-fncache.t, be more tolerant to the "Killed" message format For example, on my box I'm seeing: $TESTTMP.sh: line 250: $pid Killed hg ci -qAm z

convert: use a priority queue for sorting commits, to make sorting faster To achieve this, we turn commit sorters into classes so they can encapsulate state. This reduces the sorting time from ~30s to ~10s on a 500k-commit prefix of a repo I tried to convert. (and probably reduces the time to sort the whole repo from many tens of minutes to minutes, but I didn't try that again) The date caching gets removed because priority queue already caches the key.

dirstate-v2: don't mmap the data file when on NFS `mmap` on NFS will trigger a SIGBUS when the mmap'ed file is deleted, which wouldn't work in our case. Also, the performance advantage of using mmap on NFS is debatable at best.

rust-dirstate: trace append/no append to help debugging

rust: add debug log about skipping dirstate update

test-dirstate: use more robust method to trigger a data-file append The previous method was fragile and somewhat flaky on fast machines.

typing: add the return type hint to pycompat.rangelist() Not bothering with the args, because there are a few overloads and only 2 callers in the codebase, one of which is a test.

typing: add type hints to pycompat.maplist() The typeshed hints define 5 overloads with an increasing number of parameters on the passed function, and then a catchall that ignores the argument list on the passed function and allows an `*iterators` arg. All of our uses are fulfilled by the 1 function + 1 iterable overload, but add the second overload as a hint in case it's needed in the future.

branching: merge stable into default This show that the recent changes on default fixed the issue with transaction overwriting content in `test-transaction-wc-rollback-race.t`

transaction: tests we don't overwrite bookmark activation on abort We actually do not! Great.

transaction: tests we don't overwrite updates on abort spoiler: we do… /o\

transaction: tests we don't overwrite branch changes on abort We actually do not! Great. …Why are doing a backup of the `branch` files at transaction creation then‽

transaction: tests we don't overwrite tracking to changed file on abort spoiler: we do…

transaction: the base of a new test file checking transaction abort issue See inline documentation for details. See other changesets for actual cases.

setup: support building from an ongoing merge Before this change the two parents from the merge would duplicate some command output and modify some others in a way python 3.11 chokes on.

rhg: in path_encode, simplify a bit more Use the slices for `basename` and `ext` instead of dealing with offsets.

rhg: in path_encode, be a bit more conservative about memory usage Use [shrink_to_fit] to match the previous behavior more closely, and potentially save (a tiny bit) of memory. FWIW, I suspect this is unnecessary, but this whole MR is about simplifying things while preserving any existing optimizations.

rhg: small refactor: stop using a magical constant "+ 1" Instead, directly do what the "+ 1" was supposed to achive: call hash_encode.

rhg: in path_encode, use Vec directly instead of VecDest No need to have a trivial wrapper over the type. There's nothing confusing about vec.write_bytes(...), after all.

rhg: in path_encode, split Dest into VecDest and MeasureDest Two separate types make the write semantics easier to understand because we can consider the two sinks separately. Having two independent compiled functions for size measurement and for actual encoding seems likely to improve performance, too. (and maybe we should get rid of measurement altogether) Getting rid of [Dest] also removes the ugly option rewrapping code, which is good.

rhg: use generic DestArr in hash_mangle This simplifies code a bit more, but comes with an extra memory copy in case [destlen == dest_vec.len()]. This is probably fine, but a follow-up change is removing that too.

rhg: in path_encode, make DestArr generic over its size