test-shelve: do not use non-portable pipe-and

parsers: suppress warning of signed and unsigned comparison at nt_init Spotted by CC=clang CFLAGS='-Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wshorten-64-to-32': mercurial/parsers.c:1580:24: warning: comparison of integers of different signs: 'Py_ssize_t' (aka 'long') and 'unsigned long' [-Wsign-compare] if (self->raw_length > INT_MAX / sizeof(nodetree)) {

parsers: correct type of temporary variables for dirstate tuple fields These fields are defined as int. This eliminates the following warning spotted by CC=clang CFLAGS='-Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wshorten-64-to-32': mercurial/parsers.c:625:29: warning: comparison of integers of different signs: 'uint32_t' (aka 'unsigned int') and 'int' [-Wsign-compare] if (state == 'n' && mtime == now) {

test: enforce v1 type in 'test-bundle2-remote-changegroup.t' This bundle command is meant to generate a bundle1, we enforce that to avoid test's misbehavior when moving to general-delta by default.

test: enforce v1 type in 'test-bundle2-format.t' This bundle command is meant to generate a bundle1, we enforce that to avoid test's misbehavior when moving to general-delta by default.

editmerge: properly quote variables Previously, files with spaces would break editmerge.

commands.resolve: print out warning when only driver-resolved files remain 'hg resolve --all' will be the canonical way to run the 'conclude' step of the merge driver.

merge.mergestate: set merge driver state to 's' if there's none present This allows mergestate.mdstate() to return 's' if there's nothing to be done on the merge driver end.

merge.mergestate: only check for merge driver when property is accessed Otherwise 'hg update --clean', 'hg rebase --abort' etc wouldn't work.

localrepo.commit: abort if merge driver's conclude step hasn't been run yet This case also requires 'hg resolve --all' to be run before continuing.

merge.mergestate: add a way to get the merge driver state This will be useful to check what the status of the merge driver is.

merge.mergestate: add a way to get the other side of the merge It's surprising there was no API at all for this until now. In any case this will be needed from custom merge drivers.

commands.resolve: support printing out driver-resolved files There's no user-visible way to mark files as driver-resolved, so this status won't be visible.

histedit: make histedit prune when obsolete is enabled Back in June we made histedit use obsolete markers to cleanup when possible. This was rolled back as part of 54f9561088c7 (which should have only rolled back the --abort stuff, but rolled back everything). This caused a nasty bug when used in conjuction with the inhibit+directaccess extensions where histedit would leave old nodes around even after they had been squashed away. The root of the problem is that we first clean up old nodes, and then we clean up temp nodes. In the first pass, when we obsoleted old nodes, some would become unobsolete because they had temp nodes on top of them, thus making them stick around even after the histedit finished. The fix is to A) move the temp node cleanup to be before the old node cleanup (since they are topological on top of the old nodes), and B) use obsolete markers instead of stripping.

clonebundles: rewrite documentation There are a lot of considerations server operators need to know before deploying clone bundles. They should be documented. So I rewrote the extension docs to contain this information.

exchange: support streaming clone bundles in clone bundles Now that we have support for detecting compatible stream clone bundles in bundle specifications, we can safely add support for applying stream clone bundles to the clone bundles feature.

exchange: parse requirements from stream clone specification string Stream clone bundles can only be consumed if the consumer supports the exact format requirements that were present on the producer. This patch adds support for encoding and verifying the format requirements on the bundle specification string for a stream clone bundle are supported by the local repository. If they aren't, we raise an UnsupportedBundleSpecification, just like we do when an unknown compression or bundle type is encountered. The impetus for this patch is so the clone bundles manifest can advertise stream clone bundles and so clients can filter out stream clones with unsupported format requirements. e.g. a stream clone produced with the not-yet-invented "revlogv2" format will be ignored by clients that only support "revlogv1."

exchange: support parameters in bundle specification strings Sometimes a basic type string is not sufficient for representing the contents of a bundle. Take bundle2 for example: future bundle2 files may contain parts that today's bundle2 parser can't read. Another example is stream clone data. These require clients to support specific repository formats or they won't be able to read the written files. In both scenarios, we need to describe additional metadata beyond the outer container type. Furthermore, this metadata behaves more like an unordered set, so an order-based declaration format (such as static strings) is not sufficient. We introduce support for "parameters" into the bundle specification string. These are essentially key-value pairs that can be used to encode additional metadata about the bundle. Semicolons are used as the delimiter partially to increase similarity to MIME parameter values (see RFC 2231) and because they are relatively safe from the command line (although values will need quotes to avoid interpretation as multiple shell commands). Alternatives considered were spaces (a bit annoying to encode) and '&' (similar to URL query strings) (which will do bad things in a shell if unquoted). The parsing function now returns a dict of parsed parameters and consumers have been updated accordingly.

commands: support consuming stream clone bundles For the same reasons that we don't produce stream clone bundles with `hg bundle`, we don't support consuming stream clone bundles with `hg unbundle`. We introduce a complementary debug command for applying stream clone bundles. This command is mostly to facilitate testing. Although it may be used to manually apply stream clone bundles until a more formal mechanism is (possibly) adopted.

commands: support creating stream clone bundles Now that we have support for recognizing the streaming clone bundle type, add a debug command for creating them. I decided to create a new debug command instead of adding support to `hg bundle` because stream clone bundles are not exactly used the same way as normal bundle files and I don't want to commit to supporting them through the official `hg bundle` command forever. A debug command, however, can be changed without as much concern for backwards compatibility. As part of this, `hg bundle` will explicitly reject requests to produce stream bundles. This command will be required by server operators using stream clone bundles with the clone bundles feature.

exchange: support for streaming clone bundles Now that we have a mechanism to produce and consume streaming clone bundles, we need to teach the human-facing bundle specification parser and the internal bundle file header reading code to be aware of this new format. This patch does so. For the human-facing bundle specification, we choose the name "packed" to describe "streaming clone bundles" because the bundle is essentially a "pack" of raw revlog files that are "packed" together. There should probably be a bikeshed over the name, especially since it is human facing.

streamclone: support for producing and consuming stream clone bundles Up to this point, stream clones only existed as a dynamically generated data format produced and consumed during streaming clones. In order to support this efficient cloning format with the clone bundles feature, we need a more formal, on disk representation of the streaming clone data. This patch introduces a new "bundle" type for streaming clones. Unlike existing bundles, it does not contain changegroup data. It does, however, share the same concepts like the 4 byte header which identifies the type of data that follows and the 2 byte abbreviation for compression types (of which only "UN" is currently supported). The new bundle format is essentially the existing stream clone version 1 data format with some headers at the beginning. Content negotiation at stream clone request time checked for repository format/requirements compatibility before initiating a stream clone. We can't do active content negotiation when using clone bundles. So, we put this set of requirements inside the payload so consumers have a built-in mechanism for checking compatibility before reading and applying lots of data. Of course, we will also advertise this requirements set in clone bundles. But that's for another patch. We currently don't have a mechanism to produce and consume this new bundle format. This will be implemented in upcoming patches. It's worth noting that if a legacy client attempts to `hg unbundle` a stream clone bundle (with the "HGS1" header), it will abort with: "unknown bundle version S1," which seems appropriate.

spelling: fix typo in transaction error messages

transaction: reorder unlinking .hg/journal and .hg/journal.backupfiles After this reordering, absence of '.hg/journal' just before starting new transaction means also absence of '.hg/journal.backupfiles'. In this case, all temporary files for preceding transaction should be completely unlinked, and HG_PENDING doesn't cause unintentional reading stalled temporary files in. Otherwise, 'repo.transaction()' raises exception with "run 'hg recover' to clean up transaction" hint.

merge: make in-memory changes visible to external update hooks 51844b8b5017 (while 3.4 code-freeze) made all 'update' hooks run after releasing wlock for visibility of in-memory dirstate changes. But this breaks paired invocation of 'preupdate' and 'update' hooks. For example, 'hg backout --merge' for TARGET revision, which isn't parent of CURRENT, consists of steps below: 1. update from CURRENT to TARGET 2. commit BACKOUT revision, which backs TARGET out 3. update from BACKOUT to CURRENT 4. merge TARGET into CURRENT Then, we expects hooks to run in the order below: - 'preupdate' on CURRENT for (1) - 'update' on TARGET for (1) - 'preupdate' on BACKOUT for (3) - 'update' on CURRENT for (3) - 'preupdate' on TARGET for (4) - 'update' on CURRENT/TARGET for (4) But hooks actually run in the order below: - 'preupdate' on CURRENT for (1) - 'preupdate' on BACKOUT for (3) - 'preupdate' on TARGET for (4) - 'update' on TARGET for (1), but actually on CURRENT/TARGET - 'update' on CURRENT for (3), but actually on CURRENT/TARGET - 'update' on CURRENT for (4), but actually on CURRENT/TARGET Root cause of the issue focused by 51844b8b5017 is that external 'update' hook process can't view in-memory changes (especially, of dirstate), because they aren't written out until the end of transaction (or wlock). Now, hooks can be invoked just after updating, because previous patches made in-memory changes visible to external process. This patch may break backward compatibility from the point of view of "scheduling hook execution", but should be reasonable because 'update' hooks had been executed in this order before 3.4. This patch tests "hg backout" and "hg unshelve", because the former activates the transaction before 'update' hook invocation, but the former doesn't.

hook: centralize passing HG_PENDING to external hook process This patch centralizes passing HG_PENDING to external hook process into '_exthook()'. To make in-memory changes visible to external hook process, this patch does: - write (or schedule to write) in-memory dirstate changes, and - set HG_PENDING environment variable, if: - a transaction is running, and - there are in-memory changes to be visible This patch tests some commands with some hooks, because transaction activity of a same hook differs from each other ("---": "not tested"). ======== ========= ========= ============ command preupdate precommit pretxncommit ======== ========= ========= ============ unshelve o --- --- backout x --- --- import --- o o qrefresh --- x o ======== ========= ========= ============ Each hooks are examined separately to prevent in-memory changes from being visible to external process accidentally by side effect of hooks previously invoked.

cmdutil: make in-memory changes visible to external editor (issue4378) Before this patch, external editor process for the commit log can't view some in-memory changes (especially, of dirstate), because they aren't written out until the end of transaction (or wlock). This causes unexpected output of Mercurial commands spawned from that editor process. To make in-memory changes visible to external editor process, this patch does: - write (or schedule to write) in-memory dirstate changes, and - set HG_PENDING environment variable, if: - a transaction is running, and - there are in-memory changes to be visible "hg diff" spawned from external editor process for "hg qrefresh" shows: - "changes newly imported into the topmost" before 49148d7868df(*) - "all changes recorded in the topmost by refreshing" after this patch (*) 49148d7868df changed steps invoking editor process Even though backward compatibility may be broken, the latter behavior looks reasonable, because "hg diff" spawned from the editor process consistently shows "what changes new revision records" regardless of invocation context. In fact, issue4378 itself should be resolved by 800e090e9c64, which made 'repo.transaction()' write in-memory dirstate changes out explicitly before starting transaction. It also made "hg qrefresh" imply 'dirstate.write()' before external editor invocation in call chain below. - mq.queue.refresh - strip.strip - repair.strip - localrepository.transaction - dirstate.write - localrepository.commit - invoke external editor Though, this patch has '(issue4378)' in own summary line to indicate that issues like issue4378 should be fixed by this. BTW, this patch adds '-m' option to a 'hg ci --amend' execution in 'test-commit-amend.t', to avoid invoking external editor process. In this case, "unsure" states may be changed to "clean" according to timestamp or so on. These changes should be written into pending file, if external editor invocation is required, Then, writing dirstate changes out breaks stability of test, because it shows "transaction abort!/rollback completed" occasionally. Aborting after editor process invocation while commands below may cause similar instability of tests, too (AFAIK, there is no more such one, at this revision) - commit --amend - without --message/--logfile - import - without --message/--logfile, - without --no-commit, - without --bypass, - one of below, and - patch has no description text, or - with --edit - aborting at the 1st patch, which adds or removes file(s) - if it only changes existing files, status is checked only for changed files by 'scmutil.matchfiles()', and transition from "unsure" to "normal" in dirstate doesn't occur (= dirstate isn't changed, and written out) - aborting at the 2nd or later patch implies other pending changes (e.g. changelog), and always causes showing "transaction abort!/rollback completed"

dirstate: show develwarn for write() invocation without transaction This is used to detect 'dirstate.write()' invocation without the value gotten by 'repo.currenttransaction()' (mainly focused on 3rd party extensions).

dirstate: make dirstate.write() callers pass transaction object to it Now, 'dirstate.write(tr)' delays writing in-memory changes out, if a transaction is running. This may cause treating this revision as "the first bad one" at bisecting in some cases using external hook process inside transaction scope, because some external hooks and editor process are still invoked without HG_PENDING and pending changes aren't visible to them. 'dirstate.write()' callers below in localrepo.py explicitly use 'None' as 'tr', because they can assume that no transaction is running: - just before starting transaction - at closing transaction, or - at unlocking wlock

dirstate: remove layering violation around writing dirstate out This violation, which passes repo object to dirstate, was introduced by 09bb1ee7e73e. This patch uses 'False' instead of 'None' as default value of 'tr' argument, to distinguish "None as repo.currenttransaction() result" from "legacy invocation without explicit tr passing".