bundlerepo: uncompress changegroup in bundle1 case only Uncompressing bundle2 needs to be handled differently.

bundlerepo: move temp-bundle writing logic into a closure We will reuse this logic for bundle2

shelve: properly process bundle2 bundle

histedit: properly apply bundle2 backups If the histedit fails, we restore a backup. We make sure this backup bundle can be in bundle2 format as general delta will require.

strip: pass source and url to bundle2 processing Restoring from a 'bundle2' was missing this data.

unbundle: use 'url' argument with applybundle This will let the hook get access to the 'url' value.

applybundle: take url as argument We allow specifying the url to carry it to hooks. This gets us closer to 'bundle1.apply(...)' and will allow us to remove regressions in multiple place where we forget to pass the url to hooks.

unbundle: use 'source' argument with applybundle This will let the hook get access to the 'source' value.

applybundle: take source as argument We allow specifying the source to carry it to hooks. This gets us closer to 'bundle1.apply(...)' and will allow us to remove regressions in multiple places where we forget to pass the source to hooks.

applybundle: set 'bundle2=1' env for all transaction This should be set for all bundle2 application, we enforce that at a low level. This is for courtesy with hooks.

unbundle: use 'bundle2.applybundle' This is one such place.

bundle2: introduce an "applybundle" function There is a case where the intent is clear and the transaction is not optional. We want to be able to alter that transaction in a wide and easy way. We cannot get a unified '.apply(repo)' method for bundle1 and bundle2 yet because the api are still a bit too far apart. But this is a good step forward to get the rc out.

commands.resolve: conclude merge driver if no unresolved files are left This can happen when either 'hg resolve --all' is called or a driver-resolved file is explicitly requested. This is done as part of 'hg resolve --all' so that users still have a chance to test their changes before committing them. The exact semantics here are still to be decided. This does not impact any non-experimental features. Thanks to Pierre-Yves David for some advice about this behavior in particular, and merge drivers in general.

commands.resolve: call driverpreprocess if we haven't run it yet We need to be careful about allowing --mark and --unmark to keep working -- we don't want the user to be stuck in a weird state. The exact behavior here is still to be decided, though.

merge.applyupdates: call driverconclude after performing merge actions This will be a chance for the merge driver to finish resolving or generating any driver-resolved files. As before, having a separate error state from 'unresolved' is too big a refactoring for now, so we hack around it by setting unresolved to a positive value when necessary.

merge.applyupdates: call driverpreprocess before starting merge actions We also need to update our internal state to whatever state driverpreprocess leaves it in. Adding an error state separate from the unresolved count is too big a refactoring for now, so we hack around it by setting it to a positive value to indicate an error state.

merge: add stubs for preprocess and conclude steps of merge driver The exact semantics for what should happen (particularly with respect to error handling) are still a bit hard to pin down, so I think it's better to experiment with it as an extension for now. For now this stub will act as a convenient point for extensions to hook on.

commands.resolve: don't allow users to mark or unmark driver-resolved files Users will often be in the habit of running 'hg resolve --mark --all' after resolving merge conflicts in source files. We need to make sure this doesn't cause driver-resolved files to be marked. 'hg resolve --all' will resolve driver-resolved files, though. The weird conditional structure is to accommodate an upcoming patch.

resolve: only create wctx once This allows for status and other results on the wctx to be cached between iterations.

dirstate: fix invalid reference to self.ui

spelling: trivial spell checking

mq: consistently use qrefresh

grammar: use does instead of do where appropriate

l10n: use %d instead of %s for numbers

check-code: block non-portable pipe-and

test-shelve: do not use non-portable pipe-and

parsers: suppress warning of signed and unsigned comparison at nt_init Spotted by CC=clang CFLAGS='-Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wshorten-64-to-32': mercurial/parsers.c:1580:24: warning: comparison of integers of different signs: 'Py_ssize_t' (aka 'long') and 'unsigned long' [-Wsign-compare] if (self->raw_length > INT_MAX / sizeof(nodetree)) {

parsers: correct type of temporary variables for dirstate tuple fields These fields are defined as int. This eliminates the following warning spotted by CC=clang CFLAGS='-Wall -Wextra -Wno-missing-field-initializers -Wno-unused-parameter -Wshorten-64-to-32': mercurial/parsers.c:625:29: warning: comparison of integers of different signs: 'uint32_t' (aka 'unsigned int') and 'int' [-Wsign-compare] if (state == 'n' && mtime == now) {

test: enforce v1 type in 'test-bundle2-remote-changegroup.t' This bundle command is meant to generate a bundle1, we enforce that to avoid test's misbehavior when moving to general-delta by default.

test: enforce v1 type in 'test-bundle2-format.t' This bundle command is meant to generate a bundle1, we enforce that to avoid test's misbehavior when moving to general-delta by default.

editmerge: properly quote variables Previously, files with spaces would break editmerge.

commands.resolve: print out warning when only driver-resolved files remain 'hg resolve --all' will be the canonical way to run the 'conclude' step of the merge driver.

merge.mergestate: set merge driver state to 's' if there's none present This allows mergestate.mdstate() to return 's' if there's nothing to be done on the merge driver end.

merge.mergestate: only check for merge driver when property is accessed Otherwise 'hg update --clean', 'hg rebase --abort' etc wouldn't work.

localrepo.commit: abort if merge driver's conclude step hasn't been run yet This case also requires 'hg resolve --all' to be run before continuing.

merge.mergestate: add a way to get the merge driver state This will be useful to check what the status of the merge driver is.

merge.mergestate: add a way to get the other side of the merge It's surprising there was no API at all for this until now. In any case this will be needed from custom merge drivers.

commands.resolve: support printing out driver-resolved files There's no user-visible way to mark files as driver-resolved, so this status won't be visible.

histedit: make histedit prune when obsolete is enabled Back in June we made histedit use obsolete markers to cleanup when possible. This was rolled back as part of 54f9561088c7 (which should have only rolled back the --abort stuff, but rolled back everything). This caused a nasty bug when used in conjuction with the inhibit+directaccess extensions where histedit would leave old nodes around even after they had been squashed away. The root of the problem is that we first clean up old nodes, and then we clean up temp nodes. In the first pass, when we obsoleted old nodes, some would become unobsolete because they had temp nodes on top of them, thus making them stick around even after the histedit finished. The fix is to A) move the temp node cleanup to be before the old node cleanup (since they are topological on top of the old nodes), and B) use obsolete markers instead of stripping.

clonebundles: rewrite documentation There are a lot of considerations server operators need to know before deploying clone bundles. They should be documented. So I rewrote the extension docs to contain this information.

exchange: support streaming clone bundles in clone bundles Now that we have support for detecting compatible stream clone bundles in bundle specifications, we can safely add support for applying stream clone bundles to the clone bundles feature.

exchange: parse requirements from stream clone specification string Stream clone bundles can only be consumed if the consumer supports the exact format requirements that were present on the producer. This patch adds support for encoding and verifying the format requirements on the bundle specification string for a stream clone bundle are supported by the local repository. If they aren't, we raise an UnsupportedBundleSpecification, just like we do when an unknown compression or bundle type is encountered. The impetus for this patch is so the clone bundles manifest can advertise stream clone bundles and so clients can filter out stream clones with unsupported format requirements. e.g. a stream clone produced with the not-yet-invented "revlogv2" format will be ignored by clients that only support "revlogv1."

exchange: support parameters in bundle specification strings Sometimes a basic type string is not sufficient for representing the contents of a bundle. Take bundle2 for example: future bundle2 files may contain parts that today's bundle2 parser can't read. Another example is stream clone data. These require clients to support specific repository formats or they won't be able to read the written files. In both scenarios, we need to describe additional metadata beyond the outer container type. Furthermore, this metadata behaves more like an unordered set, so an order-based declaration format (such as static strings) is not sufficient. We introduce support for "parameters" into the bundle specification string. These are essentially key-value pairs that can be used to encode additional metadata about the bundle. Semicolons are used as the delimiter partially to increase similarity to MIME parameter values (see RFC 2231) and because they are relatively safe from the command line (although values will need quotes to avoid interpretation as multiple shell commands). Alternatives considered were spaces (a bit annoying to encode) and '&' (similar to URL query strings) (which will do bad things in a shell if unquoted). The parsing function now returns a dict of parsed parameters and consumers have been updated accordingly.

commands: support consuming stream clone bundles For the same reasons that we don't produce stream clone bundles with `hg bundle`, we don't support consuming stream clone bundles with `hg unbundle`. We introduce a complementary debug command for applying stream clone bundles. This command is mostly to facilitate testing. Although it may be used to manually apply stream clone bundles until a more formal mechanism is (possibly) adopted.

commands: support creating stream clone bundles Now that we have support for recognizing the streaming clone bundle type, add a debug command for creating them. I decided to create a new debug command instead of adding support to `hg bundle` because stream clone bundles are not exactly used the same way as normal bundle files and I don't want to commit to supporting them through the official `hg bundle` command forever. A debug command, however, can be changed without as much concern for backwards compatibility. As part of this, `hg bundle` will explicitly reject requests to produce stream bundles. This command will be required by server operators using stream clone bundles with the clone bundles feature.

exchange: support for streaming clone bundles Now that we have a mechanism to produce and consume streaming clone bundles, we need to teach the human-facing bundle specification parser and the internal bundle file header reading code to be aware of this new format. This patch does so. For the human-facing bundle specification, we choose the name "packed" to describe "streaming clone bundles" because the bundle is essentially a "pack" of raw revlog files that are "packed" together. There should probably be a bikeshed over the name, especially since it is human facing.

streamclone: support for producing and consuming stream clone bundles Up to this point, stream clones only existed as a dynamically generated data format produced and consumed during streaming clones. In order to support this efficient cloning format with the clone bundles feature, we need a more formal, on disk representation of the streaming clone data. This patch introduces a new "bundle" type for streaming clones. Unlike existing bundles, it does not contain changegroup data. It does, however, share the same concepts like the 4 byte header which identifies the type of data that follows and the 2 byte abbreviation for compression types (of which only "UN" is currently supported). The new bundle format is essentially the existing stream clone version 1 data format with some headers at the beginning. Content negotiation at stream clone request time checked for repository format/requirements compatibility before initiating a stream clone. We can't do active content negotiation when using clone bundles. So, we put this set of requirements inside the payload so consumers have a built-in mechanism for checking compatibility before reading and applying lots of data. Of course, we will also advertise this requirements set in clone bundles. But that's for another patch. We currently don't have a mechanism to produce and consume this new bundle format. This will be implemented in upcoming patches. It's worth noting that if a legacy client attempts to `hg unbundle` a stream clone bundle (with the "HGS1" header), it will abort with: "unknown bundle version S1," which seems appropriate.

spelling: fix typo in transaction error messages

transaction: reorder unlinking .hg/journal and .hg/journal.backupfiles After this reordering, absence of '.hg/journal' just before starting new transaction means also absence of '.hg/journal.backupfiles'. In this case, all temporary files for preceding transaction should be completely unlinked, and HG_PENDING doesn't cause unintentional reading stalled temporary files in. Otherwise, 'repo.transaction()' raises exception with "run 'hg recover' to clean up transaction" hint.

merge: make in-memory changes visible to external update hooks 51844b8b5017 (while 3.4 code-freeze) made all 'update' hooks run after releasing wlock for visibility of in-memory dirstate changes. But this breaks paired invocation of 'preupdate' and 'update' hooks. For example, 'hg backout --merge' for TARGET revision, which isn't parent of CURRENT, consists of steps below: 1. update from CURRENT to TARGET 2. commit BACKOUT revision, which backs TARGET out 3. update from BACKOUT to CURRENT 4. merge TARGET into CURRENT Then, we expects hooks to run in the order below: - 'preupdate' on CURRENT for (1) - 'update' on TARGET for (1) - 'preupdate' on BACKOUT for (3) - 'update' on CURRENT for (3) - 'preupdate' on TARGET for (4) - 'update' on CURRENT/TARGET for (4) But hooks actually run in the order below: - 'preupdate' on CURRENT for (1) - 'preupdate' on BACKOUT for (3) - 'preupdate' on TARGET for (4) - 'update' on TARGET for (1), but actually on CURRENT/TARGET - 'update' on CURRENT for (3), but actually on CURRENT/TARGET - 'update' on CURRENT for (4), but actually on CURRENT/TARGET Root cause of the issue focused by 51844b8b5017 is that external 'update' hook process can't view in-memory changes (especially, of dirstate), because they aren't written out until the end of transaction (or wlock). Now, hooks can be invoked just after updating, because previous patches made in-memory changes visible to external process. This patch may break backward compatibility from the point of view of "scheduling hook execution", but should be reasonable because 'update' hooks had been executed in this order before 3.4. This patch tests "hg backout" and "hg unshelve", because the former activates the transaction before 'update' hook invocation, but the former doesn't.

hook: centralize passing HG_PENDING to external hook process This patch centralizes passing HG_PENDING to external hook process into '_exthook()'. To make in-memory changes visible to external hook process, this patch does: - write (or schedule to write) in-memory dirstate changes, and - set HG_PENDING environment variable, if: - a transaction is running, and - there are in-memory changes to be visible This patch tests some commands with some hooks, because transaction activity of a same hook differs from each other ("---": "not tested"). ======== ========= ========= ============ command preupdate precommit pretxncommit ======== ========= ========= ============ unshelve o --- --- backout x --- --- import --- o o qrefresh --- x o ======== ========= ========= ============ Each hooks are examined separately to prevent in-memory changes from being visible to external process accidentally by side effect of hooks previously invoked.

cmdutil: make in-memory changes visible to external editor (issue4378) Before this patch, external editor process for the commit log can't view some in-memory changes (especially, of dirstate), because they aren't written out until the end of transaction (or wlock). This causes unexpected output of Mercurial commands spawned from that editor process. To make in-memory changes visible to external editor process, this patch does: - write (or schedule to write) in-memory dirstate changes, and - set HG_PENDING environment variable, if: - a transaction is running, and - there are in-memory changes to be visible "hg diff" spawned from external editor process for "hg qrefresh" shows: - "changes newly imported into the topmost" before 49148d7868df(*) - "all changes recorded in the topmost by refreshing" after this patch (*) 49148d7868df changed steps invoking editor process Even though backward compatibility may be broken, the latter behavior looks reasonable, because "hg diff" spawned from the editor process consistently shows "what changes new revision records" regardless of invocation context. In fact, issue4378 itself should be resolved by 800e090e9c64, which made 'repo.transaction()' write in-memory dirstate changes out explicitly before starting transaction. It also made "hg qrefresh" imply 'dirstate.write()' before external editor invocation in call chain below. - mq.queue.refresh - strip.strip - repair.strip - localrepository.transaction - dirstate.write - localrepository.commit - invoke external editor Though, this patch has '(issue4378)' in own summary line to indicate that issues like issue4378 should be fixed by this. BTW, this patch adds '-m' option to a 'hg ci --amend' execution in 'test-commit-amend.t', to avoid invoking external editor process. In this case, "unsure" states may be changed to "clean" according to timestamp or so on. These changes should be written into pending file, if external editor invocation is required, Then, writing dirstate changes out breaks stability of test, because it shows "transaction abort!/rollback completed" occasionally. Aborting after editor process invocation while commands below may cause similar instability of tests, too (AFAIK, there is no more such one, at this revision) - commit --amend - without --message/--logfile - import - without --message/--logfile, - without --no-commit, - without --bypass, - one of below, and - patch has no description text, or - with --edit - aborting at the 1st patch, which adds or removes file(s) - if it only changes existing files, status is checked only for changed files by 'scmutil.matchfiles()', and transition from "unsure" to "normal" in dirstate doesn't occur (= dirstate isn't changed, and written out) - aborting at the 2nd or later patch implies other pending changes (e.g. changelog), and always causes showing "transaction abort!/rollback completed"

dirstate: show develwarn for write() invocation without transaction This is used to detect 'dirstate.write()' invocation without the value gotten by 'repo.currenttransaction()' (mainly focused on 3rd party extensions).

dirstate: make dirstate.write() callers pass transaction object to it Now, 'dirstate.write(tr)' delays writing in-memory changes out, if a transaction is running. This may cause treating this revision as "the first bad one" at bisecting in some cases using external hook process inside transaction scope, because some external hooks and editor process are still invoked without HG_PENDING and pending changes aren't visible to them. 'dirstate.write()' callers below in localrepo.py explicitly use 'None' as 'tr', because they can assume that no transaction is running: - just before starting transaction - at closing transaction, or - at unlocking wlock

dirstate: remove layering violation around writing dirstate out This violation, which passes repo object to dirstate, was introduced by 09bb1ee7e73e. This patch uses 'False' instead of 'None' as default value of 'tr' argument, to distinguish "None as repo.currenttransaction() result" from "legacy invocation without explicit tr passing".

dirstateguard: remove layering violation around saving/restoring backup This violation, which passes 'repo' object to dirstate, was introduced by 59b5e8844eb0 and 020b12d591f3.

rebase: reduce scope of try catch in restorestate Refactoring by reduce the scope of the try catch block so that it only captures what it needs. I could have made it smaller but another patch in the series will add onto it.

rebase: on abort delete rebase state file no matter what When a user's repository is in an unfinished rebase state and they choose to abort, at a minimum, the repo should be out of that state. We've found situations where the user could not leave the state unless manually deleting the rebasestate file. This fix ensures that no matter what exception may be raised during the abort, the rebase state file will be deleted, the user will be out of the rebase state and they can get their repository into a workable condition.

contrib: make editmerge look for merge markers at the beginning of the line This fix adds a caret to the start of the regex looking for merge markers. This avoids the issue arises when you've real merge conflicts in a file that tests for the existance of merge markers in test output. Editmerge will not open on the fake/tested merge markers because they'll be indented in.

commit: abort when a committemplate is not changed If a committemplate is provided and no message is provided on the command line, and no edits are made to the commit template, then abort the commit.

localrepo.commit: disallow commit when driver-resolved files exist This code will not currently be activated because there's no code to mark files as driver-resolved in core. This point is also somewhat hard to plug into from extensions.

merge.mergestate: add a generator for driver-resolved files Just like for unresolved files above, we need to be able to tell what files are driver-resolved.

hook: for python hooks, also return whether an exception was raised The hook code treats python hooks raising an exception and returning True as the exact same. This is OK for hooks themselves, but other code that wants to invoke external code using the same underlying code is a bit more interested in making a distinction.

hook.runhooks: return a dict of result values This will be useful to other calling code that would be interested in what the individual hooks return.