Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Oct 2015 17:00:34 -0700] rev 26759
exchange: support parameters in bundle specification strings
Sometimes a basic type string is not sufficient for representing the
contents of a bundle. Take bundle2 for example: future bundle2 files may
contain parts that today's bundle2 parser can't read. Another example is
stream clone data. These require clients to support specific
repository formats or they won't be able to read the written files. In
both scenarios, we need to describe additional metadata beyond the outer
container type. Furthermore, this metadata behaves more like an
unordered set, so an order-based declaration format (such as static
strings) is not sufficient.
We introduce support for "parameters" into the bundle specification
string. These are essentially key-value pairs that can be used to encode
additional metadata about the bundle.
Semicolons are used as the delimiter partially to increase similarity to
MIME parameter values (see RFC 2231) and because they are relatively
safe from the command line (although values will need quotes to avoid
interpretation as multiple shell commands). Alternatives considered were
spaces (a bit annoying to encode) and '&' (similar to URL query strings)
(which will do bad things in a shell if unquoted).
The parsing function now returns a dict of parsed parameters and
consumers have been updated accordingly.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 15 Oct 2015 13:43:18 -0700] rev 26758
commands: support consuming stream clone bundles
For the same reasons that we don't produce stream clone bundles with `hg
bundle`, we don't support consuming stream clone bundles with `hg
unbundle`. We introduce a complementary debug command for applying
stream clone bundles. This command is mostly to facilitate testing.
Although it may be used to manually apply stream clone bundles until a
more formal mechanism is (possibly) adopted.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 17 Oct 2015 11:40:29 -0700] rev 26757
commands: support creating stream clone bundles
Now that we have support for recognizing the streaming clone bundle
type, add a debug command for creating them.
I decided to create a new debug command instead of adding support to `hg
bundle` because stream clone bundles are not exactly used the same way
as normal bundle files and I don't want to commit to supporting them
through the official `hg bundle` command forever. A debug command,
however, can be changed without as much concern for backwards
compatibility.
As part of this, `hg bundle` will explicitly reject requests to produce
stream bundles.
This command will be required by server operators using stream clone
bundles with the clone bundles feature.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 15 Oct 2015 13:00:45 -0700] rev 26756
exchange: support for streaming clone bundles
Now that we have a mechanism to produce and consume streaming clone
bundles, we need to teach the human-facing bundle specification parser
and the internal bundle file header reading code to be aware of this new
format. This patch does so.
For the human-facing bundle specification, we choose the name "packed"
to describe "streaming clone bundles" because the bundle is essentially
a "pack" of raw revlog files that are "packed" together. There should
probably be a bikeshed over the name, especially since it is human
facing.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 17 Oct 2015 11:14:52 -0700] rev 26755
streamclone: support for producing and consuming stream clone bundles
Up to this point, stream clones only existed as a dynamically generated
data format produced and consumed during streaming clones. In order to
support this efficient cloning format with the clone bundles feature, we
need a more formal, on disk representation of the streaming clone data.
This patch introduces a new "bundle" type for streaming clones. Unlike
existing bundles, it does not contain changegroup data. It does,
however, share the same concepts like the 4 byte header which identifies
the type of data that follows and the 2 byte abbreviation for
compression types (of which only "UN" is currently supported).
The new bundle format is essentially the existing stream clone version 1
data format with some headers at the beginning.
Content negotiation at stream clone request time checked for repository
format/requirements compatibility before initiating a stream clone. We
can't do active content negotiation when using clone bundles. So, we put
this set of requirements inside the payload so consumers have a built-in
mechanism for checking compatibility before reading and applying lots of
data. Of course, we will also advertise this requirements set in clone
bundles. But that's for another patch.
We currently don't have a mechanism to produce and consume this new
bundle format. This will be implemented in upcoming patches.
It's worth noting that if a legacy client attempts to `hg unbundle` a
stream clone bundle (with the "HGS1" header), it will abort with:
"unknown bundle version S1," which seems appropriate.
Matt Mackall <mpm@selenic.com> [Sat, 17 Oct 2015 15:28:02 -0500] rev 26754
spelling: fix typo in transaction error messages
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Fri, 16 Oct 2015 03:29:51 +0900] rev 26753
transaction: reorder unlinking .hg/journal and .hg/journal.backupfiles
After this reordering, absence of '.hg/journal' just before starting
new transaction means also absence of '.hg/journal.backupfiles'.
In this case, all temporary files for preceding transaction should be
completely unlinked, and HG_PENDING doesn't cause unintentional
reading stalled temporary files in.
Otherwise, 'repo.transaction()' raises exception with "run 'hg
recover' to clean up transaction" hint.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 17 Oct 2015 01:15:34 +0900] rev 26752
merge: make in-memory changes visible to external update hooks
51844b8b5017 (while 3.4 code-freeze) made all 'update' hooks run after
releasing wlock for visibility of in-memory dirstate changes. But this
breaks paired invocation of 'preupdate' and 'update' hooks.
For example, 'hg backout --merge' for TARGET revision, which isn't
parent of CURRENT, consists of steps below:
1. update from CURRENT to TARGET
2. commit BACKOUT revision, which backs TARGET out
3. update from BACKOUT to CURRENT
4. merge TARGET into CURRENT
Then, we expects hooks to run in the order below:
- 'preupdate' on CURRENT for (1)
- 'update' on TARGET for (1)
- 'preupdate' on BACKOUT for (3)
- 'update' on CURRENT for (3)
- 'preupdate' on TARGET for (4)
- 'update' on CURRENT/TARGET for (4)
But hooks actually run in the order below:
- 'preupdate' on CURRENT for (1)
- 'preupdate' on BACKOUT for (3)
- 'preupdate' on TARGET for (4)
- 'update' on TARGET for (1), but actually on CURRENT/TARGET
- 'update' on CURRENT for (3), but actually on CURRENT/TARGET
- 'update' on CURRENT for (4), but actually on CURRENT/TARGET
Root cause of the issue focused by 51844b8b5017 is that external
'update' hook process can't view in-memory changes (especially, of
dirstate), because they aren't written out until the end of
transaction (or wlock).
Now, hooks can be invoked just after updating, because previous
patches made in-memory changes visible to external process.
This patch may break backward compatibility from the point of view of
"scheduling hook execution", but should be reasonable because 'update'
hooks had been executed in this order before 3.4.
This patch tests "hg backout" and "hg unshelve", because the former
activates the transaction before 'update' hook invocation, but the
former doesn't.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 17 Oct 2015 01:15:34 +0900] rev 26751
hook: centralize passing HG_PENDING to external hook process
This patch centralizes passing HG_PENDING to external hook process
into '_exthook()'. To make in-memory changes visible to external hook
process, this patch does:
- write (or schedule to write) in-memory dirstate changes, and
- set HG_PENDING environment variable, if:
- a transaction is running, and
- there are in-memory changes to be visible
This patch tests some commands with some hooks, because transaction
activity of a same hook differs from each other ("---": "not tested").
======== ========= ========= ============
command preupdate precommit pretxncommit
======== ========= ========= ============
unshelve o --- ---
backout x --- ---
import --- o o
qrefresh --- x o
======== ========= ========= ============
Each hooks are examined separately to prevent in-memory changes from
being visible to external process accidentally by side effect of hooks
previously invoked.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 17 Oct 2015 01:15:34 +0900] rev 26750
cmdutil: make in-memory changes visible to external editor (issue4378)
Before this patch, external editor process for the commit log can't
view some in-memory changes (especially, of dirstate), because they
aren't written out until the end of transaction (or wlock).
This causes unexpected output of Mercurial commands spawned from that
editor process.
To make in-memory changes visible to external editor process, this
patch does:
- write (or schedule to write) in-memory dirstate changes, and
- set HG_PENDING environment variable, if:
- a transaction is running, and
- there are in-memory changes to be visible
"hg diff" spawned from external editor process for "hg qrefresh"
shows:
- "changes newly imported into the topmost" before 49148d7868df(*)
- "all changes recorded in the topmost by refreshing" after this patch
(*) 49148d7868df changed steps invoking editor process
Even though backward compatibility may be broken, the latter behavior
looks reasonable, because "hg diff" spawned from the editor process
consistently shows "what changes new revision records" regardless of
invocation context.
In fact, issue4378 itself should be resolved by 800e090e9c64, which
made 'repo.transaction()' write in-memory dirstate changes out
explicitly before starting transaction. It also made "hg qrefresh"
imply 'dirstate.write()' before external editor invocation in call
chain below.
- mq.queue.refresh
- strip.strip
- repair.strip
- localrepository.transaction
- dirstate.write
- localrepository.commit
- invoke external editor
Though, this patch has '(issue4378)' in own summary line to indicate
that issues like issue4378 should be fixed by this.
BTW, this patch adds '-m' option to a 'hg ci --amend' execution in
'test-commit-amend.t', to avoid invoking external editor process.
In this case, "unsure" states may be changed to "clean" according to
timestamp or so on. These changes should be written into pending file,
if external editor invocation is required,
Then, writing dirstate changes out breaks stability of test, because
it shows "transaction abort!/rollback completed" occasionally.
Aborting after editor process invocation while commands below may
cause similar instability of tests, too (AFAIK, there is no more such
one, at this revision)
- commit --amend
- without --message/--logfile
- import
- without --message/--logfile,
- without --no-commit,
- without --bypass,
- one of below, and
- patch has no description text, or
- with --edit
- aborting at the 1st patch, which adds or removes file(s)
- if it only changes existing files, status is checked only for
changed files by 'scmutil.matchfiles()', and transition from
"unsure" to "normal" in dirstate doesn't occur (= dirstate
isn't changed, and written out)
- aborting at the 2nd or later patch implies other pending
changes (e.g. changelog), and always causes showing
"transaction abort!/rollback completed"