revlog: support importing censored file revision tombstones This change allows a revision log to not fail integrity checks when applying a changegroup delta (eg from a bundle) results in a censored file tombstone. The tombstone is inserted as-is, so future integrity verification will observe the tombstone. Deltas based on the tombstone will also remain correct. The new code path is encountered for *exactly* the cases where _addrevision is importing a tombstone from a changegroup. When committing a file containing the "magic" tombstone text, the "text" parameter will be non-empty and the checkhash call is not executed (and when committing, the node will be computed to match the "magic" tombstone text).

verify: report censored nodes if configured policy is abort

context: handle censored data in an on-disk file context based on config Two possible behaviors are defined for handling censored data: abort, and ignore. When we ignore censored data we return an empty file to callers requesting the file data.

manifest: add fastdelta method to manifestdict This is another step closer to alternate manifest implementations that can offer different hashing algorithms.

manifest: move _search to module level and rename to _msearch The rename is intended to provide a slight hint that it is manifest-specific.

manifest: move manifestdict-to-text encoding to manifest class A future patch will introduce a new format, with a new class.

localrepo: access status fields by name rather than index

subrepo: use separate instances of empty lists in status We do modify the lists that make up the status in several places, so it seems risky to use the same instance of a list for several different status types. Use a separate empty list for each type instead.

summary: make status code more readable In commands.summary(), we currently zip a list of labels with a list of statuses. This means the order of the status list has to match the list of the labels, which in turn means the status elements have to be inserted into specific places in the list. Let's instead group the labels and status data we want to display in a single list of pairs.

strip: make checklocalchanges() return full status tuple By making checklocalchanges() return the full instance of the status class instead of just the first 4 elements of it, we can take advantage of the field names and not require the caller to remember the element indices.

fileset: access status fields by name rather than index

histedit: access status fields by name rather than index

shelve: access status fields by name rather than index

record: access status fields by name rather than index It is safe to pass the full status to patch.diff() since it does its own slicing.

purge: access status fields by name rather than index

largefiles: access status fields by name rather than index

keyword: access status fields by name rather than index

hgcia: access status fields by name rather than index

context: store status class instead of plain tuple in self._status This improves readability a bit by allowing us to refer to statuses by name rather than index.

status: update and move documentation of status types to status class The various status types are currently documented on the dirstate.status() method. Now that we have a class for the status types, it makese sense to document the status types there instead. Only leave the bits related to lookup/unsure in the status() method documentation.

status: update various other methods to return new class

status: create class for status lists Callers of various status() methods (on dirstate, context, repo) get a tuple of 7 elements, where each element is a list of files. This results in lots of uses of indexes where names would be much more readable. For example, "status.ignored" seems clearer than "status[4]" [1]. So, let's introduce a simple named tuple containing the 7 status fields: modified, added, removed, deleted, unknown, ignored, clean. This patch introduces the class and updates the status methods to return instances of it. Later patches will update the callers. [1] Did you even notice that it should have been "status[5]"? (tweaked by mpm to introduce the class in scmutil and only change one user)

lfutil: avoid creating unnecessary copy of status tuple In lfdirstatestatus(), the status tuple gets deconstructed, the lists get updated, and then an identical status tuple gets created and returned. Change it so we simply return the original tuple.

dirstate: separate 'lookup' status field from others The status tuple returned from dirstate.status() has an additional field compared to the other status tuples: lookup/unsure. This field is just an optimization and not something most callers care about (they want the resolved value of 'modified' or 'clean'). To prepare for a single future status type, let's separate out the 'lookup' field from the rest by having dirstate.status() return a pair: (lookup, status).

commit: update file nodeid and flags in the same place Now that we have a separate variable for the original 'm1' manifest, we can safely update the nodeid of the file in the new manifest in the same place as we update the flags.

commit: use separate variable for p1 manifest and new manifest In localrepo.commitctx(), p1's manifest is copied and used as the basis for the manifest that is about to be committed. The way the copy is updated makes it safe to use it where the original p1's manifest is wanted. For readability, though, a separate variable for each purpose would be clearer. Make it so.

commit: remove dead initialization of 'lock' The 'lock' variable is initialized to None, but before it's ever read, it's assigned again.

commit: reduce scope of 'removed' variable The variable is closely related to 'added' and 'changed', so it makes sense to have it declared next to them.

rebase: fix some weird mixed-case naming

rebase: move duplicatecopies next to merge This is preparation for removing open-coded rebase/graft operations. As a side-effect, this exposes proper renames in the working copy when there are conflicts, which shows up in test-shelve.t.

histedit: use merge.graft

graft: use merge.graft

merge: add merge.graft helper This will help unify all the open-coded graft/rebase operations.

duplicatecopies: move from cmdutil to copies This is in preparation for moving its primary caller into merge.py, which would be a layering violation in the current location.

histedit: fix indent The duplicatecopies call should be part of the rebase block.

graft: move rebase cleanup code next to actual rebase This is prep for refactoring the rebase logic.

shelve: add a bundlerepo method

dirstate: merge falls through to otherparent This lets us more correctly fix the state when we use setparents, as demonstrated in the change in test-graft.t.

dirstate: use 'm' state in otherparent to reduce ambiguity In rebase-like operations where we abandon the second parent, we can correctly fix up the state in setparents.

dirstate: properly clean-up some more merge state on setparents

phases: move root phase assignment to it's own function This moves the initial root phase assignment to it's own function. Future patches which make phase calculations lazy will use this function to pre-fill certain phases which can be deduced from the roots.

phases: add invalidate function Phase cache invalidation was spread all over the place. Let's add a function to unify it. Later more will be added to this function.

phases: change phase command change detection A future patch is going to make phase computation lazy, so the phase command can no longer read and diff the entire phase list directly. This changes the phase command to build it's own list for diff purposes.

spanset: remove `.set()` definition All my friends are dead.

generatorset: remove `.set()` definition All my friends are dead.

addset: remove `.set()` definition All my friends are dead.

filteredset: remove `.set()` definition All my friends are dead.

baseset: remove `set()` definition All my friends are dead.

abstractsmartset: remove `set()` method definition Now that all usages have been removed, we can drop this not so useful part of the API. We can note that the name was wrong all along...

match: check if an object is a baseset using `isascending` instead of `set` The `set()` method is going away.

getset: check if an object is a baseset using `isascending` instead of `set` The `set()` method is going away.

fullreposet: detect smartset using "isascending" instead of "set" The `.set()` function is going away.

fullreposet: drop custom sets but not smartsets detection All custom classes use by revsets are smartsets now. We drop the special-casing.

addset: drop `.set()` usage during iteration We can use the containment check directly.

baseset: access `_set` directly for containment check The `.set()` method is going away.

baseset: make `_set` a property cache This will remove the need for `baseset.set()`.

graphnode: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-_hexlist: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-_intlist: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-_list: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-roots: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

histedit: stabilise the order nodes that are stripped The `nodes` object is a set. We sort it to get stable order. This is going to prevent revsets from getting confused when removing a `.set()` call in `roots`.

revset-origin: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-last: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-limit: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-destination: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-children: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-branch: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-rangeset: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset-only: remove usage of `set()` All smartset classes have fast lookup, so this function will be removed soon.

revset: cache most conditions used in `filter` Except when stated otherwise, the condition used in `smartset.filter` will be cached. A new argument has been introduced to disable that behavior. We use it for filters created from `and` and `sub` operations. This gives massive performance boosts for revsets with expensive conditions. revset: branch(stable) or branch(default) before) wall 4.329070 comb 4.320000 user 4.310000 sys 0.010000 (best of 3) after) wall 2.356451 comb 2.360000 user 2.330000 sys 0.030000 (best of 4) revset: author(mpm) or author(lmoscovicz) before) wall 4.434719 comb 4.440000 user 4.440000 sys 0.000000 (best of 3) after) wall 2.321720 comb 2.320000 user 2.320000 sys 0.000000 (best of 4)

baseset: empty or one-element sets are ascending and descending The empty set is full of interesting properties. In the ordering case, the one element set is too.

filteredset: drop explicit order management Now that all low-level smartset classes have proper ordering and fast iteration management, we can just rely on the subset in filteredset.

revset: restore order of `or` operation as in Mercurial 2.9 Lazy revset broke the ordering of the `or` revset. We now stop assuming that two ascending revset are combine into an ascending one. Behavior in 3.0: 3:4 or 2:5 == [2, 3, 4, 5] Behavior in 2.9: 3:4 or 2:5 == [3, 4, 2, 5] We are adding a test for it. For unclear reason, the performance `or` revset with expensive filter are getting even worse than they used to be. This is probably caused by extra uncached containment check or iteration. revset #9: author(lmoscovicz) or author(mpm) before) wall 3.487583 comb 3.490000 user 3.490000 sys 0.000000 (best of 3) after) wall 4.481486 comb 4.480000 user 4.470000 sys 0.010000 (best of 3) revset #10: author(mpm) or author(lmoscovicz) before) wall 3.164839 comb 3.170000 user 3.160000 sys 0.010000 (best of 3) after) wall 4.574965 comb 4.570000 user 4.570000 sys 0.000000 (best of 3)

revset-_descendant: rework the whole sorting and combining logic We use the & operator to combine with subset (since this is more likely to be optimised than filter) and we enforce the sorting of the result. Without this enforced sorting, we may result in a different iteration order than the set _descendent was computed from. This reverts a bad `test-glog.t` change from 69402eb72115. Another side effect is that `test-mq.t` shows `qparent::` including `-1` if `qparent is -1`. This sound like a positive change. This has good and bad impacts on the benchmarks, here is a good ones: revset: 0:: before) wall 0.045489 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) after) wall 0.034330 comb 0.030000 user 0.030000 sys 0.000000 (best of 100) revset: roots((0::) - (0::tip)) before) wall 0.134090 comb 0.140000 user 0.140000 sys 0.000000 (best of 63) after) wall 0.128346 comb 0.130000 user 0.130000 sys 0.000000 (best of 69) revset: ::p1(p1(tip)):: before) wall 0.143892 comb 0.140000 user 0.140000 sys 0.000000 (best of 55) after) wall 0.124502 comb 0.130000 user 0.130000 sys 0.000000 (best of 65) revset: roots((0:tip)::) before) wall 0.204966 comb 0.200000 user 0.200000 sys 0.000000 (best of 43) after) wall 0.184455 comb 0.180000 user 0.180000 sys 0.000000 (best of 47) Here is a bad one: revset: (20000::) - (20000) before) wall 0.009592 comb 0.010000 user 0.010000 sys 0.000000 (best of 222) after) wall 0.029837 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)

addset: do lazy sorting The previous implementation was consuming the whole revset when asked for any sort. The addset class is now doing lazy sorting like all other smarset classes. This has no significant impact in the benchmark as-is. But this is important to later change.

test-import.t: use proper revset order This test, written after 3.0, is relying on addset being enforced ascending if both side are ascending. We are about to restore the ordering to 2.9 behavior (elements are ordered in the order they are specified). We fix the test before fixing the order.

baseset: drop custom __sub__ method This add method is enforcing non-laziness, disabling multiple optimisations. Benchmarks do not spot any significant difference but real usecase may. This will also be important for further improvements to addset later in this series.

baseset: drop custom __and__ method This add method is enforcing non-laziness, disabling multiple optimisations. Benchmarks do not spot any significant regression but real usecase may. This even gives some speedup in some cases: revset #15: min(0::) before) wall 0.001247 comb 0.000000 user 0.000000 sys 0.000000 (best of 1814) after) wall 0.000942 comb 0.000000 user 0.000000 sys 0.000000 (best of 2367) This will also be important for further improvement to addset later in this series.

baseset: drop custom __add__ method This add method is enforcing non-laziness, disabling multiple optimisations. Benchmarks do not spot any significant differences but real usecase may. This will also be important for further improvements to addset later in this series.

obsolete: use format version 1 as the default for obsstore

test-obsolete: remove subminute timezone in test Obsmarker format "1" does not supports sub minute timezone. So we change the test to something slightly more sensible.

obsolete: add a "format.obsstore-version" config option This option controls what version of the binary format to use when creating a new obsstore file. Default is still the old format. No safeguards are currently placed around the option value, but no clueless users are in danger of harm since it is undocumented.

obsolete: introduce a new binary encoding for obsmarkers (version 1) This new encoding explicitly stores the date and parents allowing a significantly faster marker decoding. See inline documentation for details. This format is not yet used to store format on disk. But it will be used in bundle2 exchange if both side support it. Support for on-disk format is coming in other changesets.

obsstore: add a flag for sha256 hashes We add flag to inform that the marker is using sha256 hashes. As format 0 is not able to handle sha256 hashes (32 bytes long), we plain crash if we even attempt to encode a sha256 with it.

obsolete: use uint## in the format documention This is shorter and kind of more readable for people who care about binary format.

obsolete: gather _fm0 meta encoding with other _fm0 code

obsolete: _rename decodemeta to _fm0decodemeta This will be format zero specific.

obsolete: _rename encodemeta to _fm0encodemeta This will be format zero specific.

obsolete: store metadata as a tuple of (key, value) pairs (API) Different formats will encode metadata in different ways. So we cannot keep the binary blob in the object anymore. We use a tuple to ensure it is immutable and hashable.

merge with stable

templater: fix ifcontains when list is a string (issue4399)

shelve: don't delete "." when rebase is a no-op (issue4398) When unshelving and facing a conflict, if we resolve all conflicts in favour of the committed changes instead of the shelved changes, then the ensuing implicit rebase is a no-op. That is, there is nothing to rebase. In this case, there are no extra intermediate shelve commits to strip either. Prior to this change, the commit being unshelved to would be marked for destruction in a rather catastrophic way. The relevant part of the test case failed as follows: $ hg unshelve -c unshelve of 'default' complete $ hg diff warning: ignoring unknown working parent 33f7f61e6c5e! diff --git a/a/a b/a/a new file mode 100644 --- /dev/null b/a/a @@ -0,0 1,3 @@ a c x $ hg status warning: ignoring unknown working parent 33f7f61e6c5e! M a/a ? a/a.orig ? foo/foo $ hg summary warning: ignoring unknown working parent 33f7f61e6c5e! parent: -1:000000000000 (no revision checked out) branch: default commit: 1 modified, 2 unknown (new branch head) update: 4 new changesets (update) With this change, this test case now passes.

merge: make error message consistent with other commands If a merge is attempted when another merge is already ongoing, we give the message "outstanding uncommitted merges". Many other commands (such as backout, rebase, histedit) give the same message in singular form. Since the singular form also seems to make more sense, let's use that for 'hg merge' as well.

test-run-tests: add a test for detection of failure to start a server This also highlights a bug: right now we print "2 failed" but we only ran one test.

run-tests: more accurate/helpful message than "diff generation failed" Diff generation didn't really fail, it recognized that an hg serve server has failed to start, and thus skipped the diff generation intentionally. The most common reason for a server to fail to start is that the port was already in use, so output HGPORT as well, to help finding it (since pgrep -f 'hg serve' is not sufficient, if the command line is something like 'hg -R main serve')

run-tests: handle --jobs and --first gracefully Without this change, --first causes currently-running tests to explode in violent and surprising ways when their temporary directory gets cleaned up. Now we just suppress failure messages from non-first failures when running in --first mode.

config: use the same hgrc for a cloned repo as for an uninitted repo This just copies the same local sample hgrc, except it sets the default path to the repo it was cloned from. This is cut-and-paste from the local sample hgrc, but I think it's acceptable, since the two pieces of code are right next to each other and they're small. There is danger of them going out of synch, but it would complicate the code too much to get rid of this C&P. I also add ui as an import to hg.py, but with demandimport, this should not be a noticeable performance hit.

config: give a more detailed sample repo config Some examples of the typical configurations that one might want to do in an .hg/hgrc file. This includes a default-push that happens to point to the same location as my-fork. I insist on the myfork terminology for a server-side clone. Bitbucket, Github, and others have widely popularised this meaning of "fork". This also includes a gentle nudge to use a repo-specific username, which is something that people might not instinctively realise is an option.

smartset: drop infamous ascending, descending All your friends are dead.

fullreposet: use `isascending` instead of `ascending` to recognise smartsets `ascending` is going to be removed.

fullreposet: use `sort` to enforce the order The `ascending` and `descending` methods are useless.

revancestors: replace `descending` with `sort(reverse=False)`

_descendants: replace `ascending()` with `sort()`

_descendants: directly use smartset As `addset` objects are proper smartset objects, we do not need to make any transformation of the result.

baseset: explicitly track order of the baseset A baseset starts without an explicit order. But as soon as a sort is requested, we simply register that the baseset has an order and use the ordered version of the list to behave accordingly. We will want to properly record the order at creation time in the future. This would unlock more optimisation and avoid some sorting.

baseset: fix isascending and isdescending We now have sufficient information to return the proper value there.

baseset: prepare lazy ordering in __iter__ We'll explicitly track the order of the baseset to take advantage of the ascending and descending lists during iteration.

baseset: implement a fastasc and fastdesc Baseset contains already-computed revisions. It is considered "cheap" to do operations on an already-computed set. So we add attributes to hold version of the list in ascending and descending order and use them for `fastasc` and `fastdesc`. Having distinct lists is important to provide correct iteration in all cases. Altering a python list will impact an iterator connected to it. eg: not preserving order at iterator creation time >>> l = [0, 1] >>> i = iter(l) >>> l.reverse() >>> list(i) [1, 0] eg: corrupting in progress iteration >>> l = [0, 1] >>> i = iter(l) >>> i.next() 0 >>> l.reverse() >>> i.next() 0

baseset: stop inheriting from built-in list class The baseset is doing more and more smartset magic and using its list-like property less and less. So we store the list of revisions in an explicit attribute and stop inheriting. This requires reimplementing some basic methods.

strip: stop calling `remove` on smartset The `remove` method is not part of the smartset specification. We use a plain old list comprehension instead.

rebase: transform the smartset to a list before comparing with a list This is highly suboptimal but smartsets are not comparable to lists yet.

merge.update: use `first` instead of direct indexing This makes it compatible with all smartset classes.

qimport: use `first` and `last` instead of direct indexing This makes it compatible with all smartset classes.

rebase: use `last` instead of direct indexing This makes it compatible with all smartset classes.

mq: use `last` instead of direct indexing This makes it compatible with all smartset classes.

repair: use `first` instead of direct indexing This makes it compatible with all smartset classes.

rangeset: use `first` and `last` instead of direct indexing This makes it compatible with all smarsets classes.

revpair: use `first` and `last` instead of direct indexing This makes it compatible with all smarsets classes.

revsingle: use `last` instead of direct indexing This makes it compatible with all smarsets classes.

revset-limit: use boolean testing instead of `len(revs) < 1` I'm not sure why we wrote it that way. But smartsets have faster/lazier non-zero testing than length computation.

filteredset: implement `first` and `last`

baseset: implement `first` and `last` methods

generatorset: implement first and last methods

addset: implement first and last methods The implementation is non-lazy for now. One may want to make it more lazy in the future.

spanset: implement `first` and `last` methods

smartset: add first and last methods In multiple places in the code, we use `someset[0]` or `someset[-1]`. This works only because the `someset` is usually a baseset. For the same reason we introduce a `first` and `last` methods to be implemented for all smartset classes.

getgraphlogrevs: remove user of baseset.append A `baseset` has multiple cached results and will get even more in the future. Making it an object "populated once" like the other smartsets makes it both safer and simpler. The append method will be removed at some point.

getlogrevs: remove user of baseset.append A `baseset` has multiple cached results and will get even more in the future. Making it an object "populated once" like the other smartsets makes it both safer and simpler. The append method will be removed at some point.

revset-last: remove user of baseset.append A `baseset` has multiple cached results and will get even more in the future. Making it an object "populated once" like the other smartsets makes it both safer and simpler. The append method will be removed at some point.

revset-limit: remove user of baseset.append A `baseset` has multiple cached results and will get even more in the future. Making it an object "populated once" like the other smartsets makes it both safer and simpler. The append method will be removed at some point.

mq: use `revs.sort()` to ensure the set is ascending Sorting is super-cheap with the new smartset class, so we can use it to enforce the order. Otherwise all smartset classes would have to allow direct indexing.

baseset: use default value instead of [] when possible For pure cleanup purposes, we replace all the occurences of `baseset([])` with `baseset()`.

generatorset: implement isascending and isdescending

generatorset: explicitly track iteration order The expected iteration order may be different than the fast iteration order (eg: ancestors(42) is expected to be iterated upward but is fast/lazy to compute downward. So we explicitly track the iteration order and enforce it if the manual iteration is requested. Default expected iteration order of a generator set is ascending because I'm not aware of any descending revset that need a generatorset. The first to find such descending revset will have the pleasure to make this configurable.

addset: drop caching through generatorset The utility of this cache is debatable (no visible benchmark impact) and using generatorset for such purpose makes the code complicated. We drop it for now. Someone can reintroduce a smart version of it in the future if it is detected to be relevant.

generatorset: get list-based fast iterations after the generator is consumed When all revisions are known, we shortcut most of the class logic to use list iteration instead. The cost of the sort is expected to be non-significant. The list creation and sorting could be done lazily in the future. We have to copy the list to not break existing iterator created before we finished consuming the generator.

generatorset: move iteration code into _iterator _iterator handles the generator iteration. The `__iter__` method will need changes to handle ordering-related information.

generatorset: stop using a base as the _genlist It does not add anything and makes it more complicated to have a simple baseset implementation.

generatorset: drop the leading underscore in the class name This is a real smart set now.

generatorset: update the docstring now that it is a smartset The documentation was still stating that this class was not a smartset. We drop that part.

addset: drop the leading underscore from the class name This class is now a real smartset.

addset: this is a smartset, update the docstring The documentation was still stating that this class is a not a smartset. We drop that part.

addset: use the ascending argument in _iterordered Fix a bug where fastasc and fastdesc were iterator in the same order as self._ascending.

rebase: add help examples

rebase: attempt to clarify --base

manifest: rearrange add() method and add comments for clarity Omit the check of bool(p1) since it's always true in practice: it will either be nullid or some valid manifest sha, and we know nullid won't ever be in the cache so we can simplify understanding of this code.

manifest: simplify manifest.add() by making args required I verified that changed was never false (it was always a 2-tuple) by adding @@ -220,6 +225,8 @@ class manifest(revlog.revlog): def add(self, map, transaction, link, p1=None, p2=None, changed=None): + if not changed: + assert False, 'changed was %r' % changed # if we're using the cache, make sure it is valid and # parented by the same node we're diffing against if not (changed and p1 and (p1 in self._mancache)): and observing that the test suite still passed. Making all the arguments required should help future readers understand what's going on here.

manifest: move manifest parsing to module-level We'll need this in the sharded manifest hashing routine, and I need to tweak it anyway, so make it module-level now.

revlog: move references to revlog.hash to inside the revlog class This will make it possible for subclasses to have different hashing schemes when appropriate. I anticipate using this in manifests. Note that there's still one client of mercurial.revlog.hash() outside of revlog: mercurial.context.memctx uses it to construct the file entries in an in-memory manifest. I don't think this will be a problem in the immediate future, so I've left it as-is.

revlog: mark nullhash as module-private No other module should ever need this, so mark it with _ so nobody tries to use it.

ui: disable echo back of prompt input if ui is set to non-tty purposely 9ab18a912c44 is nice for test output, but it also affects command-server channel. Command-server client shouldn't receive echo-back message, which makes it harder to parse the output.

dirstate: cache util.normcase while constructing the foldmap This is a small win on OS X. hg perfdirstatefoldmap: before: wall 0.399708 comb 0.410000 user 0.390000 sys 0.020000 (best of 25) after: wall 0.386331 comb 0.390000 user 0.370000 sys 0.020000 (best of 25)

normcase: for darwin, use fast ASCII lower Constructing the foldmap is much faster on OS X now. For a large real-world repo, hg perfdirstatefoldmap: before: wall 0.805278 comb 0.800000 user 0.790000 sys 0.010000 (best of 13) after: wall 0.399708 comb 0.410000 user 0.390000 sys 0.020000 (best of 25) This is a nice boost to 'hg status', especially with the third-party hgwatchman extension enabled (which eliminates stat calls). For the above repo, 'hg status' goes from 1.17 seconds to 0.74.

perf: add a way to measure the perf of constructing the foldmap Constructing the foldmap is a necessary part of operations like 'hg status' on OS X. This command allows us to measure the perf of constructing it.

encoding.lower: use fast ASCII lower This benefits, among other things, the case collision auditor. On a Linux system with a large real-world repo where all filenames are ASCII, hg perfcca: before: wall 0.260157 comb 0.270000 user 0.230000 sys 0.040000 (best of 38) after: wall 0.164616 comb 0.160000 user 0.160000 sys 0.000000 (best of 54)

parsers: add a function to efficiently lowercase ASCII strings We need a way to efficiently lowercase ASCII strings. For example, 'hg status' needs to build up the fold map -- a map from a canonical case (for OS X, lowercase) to the actual case of each file and directory in the dirstate. The current way we do that is to try decoding to ASCII and then calling lower() on the string, labeled 'orig' below: str.decode('ascii') return str.lower() This is pretty inefficient, and it turns out we can do much better. I also tested out a condition-based approach, labeled 'cond' below: (c >= 'A' && c <= 'Z') ? (c + ('a' - 'A')) : c 'cond' turned out to be slower in all cases. A 256-byte lookup table with invalid values for everything past 127 performed similarly, but this was less verbose. On OS X 10.9 with LLVM version 6.0 (clang-600.0.51), the asciilower function was run against two corpuses. Corpus 1 (list of files from real-world repo, > 100k files): orig: wall 0.428567 comb 0.430000 user 0.430000 sys 0.000000 (best of 24) cond: wall 0.077204 comb 0.070000 user 0.070000 sys 0.000000 (best of 100) lookup: wall 0.060714 comb 0.060000 user 0.060000 sys 0.000000 (best of 100) Corpus 2 (mozilla-central, 113k files): orig: wall 0.238406 comb 0.240000 user 0.240000 sys 0.000000 (best of 42) cond: wall 0.040779 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) lookup: wall 0.037623 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) On a Linux server-class machine with GCC 4.4.6 20120305 (Red Hat 4.4.6-4): Corpus 1 (real-world repo, > 100k files): orig: wall 0.260899 comb 0.260000 user 0.260000 sys 0.000000 (best of 38) cond: wall 0.054818 comb 0.060000 user 0.060000 sys 0.000000 (best of 100) lookup: wall 0.048489 comb 0.050000 user 0.050000 sys 0.000000 (best of 100) Corpus 2 (mozilla-central, 113k files): orig: wall 0.153082 comb 0.150000 user 0.150000 sys 0.000000 (best of 65) cond: wall 0.031007 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) lookup: wall 0.028793 comb 0.030000 user 0.030000 sys 0.000000 (best of 100) SSE instructions might help even more, but I didn't experiment with those.

match: remove unnecessary setting of self._always The 'always' class calls its parent constructor with an empty list of patterns, which will result in a matcher that always matches. The parent constructor will set self._always to True in such cases, so there is no need to set it again.

bookmarks: port to generic templater

bookmarks: split ui.write() so that it can be easily ported to formatter api Test output changes because color labels are applied separately.

bookmarks: iterate bookmarks list even if it is known to be empty This clarifies that "no bookmarks set" is displayed in addition to the list of bookmarks. In JSON output, for example, [] should be written if empty, and "no bookmarks set" message should be skipped.

repoview: remove hiddencache verification We have been running hiddencache verification since 3.1.1 and so far not received a bug report concerning it. Therefore we remove the verification code and make the hiddencache authoritive. That way we get the intended speedup.

color: reorganise and sectionify the help text The color docstring was getting long. This splits it up into bite-sized sections and rearranges the order of the paragraphs a little to match these sections.

color: update description of the extension The color extension long ago ceased to work only for the status and qseries commands.

help: show all nested subsections of a section with `hg help foo.section` Used to be that `hg help hgrc.paths` would show "paths" ------- Assigns symbolic names to repositories. The left side is the symbolic name, and the right gives the directory or URL that is the location of the repository. Default paths can be declared by setting the following entries. and stop there. Obviously the result seems better as shown in the attached test.

help: fix output of sections in `hg help foo.somesection` There was a bug in c3c3dd31fe1c. The block that added definitions to getsections should have been an elif, not an if. Otherwise section titles get added twice, since the else clause would always get executed for section titles.

log: show phase in hg log -v with the phase template It seems weird that `hg log -v -T phases` would be *less* verbose than `hg log -T phases`. This cset corrects that oversight.

log: add labels to the phase template This copies the labelled default template and just adds an extra {phase} keyword as necessary.

log: rewrite default template to use labels (issue2866) This is a complete rewrite of the default template to use labels. This seems ultimately useless to me in most cases. The biggest benefit of this patch to me seems to be a fairly complicated example of the templating engine. It was a lot of hard work to figure out the precise acceptable syntax, since it's almost undocumented. Hat tip to Steve Losh's smartlog template, which helped me figure out a lot of the syntax. Hopefully later I can use the present default log template as an example for documenting the templating engine. A test is attached. My goal was to match the --color=debug output, which may differ slightly in newlines from the actual ANSI escape codes output. I consider this an acceptable invisible deviation. There seems to be a considerable slowdown with this rewrite. Before: $ time hg log -T default -r .~100::. > /dev/null real 0m0.882s user 0m0.812s sys 0m0.064s $ time hg log -T default -r .~100::. > /dev/null real 0m0.872s user 0m0.796s sys 0m0.068s $ time hg log -T default -r .~100::. > /dev/null real 0m0.917s user 0m0.836s sys 0m0.076s After: $ time hg log -T default -r .~100::. > /dev/null real 0m1.480s user 0m1.392s sys 0m0.072s $ time hg log -T default -r .~100::. > /dev/null real 0m1.500s user 0m1.400s sys 0m0.088s $ time hg log -T default -r .~100::. > /dev/null real 0m1.462s user 0m1.364s sys 0m0.092s Following the maxim, "make it work, profile, make it faster, in that order", I deem this slowdown acceptable for now. I suspect but have not confirmed that a big slowdown comes from calling keywords twice in the file templates, once to test the existence of output and again to actually list the output. If so, a simple speedup might be to improve the templating engine to cache keywords when called more than once on the same revision. TODO: I found a bug while working on this. The following stack traces: hg log -r . -T '{ifcontains(phase, "secret public", "lol", "omg")}\n'

log: do not hide the public phase in debug mode (BC) When 51fc43253a52 introduced phases to the `hg log --debug` output, it also disabled outputting public phase. At the same time, it always shows phases in the default template, `hg log --debug -T default`. Those two should produce the same output, but they don't. I think it makes a lot more sense to always show all phases. There's already loss of backwards compatibility in this case when using a newer hg on an old hg repo, since draft commits will show up in the output of `hg log --debug`. Finally, I just don't think that any sort of information should be hidden with --debug. This flag should be about showing as much information as possible.

templater: set the correct phase for parents Akin to f6371cc62d2a which did this for `hg log`, the following sets the correct phase for the {phase} keyword when the context is a parent of the current cset. This allows templates such as the following to be defined, parent = '{label("log.parent changeset.{phase}", "parent: {rev}:{node|formatnode}")}\n' which when called on a parent (e.g. with the `parents` template keyword), will produce the correct phase.

color: omit debug label output on empty strings This is most noticeable when using custom templates. Before this patch, a template like {label("foo.bar", baz)} would emit [foo.bar|] whenever baz was empty. This cset simply omits all output when baz is empty.

tests: make hghave list features alphabetically

revset: remove the now unused _descgeneratorset class

revset: use _generatorset in _revancestors The _descgeneratorset class is going away.

revset: remove now unused class _ascgeneratorset

revset: use _generatorset directly in _revdescendant _ascgeneratorset is going away.

generatorset: move membership testing on ordered gen to the main class We are phasing out the ordered version of the class to simplify the code.

generatorset: make use of the new mechanism in the subclass Until we remove them, we use the new parameter of _generatorset to make sure the code is run.

generatorset: make it possible to use gen as fastasc or fastdesc We gain a parameter to inform that the generator is ascending or descending. If the generator is ordered, it is also used for the `fastasc` or `fastdesc` version. The _ascgeneratorset and _descgeneratorset class will be removed soon.

baseset: rely on the abstractsmartset implementation for filter

_orderedsetmixin: drop this now unused class All my friends are dead.

spanset: drop _orderedsetmixin inheritance The min/max method are as well provided by abstractsmartset.

orderedlazyset: drop this now unused class All my friends are dead.

_descendant: use filteredset instead of orderedlazyset The orderedlazyset class is going away. Filteredset gives the same service.

addset: use the base implementation for ascending and descending

addset: use base implementation for __filter__

addset: use base implementation for __add__

addset: use base implementation for __sub__

addset: use base implementation for __and__

addset: promote to real smartset Better revset performance are also achieved with less overlay. There is no good reason for addset to not be a smartset. We can replace the `_orderedsetmixin` inheritance since `abstractsmartset` has efficient min and max too.

addset: add a __nonzero__ method This is required to be a full smartset (not sure what was happening before that...)

addset: offer a fastasc and fastdesc methods If the underlying object offers fast iterators, we use them to provide fast iterators too.

addset: split simple and ordered iteration We have two goals here. First, we would like to restore the former iteration order we had in 2.9. Second, we want this logic to be reusable for `fastasc` and `fastdesc` methods.

generatorset: promote to smartset This is not going to be efficient but we need all basic set classes to be smartsets for the other classes to work.

generatorset: implement __nonzero__ This is necessary to become a real smartset.

spanset: use base implementation for __add__

spanset: use base implementation for __sub__

spanset: use base implementation for __and__

spanset: use base implementation for filter

filteredset: use base implementation for filter

filteredset: use base implementation for __add__

filteredset: use base implementation for __sub__

filteredset: use base implementation for __and__

abstractsmartset: add default implementation for __sub__

abstractsmartset: add default implementation for __add__

abstractsmartset: add default implementation for __and__

abstractsmartset: add default implementation for filter

lazyset: rename the class to filteredset All smartsets try to be lazy. The purpose of this class is to apply a filter on another set. So we rename the class (and all its occurences) to `filteredset`.

lazyset: add order awareness to the class Just a bit of extra code makes the lazyset aware of order. This renders orderedlazyset useless. At some point, the `subset` will become responsible for this ordering logic. But we are not there yet because the various objects used as subsets are not good enough.

lazyset: remove min/max This is now handled by abstractsmartset.

baseset: remove min/max methods This is now handled by the base class.

abstractsmartset: add a default implementation for min and max This default implementation takes advantage of the fast iterator if available.

lazyset: drop now useless ascending/descending definition

lazyset: inherit the fastasc and fastdesc method from subset When the filtered subset has such methods, we can use them. It is implemented as properties to be able to quickly return None if no corresponding fastasc exists on the subset.

lazyset: split the iteration logic from the condition filtering logic So that the filter can be reused by `fastasc` or `fastdesc`.

spanset: do a single range check in __contains__ Now that `start <= end` is always true, we can simplify this function.

spanset: enforce the order lazily to gain `fastasc` and `fastdesc` methods Instead of having the direction of iteration enforced through the ordering of `start` and `end` attributes of spanset, we encode the iteration direction in an explicit attribute and always store start < end. The logic for sort and reverse has to be updated. The __iter__ is now based on the newly introduced `fastasc` and `fastdesc` methods. This will allow other code simplifications in the future.

abstractsmartset: document the `fastasc` and `fastdesc` attributes/methods See the in-line documentation for details. (This is the beginning of a massive overhaul of revset).

spanset: remove ascending/descending implementation We can rely on their implementation in abstractsmartset.

baseset: remove ascending/descending redefinition We can rely on the abstractsmartset implementation.

abstractsmartset: default implementation for `ascending` and `descending` These two methods are actually silly aliases for `sort()` and `sort(reverse=True)`. So we get that aliasing at the abstractsmartset level. We will slowly phase out all the custom implementations and eventually remove any mentions of it from the code.

revert: bring back usage of `subset & ps` in `parents` Changeset 95af98616aa7 switched the order of the operand of the "&" computation to work around an issue from repo-wide spanset. The need for a workaround has been alleviated by the introduction of `fullreposet`. So we restore it to normal. The benchmark shows no significant changes as expected. We also revert the bogus test change introduced by 95af98616aa7. The order is actually important.

color: update docstring for debug option 964dd1c491ca updated the format for the --color=debug option. This updates the documentation to match this new format.

color: document the possibility to colourise tabs in diffs This is a no-op change that simply suggests that tabs can be colourised, but defaults to no colour effects for tabs. This complements cset c343557a8442. Like the similar effects for phases from f8e2aebbb24c, we set the default effects to '' instead of None, so that this is a true noop change. Otherwise, the diff.tab effect would override (i.e. neutralise) the effect of the surrounding label.

contrib/synthrepo: walk a repo's directory structure during analysis Augments the analyze command to additionally walk the repo's current directory structure (or of any directory tree), counting how many files appear in which paths. This data is saved in the repo model to be used by synthesize, for creating an initial commit with many files. This change is aimed at developing, testing and measuring scaling improvements when importing/converting a large repository to mercurial.

contrib/synthrepo: generate initial repo contents using directory shape model Augments the synthesize command to use an additional parameter to the analyzed repo model: the number of files in each directory at a given snapshot. Before synthesizing history, an arbitrary number of files will be generated in a distribution matching the analyzed directory structure. Intended for developing, testing and measuring scaling improvements when importing/converting a large repository to Mercurial.

filemerge: switch the default name for internal tools from internal:x to :x

filemerge: introduce :x as short version of internal:x merge tool name "internal:" is so much typing - it is frequently specified on the command line and gives options that are longer than any other I have seen.

branches: include active, closed and current flags in template output

branches: merge white space to format string Diffs of test output should be harmless. A white space character is moved into "log.changeset" color region.

branches: port to generic templater

branches: format rev as integer that is necessary for generic templater

formatter: add general way to switch hex/short functions This seems a bit awkward, but it can avoid duplicates in annotate, tags, branches and bookmarks. I guess fm.hexfunc can eventually be removed (or redesigned) when it gets template backend.

locarepo: remove the `pull` method (API) All the logic of this function is in the `exchange.pull` function for some time. We just stop calling `localrepo.pull` in `command.pull` to have access to more information. Leaving `localrepo.pull` in place will let third-party extensions wrap it but it would never be called by `hg pull` making the wrapping useless. Therefore, the method is removed so that third-party code fail noisily and get properly upgraded.

transplant: use exchange.pull localrepo.pull is going away. See 4d52e6eb98ea for details.

convert-hg: use localrepo.pull localrepo.pull is going away. See 4d52e6eb98ea for details.

fetch: use exchange.pull localrepo.pull is going away. See 4d52e6eb98ea for details.

localrepo: use exchange.pull when cloning localrepo.pull is going away. See 4d52e6eb98ea for details.

subrepo: use exchange.pull localrepo.pull is going away, see 4d52e6eb98ea for details.

commands: directly use exchange.pull localrepo.pull is going away. See explanations in 4d52e6eb98ea.

exchange: have `pull` return the pulloperation object We mimic what was done for `push` for similar reason. We are about to drop `localrepo.pull` (for consistency with dropping `localrepo.push` and we better have an API as extensible as `push` is. Find explanations about localrepo.push removal in 4d52e6eb98ea.

revset: introduce an abstractsmartset class This class documents all methods required by a smartset. This makes it easier for people to respect the API and ensure we fail loudly when something does not. It will later also contain common default implementations for multiple methods, making it easier to have smartset classes with minimal work.

revset: add a `__nonzero__` to baseset We are about to add a base class for `baseset` with an abstract `__nonzero__` method. So we need this method to be explicitly defined to avoid issues. The built-in list object apparently does not have a `__nonzero__` and relies on `__len__` for this purpose?

revset: drop isinstance(baseset) in spanset.__sub__ As baseset now has a fast __contains___ operator, this `baseset.set()` dance is no longer needed. No regressions are visible in the benchmark.

revset: drop isinstance(baseset) in spanset.__and__ As baseset now has a fast __contains___ operator, this `baseset.set()` dance is no longer needed. No regressions are visible in the benchmark.

revset: drop isinstance(baseset) from baseset.__and__ As baseset now has a fast __contains___ operator, this `baseset.set()` dance is no longer needed. No regressions are visible in the benchmark.

revset: use direct access to __contains__ in spanset.__sub__ Using `x.__contains__(r)` instead of `r in x` does not matter for built-in type (set) but have a positive impact for all other classes. This will let us drop some usage of baseset.set() in future patches. This also probably improves some performance.

revset: use a single return statement in matcher function This makes it easy to insert post processing and debug code on the returned value.

obsolete: replace "nb" notation with "num" (for "number") As requested by Matt Mackall.

shelve: remove unused status variables

gpg: use an abort hint and don't mention --force

gpg: move test of force before status call

gpg: drop unnecessary slicing of status array The call to repo.status() does not request status for clean files, so there is no reason to slice it out from the result. Leaving the tuple untouched will simplify a future change.

tag: use an abort hint

tag: only check the status of .hgtags

tag: properly abort if an unknown or ignored .hgtags is present This was a regression introduced by 4faaa0535ea7 in 2008.

summary: remove unused code for listing ignored files The call to repo.status() does not request ignored files to be listed, so remove the code for printing them.

fetch: use cmdutil.bailifchanged() Use the existing method cmdutil.bailifchanged() instead of implementing it again in fetch.py. An effect of this is that the error messages in case of uncommited changes will be different.

ui: fix comment about non-interactive prompts This fixes my earlier in-flight addition to Mads' change.

formatter: convert booleans to json It will be used in branches output.

merge: mute the status message when bid merge kicks in Bid merge is now the default and it is not necessary to tell the user that an experimental feature kicked in. (It could however still be relevant to get a notice that it is one of the rare criss-cross merge situations so the user is warned that the situation is more tricky than usual.)

merge: use bid merge by default (BC) In most cases merges will work exactly as before. The only difference is in criss-cross merge situations where there is multiple ancestors. Instead of picking an more or less arbitrary ancestor, it will consider both ancestors and pick the best bids. Bid merge can be disabled with --config merge.preferancestor='!'.

changectx: skip all invalid merge.preferancestor values A better fix for 17011b36aac7 that will ignore other kinds of "invalid" revisions.

revset: rely on built in iterator when possible in _generatorset.__iter__ Doing manual iteration is expensible. We rely on built in list iteration whenever possible. The other case has to become a closure we cannot have a both yield and return in the same function.

revset: prefetch an attribute in _generatorset.__iter__ Python's attribute lookup are expensible, lets do less of them. This gives us a 7% speedup on this revset iteration (from 0.063403 to 0.059032)

templater: add count template filter, plus tests Previously there was no way of telling how much children or bookmarks or tags a certain changeset has in a template. It was possible to tell if a changeset has either 0 or not 0 bookmarks, but not to tell if it has 1 or 2 of them, for example. This filter, simply named count, makes it possible to count the number of items in a list or the length of a string (or, anything that python's len can count). E.g.: {children|count}, {bookmarks|count}, {file_adds|count}. Testing the filter on node hash and shortened node hash is chosen because they both have defined length. As for lists of strings - children, tags and file_adds are used, because they provide some variety and also prove that what's counted is the number of string items in the list, and not the list stringified (they are lists of non-empty, multi-character strings). Additionally, revset template function is used for testing the filter, since the combination is very flexible and will possibly be used together a lot. (The previous version of this patch had an incorrect email subject and was apparently lost - patchwork says the patch has been accepted, but it's not so. The changes between that and this patch are minimal: now the filter does not disturb the alphabetical order of function definitions and dict keys.)

push: update bookmarks (on server) within a transaction A nice side effect is that bookmarks sent through bundle2 are updated within the same transaction as all other changes.

pull: perform bookmark updates in the transaction

bookmark: add a `bmstore.recordupdate` to plug bookmarks into the transaction Instead of manually writing bookmarks when they are updated, we can just record this update to the transaction and rely on it to update the on-disk file.

bookmarks: split bookmark serialization and file handling If we want to handle bookmarks in a transaction we need to decouple the file handling and the actual production of the content. This is similar to how we handle phases in transaction.

transaction: allow generating file outside of store We allow a vfs argument to `addfilegenerator`. This allows generating files outside of the store directory like bookmarks. However, this is not really working since we do not have the infrastructure to backup and restore files outside of store. By chance, the bookmark file is already backed up by another mechanism so we can restrict this new feature to bookmarks (which is our only interest here) and proceed.

transaction: work around and document issue with file backup The backup restoration is actually hard-coded for the main file. And this hard-coded list is the only one used when repairing an interrupted transaction from another process. Solving this problem is out of the scope of this series so we document it and work around its implications.

bundle2: remove an explicit packing The final writing of the empty part was done explicitly. We now using proper pack call using symbolic constant. This open simple change in the bundle2 format.

bundle2: split test in two We split the test between the one dedicated to the binary format and the one dedicated to checking the exchange of data using pull and push.

pull: merge bookmark updates and imports We do all the things in one go now, updating existing bookmark, adding new ones, and overwriting the ones explicitly specified for --bookmark. This impacts the tests by removing some duplicated or unnecessary output.

pull: gather explicit bookmark pulls with bookmark updates There is no reason to make them at different times. So we gather them. This is the first step toward merging them.

subrepo: stop pulling bookmark manually Bookmark pulling is now done with all the other pull steps in the `exchange.pull` function.

pull: retrieve bookmarks through bundle2 We can retrieve any pushkey namespace through bundle2. So we also ask for bookmark data and use them to update the local repo.

pull: retrieve bookmarks before obsmarkers Retrieving bookmarks before obsmarkers will avoid turning some changesets hidden right before making them visible again if a bookmark keeps them visible.

pull: move bookmark pulling into its own function This requires adding an attribute on the pulloperation object. The bookmark pulling is protected behind a "todostep" as other steps of pull.

pull: perform the todostep inside functions handling old way of pulling This matches what is done during push.

bookmark: remove now unused `pushtoremote` function Everything is unified in the push discovery now.

push: gather all bookmark decisions together The discovery phases for bookmarks now use the list of explicitly pushed bookmarks to do addition, removal and overwriting. Tests are impacted because this reduces the amount of listkeys calls issued, removes some duplicated messages and improves the accuracy of some messages.

push: prepare the issue of multiple kinds of messages To gather all the bookmark pushing actions together, we need code performing those actions to be ready for them. We need to be able to produce different messages for different actions.

push: set bkresult when pushing bookmarks through bundle2

clone: remove duplicated bookmark pulling Now that all clone methods (copy, pull and push) also transport bookmarks, we can safely drop the manual pulling that was performed during clone.

clone: explicitly push bookmarks when cloning from local to remote We need to explicitly push all local bookmarks when doing the clone from a local repo to a remote one through ssh. This will let us remove the manual export of bookmarks in clone and rely on the official exchange in push and pull instead.

clone: copy `.hg/bookmarks` during copy clone Now that the standard pull function includes bookmarks, we need to ensure that a copy clone also copies the bookmarks. This will let us drop the manual bookmark pulling done independently during a clone.

pull: move bookmark movements inside the `exchange.pull` There is no reason for bookmarks to get a special treatment. As a first step we move the code as is in the `exchange.pull` function. Integration with the rest of the flow will come later. Adding bookmarks to pull means that most clone paths are now pulling bookmarks through pull. We ensure that bookmark-update messages are properly suppressed in that case. In test-pull-http.t the 'requesting all changes' message disappear because we now get the authentication error on the `listkeys`command before such message is printed.

bookmarks: allow `updatefromremote` to be quiet This will be useful to use the function during clone (clone is not printing any bookmark data right now)

util: fix sorteddict.pop When using `.pop` on such object the list was not cleared of the popped key, leading to crash.

localrepo: pass arbitrary kwargs from `repo.pull` to `exchange.pull` We'll add bookmark-related arguments to `repo.pull` so we need to widen the signature of `repo.pull`. We should probably kill `repo.pull` now that `repo.push` is dead but this is outside the scope of this series.

pull: use `other.url()` as the source of a bookmark pull We want to move the bookmarks movement into `exchange.pull`, for this purpose we need to stop relying on variables from `command.pull`.

setup: set mode 644 or 755 on installed files

branches: reduce nesting in for loop

i18n: use datapath for i18n like for templates and help To avoid circular module dependencies we initialize i18n from util when datapath is found.

help: don't search randomly for help data - trust util.datapath The search was introduced in c904e76e3834 without a convincing explanation why it should be necessary ... except for consistency with templater handling. Now, just keep it simple.

templater: don't search randomly for templates - trust util.datapath The search was introduced in 2653740d8118. It might have been necessary back then when using __file__ directly and frozen-ness wasn't considered. Now we should know exactly where the templates can be found.

templater: inline global 'path' list in templatepaths

templater: introduce templatepaths for getting paths searched for templates Avoid function with different return types depending on parameters.

util: introduce datapath for getting the location of supporting data files templates, help and locale data is normally stored as sub folders in the directory containing the source of the mercurial module. In a frozen build they live as sub folders next to 'hg.exe' and 'library.zip'. These different kind of data were handled in different ways. Unify that by introducing util.datapath. The value is computed from the environment and is always used, so we just calculate the value on module load.

util: move _hgexecutable a few lines, closer to where it is used

hgk: define bookmark colour explicitly, as Tk 8.6 has changed their meaning

largefiles: remove 'forget' list that's always empty

bookmarks: fix divergent bookmark path normalization

exchange: remove a broken i18n abuse

bookmark: make the search for divergent names more robust We translate the "url we update from" and "the url in the config" into their canonical representation. This is useful for urls that have multiple equivalent forms: /foo/bar/ == file:/foo/bar/ == file:///foo/bar eg: hg pull --config path.bar=/foo/bar/ file:/foo/bar

pull: gather all bookmark-pulling code together Pulling bookmarks is done in two rounds. First we do a simple update, then we use the content of the `bookmark` argument to possibly overwrite some bookmark with their remote location. The second step was not done right after the first one for some obscure reason. We now perform them one after the other.

push: perform bookmark export in the push function This part is responsible for adding new bookmarks on the remote. Before that, it was done on its own in `commands.push`. The export is still not integrated with the rest of the push process, but at least it now dwells in the right function.

push: add `pushoperation.bkresult` This attribute will be used to carry the result of the bookmark push (when the `push` function will effectively be doing any bookmarks pushes)

push: pass list of bookmark to `exchange.push` Currently stored but not unused. This parameter will control bookmarks explicitly pushed (added to the server if missing).

exchange: import bookmarks as bookmod Using the original names makes it difficult to use `bookmarks` as a variable name.

push: sanitize handling of bookmark push return value Mixing return and assignment does not make sense, let's unify this.

locarepo: remove the `push` method (API) All the logic of this function is in the exchange.push function for some time. We just stop calling `localrepo.push` in `command.push` to have access to more information. Leaving `localrepo.push` in place will let third-party extensions wrap it but it would never be called by `hg push` making the wrapping useless. Therefore, the method is removed so that third-party code fail noisily and get properly upgraded.

push: `exchange.push` instead of `localrepo.push` The latter is going away.

clone: use `exchange.push` instead of `localrepo.push` The latter is going away.

push: use `exchange.push` in `commands.push` To gain access to all results from the push, we need to have access to the `pushoperation` object. We call `exchange.push` to do so. It is impossible to just change the `localrepo.push` signature because the change may be too subtle to be caught by external extension wrapping `localrepo.push`. This mean we'll have to kill `localrepo.push` because just using `exchange.push` in `commands.py` would silently disable all wrapping around `localrepo.push` by third-party extensions. So we'll remove it in a later changeset to get such extensions to fail noisily.

push: `exchange.push` now returns the `pushoperation` object Returning the pushop object gives access to more information (upcoming bookmark push result for example). `localrepo.push` currently extracts the `cgresult` for callers.

push: rename `pushop.ret` to `pushop.cgresult` We are about to introduce more results-related attributes on pushop (for bookmarks) so we need a more distinctive name. We now use `cgresult` as `pulloperation` does.

obsolete: use version constants in the format mapping

obsolete: use the `version` argument in encodemarkers Recent scientific studies show that assigning a variable have no effect on a unrelated constant. We therefore use the variable where we intended to in the first place.

obsolete: gather all contents related to format version 0 in a single place All contents includes documentation, constants, and functions, so we gather all of those things into a single location. The diff is confusing because it cannot understand what code is semantically moved around in this grand migration.

revert: properly back up added files with local modification These files were previously not backed up because the backup mechanism was not smart enough. This leads to data lose for the user since uncommitted contents were discarded. We now properly move the modified version to <filename>.orig before deleting it. We have to use a small hack to do a different action if "--no-backup" is specified. This is needed because the backup process is actually a move (not a copy) so the file is already missing when we backup. The internet kitten is a bit disapointed about that, but such is life. This patch concludes the "lets refactor revert" phases. We can now open the "Lets find stupid bug with renames and merge" phases. I'm sure that now that the code is clearer we could do it in another simpler way, but I consider the current improvement good enough for now.

revert: track added files with local modifications Those files need to be backed up but are currently not. We compute the set of them to be able to use a different backup strategy in the next changeset.

revert: distinguish between "check" and "backup" strategy "check" behaves as backup did before. We check if the current file differs from destination and we create a backup if it does. This is used for untracked files that will be overwritten by formerly-deleted files. We have to do the manual check since no status output can provide the content comparison. "backup" is now doing unconditional backup. This can be used for files seen as modified compared to both the target and the working directory. In such a case, we know that the file differs from target without actually comparing any content. This new "backup" strategy will be especially useful in the case of files added between the target and the working directory -parent- with additional modifications in the working directory -itself-. In that case we know we need to back it up, but we cannot run the content check as the files does not exists in target.

revert: small refactoring in the way backup value are handled The current backup value may have two different values: 1. Do not try to do backup 2. Do backup if applicable We are about to move to: 1. Do not try to do backup 2. Do backup if applicable 3. Do backup in all cases So we change the current values to make room for the new one.

merge with stable

Added signature for changeset f768c888aaa6

Added tag 3.1.2 for changeset f768c888aaa6

parsers: fix Py2.4 argument parsing issue Since fa53d66b45a8, we were getting this strange message with Py2.4: TypeError: argument 1 must be impossible<bad format char>, not int ..because we were using the 'n' type specifier introduced in 2.5. It turns out that offset is actually a revision number index, which ought to be an int anyway. So we store it in an int, use the 'i' specifier, rely on Py_ParseTuple for range checking, and rename it to avoid type confusion.

merge with i18n

diff: document the nobinary option Since 3fbef7ac26f0, we have a diff.nobinary option. This is handy, but the only way I found out about it was by looking at the release notes for 3.1, which is not something I normally do.

i18n-pt_BR: minor fixes and rewording on histedit help text

i18n-pt_BR: synchronized with c712238c4f9b

merge with i18n

i18n-pt_BR: synchronized with 802dffd62de5

filelog: censored files compare against empty data, have 0 size To support "status" operations against working directories that are the children of censored revisions, filelog must define "cmp" and "size" for censored content.

filelog: raise CensoredNodeError when hash checks fail with censor metadata With this change, when a revlog revision hash does not match its content, and the content is empty with a special metadata key, the integrity failure is assumed to be intentionally caused to remove sensitive content from repository history. To allow different Mercurial functionality to handle this scenario differently a more specific exception is raised than "ordinary" hash failures. Alternatives to this approach include, but are not limited to: - Calling a hook when hashes mismatch to allow arbitrary tombstone validation. Cons: Irresponsibly easy to disable integrity checking altogether. - Returning empty revision data eagerly instead of raising, masking the error. Cons: Push/pull won't roundtrip the tombstone, so client repos are unusable. - Doing nothing differently at this layer. Callers must do their own detection of tombstoned data if they want to handle some hash checks and not others. - Impacts dozens of callsites, many of which don't have the revision data - Would probably be missing one or two callsites at any given time - Currently we throw a RevlogError, as do 12 other places in revlog.py. Callers would need to parse the exception message and/or ensure RevlogError is not thrown from any other part of their call tree.

error: add CensoredNodeError, will be thrown when content deliberately erased This change introduces the error plus a corresponding catch in dispatch, to provide localized error messages. The verb "censor" is used in this commit and all following to refer to erasing the content of a revlog revision (filelog, for now) without recalculating node IDs, leaving that revision invalid. Further work must be done to safely share such revision data with compliant clients. I find the analogy to censorship straightforward; for less politically charged options, consider "erase", "excise", "expunge", or "blackhole".

files: cache repo.dirstate For a large repo, 'hg files' goes from 2.27 seconds to 1.92.

files: only check for removed, not unknown or missing ctx.matches() doesn't return unknown files, and repo.dirstate[f] never returns '!' for an answer.

files: use ctx.matches instead of ctx.walk ctx.matches() is an optimized form of ctx.walk() when we don't care about the state of files on disk. For a large repo, 'hg files > /dev/null' drops from 3.7 seconds to 2.3.

files: actually filter out removed files 'hg files' makes an attempt to filter out removed files, but that doesn't work because repo.dirstate[f] returns lowercase 'r', not uppercase.

subrepo: remove superfluous newline from subrepo prompt

ui: show prompt choice if input is not a tty but is forced to be interactive The tests often set ui.interactive to control normally interactive prompts from stdin. That gave an output where it was non-obvious what prompts got which which response, and the output lacked the newline users would see after input. Instead, if the input not is a tty, write the selection and a newline.

merge with stable

help: basic support for showing only specified topic sections For instance, 'hg help config.files' will show only the Files section.

help: use OS containers for config file lists

help: support OS-specific help sections Containers of the form: .. container:: verbose.<os> are shown by default on the given OS (or with -v).

minirst: allow multiple container types for prune If we have a container of type x.y, then we can preserve it by keeping x or y.

config: don't read the same config file twice In some cases some config files would be read twice and shown twice in showconfig --debug.

templater: fix precedence of --style and --template options Since e3eb480a9391, --template option is ignored if --style is specified, which is wrong according to the doc of show_changeset(): Display format will be the first non-empty hit of: 1. option 'template' 2. option 'style' ...

shelve: avoid writing file that is never read from The contents of the .files file has not been used since 1d7a36ff2615 (shelve: use rebase instead of merge (issue4068), 2013-10-23), so stop writing it. Where we currently use the presence of the file as a check for a valid shelve name, switch to checking for the .patch file.

hgk: don't break on repositories with obsolete changesets Check the existence of a changeset before adding it to the list returned by debug-rev-list command.

test: protect the run-tests.py --json test behind an hghave rule We add a rules to detect availability of a json module and skip if json is not available.

merge with stable

hgweb: refresh hgweb.repo on phase change (issue4061) Make hgweb.refresh() also look at phaseroots file (in addition to 00changelog.i file) and reload the repo when os.stat returns different mtime or size than cached, signifying the file was modified. This way if user changes phase of a changeset (secret <-> draft), there's no need to restart hg serve to see the change.

help: fix typo in log examples

ssl: on OS X, use a dummy cert to trick Python/OpenSSL to use system CA certs This will give PKI-secure behaviour out of the box, without any configuration. Setting web.cacerts to any value or empty will disable this trick. This dummy cert trick only works on OS X 10.6+, but 10.5 had Python 2.5 which didn't have certificate validation at all.

ssl: refactor sslkwargs - move things around a bit, preparing for next change

revert: special case 'hg revert --all' On large repos, hg revert --all can take over 13 seconds. This is mainly due to it walking the tree three times: once to find the list of files in the dirstate, once to find the list of files in the target, and once to compute the status from the dirstate to the target. This optimizes the hg revert --all case to only require the final status. This speeds it up to 1.3 seconds or so (with hgwatchman enabled). Further optimizations could be done for the -r NODE and pattern cases, but they are significantly more complex.

test-commandserver: make runcommand message bolder It seems ' runcommand' is difficult to distinguish from command output. '*** runcommand' is slightly better.

test-commandserver: rewrite manual substitution of output by (glob) match

test-commandserver: remove redundant banner output Since test output was inlined, "testing <func>" message should no longer be necessary.

test-commandserver: remove .py test which was ported to .t

test-commandserver: port test functions from .py to .t Though we have to duplicate import statements, .t test is still more handy than .py test which has cryptic .out file. This change allows to skip a part of test by #if conditional, which my next patch series depends on.

test-commandserver: add stub for .t test by copying .out with 2-space indent Currently test-commandserver.t only contains output, so it is skipped by #require false.

test-commandserver: split helper functions to new hgclient module This prepares for porting test-commandserver.py to .t test. Though command-server test needs many Python codes, .t test will be more readable than .py test thanks to inlined output.

heredoctest: do not append extra newline character to continuation line Trailing newline characters are kept in lines.

heredoctest: use the same dict for local/global contexts as in doctest In order to mimic module-level evaluation, globals and locals should be the same object, so doctest does not pass separate locals dict. https://docs.python.org/2.7/reference/simple_stmts.html#exec This fixes NameError in the following example: >>> import foo >>> def bar(): ... foo # must exist in globalvars

cmdserver: drop useless in_ attribute from channeledoutput The previous patch makes sure that in_ == out, so it's no longer needed to keep in_ for __getattr__. Also, it seems strange for channeledoutput to have in_ stream which is actually a writable file object.

cmdserver: get file attributes of 'e'-channel from stdout, not from stderr It seems wrong to get attributes from object different than the underlying file. In the following example, it doesn't make sense to flush stderr after writing to stdout: self.ferr.write(str(a)) if not getattr(self.ferr, 'closed', False): self.ferr.flush()

cmdserver: correct doc of channeledoutput in_ is only used as the source of file attributes.

merge with stable

commands: add debuglocks to report/clear lock state

test-convert-git.t: make copy detection deterministic Since both 'bar' and 'bar-copied' matched 'bar-copied2', Git's copy detection would sometimes result in 'bar' being the source for 'bar-copied2' and sometimes 'bar-copied'. Change bar in a separate commit so that that would no longer be the case.

revsetbenchmark: add a rebase-related revset to the benchmark list

revsetbenchmark: allow comments ('#' prefix) in the revset input

revsetbenchmark: make it clear that revsets may be read from stdin

tags: use full hash for formatter output as in log or annotate commands

tags: change field name of formatter output to be the same as log command Since -T option is not public yet, this won't break backward compatibility.

files: correct topic of formatter

revert: move targetsubs calculation down to its use A future patch will be reorganizing this section of the code into two paths, and targetsubs complicates this by existing in the middle of one path, but not the other. We fix that by moving it 200 lines down, to the only place it's used.

help: document that default hgweb style is called paper (issue4373)

help: update help for hgweb template and style (issue4373)

bundle2: rename functions that have the same name

mq: write headers of new patches using patchheader

mq: refactor patchheader header ordering to match export (BC) The refactoring also gives more robust and extendable handling of other HG headers.

mq: upgrade non-plain patches to HG format when setting parent in patchheader Parent will now always be updated or added when qrefreshing HG patches. Plain patches will not be changed, but patches that neither are plain nor HG will be upgraded to HG patches on first refresh.

mq: make patchheader .plainmode more reliable Instead of having to make extra checks whenever we use .plainmode, let the initial value consider the actual patch header content.

bundle2: remove heads and common arguments to getbundle parts generators

bundle2: separate bundle10 and bundle2 cases in getbundle() The primary goal is to make it easier for extensions to alter how bundle2 parts are laid out. They now can use the getbundle2partsgenerator decorator to add new parts, or directly act on getbundle2partsmapping to wrap existing part functions. Note the 'request for bundle10 must include changegroup' error was kept under the same conditions as before, although the logic changes don't make it obvious.

bundle2: pass b2caps down to functions adding bundle2 parts for getbundle

parsers: fix uninitialize variable warning The heads pointer is not initialized correctly if filter is false, causing both clang and gcc to issue a warning. Correctly initialize heads to NULL.

revset: use `subset &` in bare `p2()` This takes advantage of the `fullreposet` smartness with a nice speedup. It's a similar speedup to `p1()` when a merge is in progress (the non merge case is already lightning fast anyway.)

revset: use `subset &` in bare `p1()` This takes advantage of the `fullreposet` smartness and yields a nice speedup. revset #0: p1() 0) wall 0.003256 comb 0.010000 user 0.010000 sys 0.000000 (best of 527) 1) wall 0.000066 comb 0.000000 user 0.000000 sys 0.000000 (best of 23224)

revset: use `subset &` in `rev` This takes advantage of the `fullreposet` smartness and yields a nice speedup. revset #0: rev(25) 0) wall 0.005480 comb 0.000000 user 0.000000 sys 0.000000 (best of 305) 1) wall 0.000052 comb 0.000000 user 0.000000 sys 0.000000 (best of 21891)

revset: use `subset &` in `origin` This takes advantage of the `fullreposet` smartness. revset #0: origin(tip) 0) wall 0.005353 comb 0.000000 user 0.000000 sys 0.000000 (best of 354) 1) wall 0.003080 comb 0.000000 user 0.000000 sys 0.000000 (best of 446)

revset: use `subset &` in `follow` This takes advantage of the `fullreposet` smartness. revset #0: follow(COPYING) 0) wall 0.002446 comb 0.000000 user 0.000000 sys 0.000000 (best of 735) 1) wall 0.000331 comb 0.000000 user 0.000000 sys 0.000000 (best of 5672)

revset: use `subset &` in `filelog` This takes advantage of the `fullreposet` smartness. revset #0: file(COPYING) 0) wall 3.179066 comb 3.180000 user 3.140000 sys 0.040000 (best of 3) 1) wall 2.723699 comb 2.730000 user 2.690000 sys 0.040000 (best of 4)

revset: use `subset &` in `divergent` This takes advantage of the `fullreposet` smartness. revset #0: divergent() 0) wall 0.002047 comb 0.000000 user 0.000000 sys 0.000000 (best of 813) 1) wall 0.000052 comb 0.000000 user 0.000000 sys 0.000000 (best of 22757)

revset: use `subset &` in `bisect` This takes advantage of the `fullreposet` smartness. revset #0: bisect(range) 0) wall 0.014007 comb 0.010000 user 0.010000 sys 0.000000 (best of 115) 1) wall 0.005556 comb 0.010000 user 0.010000 sys 0.000000 (best of 235)

revset: use `subset &` in `ancestorspec` This takes advantage of the `fullreposet` smartness. revset #0: tip~25 0) wall 0.004800 comb 0.010000 user 0.010000 sys 0.000000 (best of 259) 1) wall 0.002475 comb 0.000000 user 0.000000 sys 0.000000 (best of 717)

revset: use `subset &` in `bookmark` Speedup, Weeeeeee! revset #0: bookmark() 0) wall 0.002240 comb 0.000000 user 0.000000 sys 0.000000 (best of 571) 1) wall 0.000132 comb 0.000000 user 0.000000 sys 0.000000 (best of 14059)

revset: use `subset &` in `outgoing` This should give us the same benefit as elsewhere. Result is simpler (and "faster"). Outgoing is dominated by the discovery so no benchmark is provided.

revset: avoid in loop lookup in _generatorset._consumegen Python lookups are slow, so do all lookup outside of the for loop. This provide a small but still significant speedup: revset #0: 0:: 0) wall 0.063258 comb 0.060000 user 0.060000 sys 0.000000 (best of 100) 1) wall 0.057776 comb 0.050000 user 0.050000 sys 0.000000 (best of 100)

revset: reduce dict lookup in lazyset.__contains__ Avoid an extra dict lookup when we have to compute the value. No visible performance impact but this shaves the yak a few extra nanometers.

revset: do less lookup during spanset.__contains__ Attribute lookup is slow in python. So this version is going to be a bit faster. This does not have a visible impact since the rest of the stack is much slower but this shaves the yak a few extra nanometers. Moreover the new version is more readable so it worth doing this change for code quality purpose. This optimisation was approved by a core python dev.

largefiles: simplify iteration over standins Instead of iterating over all files in the context and ignoring those that are not standins, pass a standin-matcher to the context and iterate over only the files matching. Apart from making the intent clearer, this implementation will also benefit from any future optimizations done to the manifest walking code.

largefiles: avoid using 'lfiles' variable for two purposes The variable 'lfiles' is first used for a set of the names of all the large files. It is then overwritten with a tuple like the ones returned from status(). To reduce confusion, let's create a separate variable for the second use.

largefiles: remove unnecessary clearing of status fields At the end of lfilesrepo.status(), we clear the lists of unknown, ignored and clean files, depending on the values of 'listunknown' etc. The lists originate from other calls to status(), and it is only 'clean' that may get updated after the calls. Let's remove the need to clear any of the lists by explicitly only adding to 'clean' when 'listclean' is true.

mq: simplify patchheader handling of the empty line before the diff Don't try to append empty lines to HG patch headers - instead, add them in str method. This minor change removes some apparently redundant code and makes the code more robust.

mq: write '# Parent ' lines with two spaces like export does (BC) This aligns "Parent" node IDs with "Node ID" node IDs.

mq: write headers for new HG patches in the same order as export (BC)

mq: correctly make an empty line after description in new patches There would in some cases be an empty line between headers and the description - that does not seem right. There should also be an empty line between description and diff - but that was missing. These two mistakes would sometimes make it up for each other so we fix both at once to just show the improvement. Instead of writing an extra newline when writing a header line, write an extra line when it not is written as a part of the description but is necessary anyway.

largefiles: extract 'orig' method in reposetup.status

largefiles: extract 'orig' method in lfilesctx.filectx

largefiles: extract 'orig' method in lfilesmanifestdict.__contains__

largefiles: reduce indentation by dropping 'else' block after 'return'

largefiles: remove dead assignment left over from change 1ff42ee98446

match: simplify brittle predicate construction In match.__init__(), we create the matchfn predicate by and-ing together the individual predicates for includes, excludes (negated) and patterns. Instead of the current set of nested if/else blocks, we can simplify by adding the predicates to a list and defining the overall predicate in a generic way based on the components. We can still optimize it for the 0-length and 1-length cases. This way, there is no combinatorial explosion to deal with if new component predicates are added, and there is less risk of getting the overall predicate wrong.

convert: change default for git rename detection to 50% This default mirrors the default for 'git diff'. Other commands have slightly different defaults -- for example, the move/copy detection for 'git blame' assumes that a hunk is moved if more than 40 alphanumeric characters are the same, or copied if more than 20 alphanumeric characters are the same. 50% seems to be the most common default, though.

convert: simplify git.similarity parsing

revset: fast implementation for fullreposet.__and__ "And" operation with something that contains the whole repo should be super cheap. Check method docstring for details. This provide massive boost to simple revset that use `subset & xxx` revset #0: p1(20000) 0) wall 0.002447 comb 0.010000 user 0.010000 sys 0.000000 (best of 767) 1) wall 0.000529 comb 0.000000 user 0.000000 sys 0.000000 (best of 3947) revset #1: p2(10000) 0) wall 0.002464 comb 0.000000 user 0.000000 sys 0.000000 (best of 913) 1) wall 0.000530 comb 0.000000 user 0.000000 sys 0.000000 (best of 4226) No other regression spotted. More performance improvements are expected in the future as more revset predicate are converted to use `subset & xxx` The relaxed way `fullreposet` handles "&" operation may cause some trouble for people comparing smartset from different filter levels. I'm not sure such people exist and we can improve that aspect in later patches.

revset: turn spanset into a factory function We rename the `spanset` class to `_spanset`. `spanset` is now a function that builds either a `fullreposet` or a `_spanset` according to the argument passed. At some point, we may force people to explicitly use the `fullreposet` constructor, but the current approach makes it easier to ensure we use the new class whenever possible and focus on the benefits of this class.

revert: add a fullreposet class Every revset evaluation starts from `subset = spanset(repo)` and a lot of revset predicates build a `spansetrepo` for their internal needs. `spanset` is a generic class that can handle any situation. As a result a lot of operation between spanset result in an `orderedlazyset`, a safe object but suboptimal in may situation. So we introduce a `fullreposet` class where some of the operation will be overwritten to produce more interesting results.

obsolete: ensure that `getrevs` always return a set When a set of revisions was empty, we were using an empty tuple. We now return an empty frozenset to ensure the object could be used in an operation that requires a set.

hgweb: fail if an invalid command was supplied in url path (issue4071) Traditionally, the way to specify a command for hgweb was to use url query arguments (e.g. "?cmd=batch"). If the command is unknown to hgweb, it gives an error (e.g. "400 no such method: badcmd"). But there's also another way to specify a command: as a url path fragment (e.g. "/graph"). Before, hgweb was made forgiving (looks like it was made in 44c5157474e7) and user could put any unknown command in the url. If hgweb couldn't understand it, it would just silently fall back to the default command, which depends on the actual style (e.g. for paper it's shortlog, for monoblue it's summary). This was inconsistent and was breaking some tools that rely on http status codes (as noted in the issue4071). So this patch changes that behavior to the more consistent one, i.e. hgweb will now return "400 no such method: badcmd". So if some tool was relying on having an invalid command return http status code 200 and also have some information, then it will stop working. That is, if somebody typed foobar when they really meant shortlog (and the user was lucky enough to choose a style where the default command is shortlog too), that fact will now be revealed. Code-wise, the changed if block is only relevant when there's no "?cmd" query parameter (i.e. only when command is specified as a url path fragment), and looks like the removed else branch was there only for falling back to default command. With that removed, the rest of the code works as expected: it looks at the command, and if it's not known, raises a proper ErrorResponse exception with an appropriate message. Evidently, there were no tests that required the old behavior. But, frankly, I don't know any way to tell if anyone actually exploited such forgiving behavior in some in-house tool.

keepalive: fix how md5 is used The code in keepalive dates from when it was importing the md5 module directly and uses md5.new. Since then, what 'md5' means has been changed from an import of the md5 module to being a function using the right module between hashlib and md5, so the md5.new idiom doesn't work anymore.

merge with stable

revset: add an optimised baseset.__contains__ (issue4371) The baseset class is based on a python list. This means that base.__contains__ was absolutely as crappy as list.__contains__. We now rely on __contains__ from the underlying set. This will avoid having to explicitly convert the baseset to a set (using baseset.set()) whenever one want fast membership test. Apparently there is already code that forgot to do such conversions since we observe a massive speedup in some test. revset #25: roots((0::) - (0::tip)) 0) wall 2.079454 comb 2.080000 user 2.080000 sys 0.000000 (best of 5) 1) wall 0.132970 comb 0.130000 user 0.130000 sys 0.000000 (best of 65) No regression is observed in benchmarks. This change improve the issue4371 back to acceptable situation (but are still slower than manual substraction)

merge with stable

commands: deprecate the parents commands It's replaced by 'hg summary' or hg log -r 'parents(foo)' and doesn't need to take up space in our command list anymore.

revset: remove nullrev from the bookmark computation Same as for other revset we sanitize the content of the set to be able to rely on it more.

revset: unify code flow in `bookmark` We refactor the code of the bookmark revset to have a single return. This will allow us to sanitize the content of the set.

revset: remove invalid value in the origin set Same as the parents related revsets, origin had some invalid value in the computed set. We remove them.

revset: remove nullrev from set computed in parents() The old code relied on the subset contents to get rid of invalid values. We would like to be able to rely more on the computation in parents() so we filter out the invalid value.

revset: refactor parents() into a single return point Both paths are doing similar thing in the end. We refactor the function so that the `ps` set is commonly used at the end. This will end excluding `nullrev` from this set in a future patch

revset: remove nullrev from set computed in p1() and p2() The old code relied on the subset contents to get rid of invalid values. We would like to be able to rely more on the computation in p1() and p2() so we filter out the invalid value

revset: document the choice made in __generatorset.__iter__ The method code looks a bit ugly but has good reasons to. We document them to prevent naive refactoring in the future.

help: mention mode in hg log --removed help (issue4381)

commit: catch changed exec bit on files from p1 (issue4382)

revert: add a `drop` action This prevents the need for a try except in the `_performrevert` code.

revert: explicitly track added but deleted file Added + deleted file are files that need to be untracked from the dirstate but that are already missing on disk. The current `_performrevert` code is handling that with exception catching. We will be able to do better with a dedicated set.

revert: have an explicit action for "forget" The distinction between "remove" and "forget" used to be in special logic checking for the state of the file in the dirstate. Now that we have dedicated filtering, we can stop relying on this logic and have two distinct actions.

revert: split between newly added file and file added in other changeset These two kinds of files are handled differently. One is deleted and the other is just forgotten (the file is untracked but left in place). The distinction is done in the `_performrevert` code itself and we would like to get ride of this.

revset: stop using a baseset instead of a plain list in _revsbetween The function internal code needs a list. Lets use a list.

run-tests: added 'cuser', 'csys' time info in report.json file This patch adds up a 'cuser' and 'csys'(cputime) info in report.json file which generated when --json is enabled while testing. Now the new format of report.json file is as below. testreport ={ "test-success.t": { "csys": "1.041", "cuser": "1.041", "result": "success", "time": "2.041" } "test-failure.t": { "csys": "1.041", "cuser": "1.041", "result": "failure", "time": "4.430" } "test-skip.t": { "csys": "1.041", "cuser": "1.041", "result": "skip", "time": "3.754" } }

import: let --exact 'work' with --no-commit (issue4376)

obsolete: use C code for headrevs calculation Previously, if there were filtered revs the repository could not use the C fast path for computing the head revs in the changelog. This slowed down many operations in large repositories. This adds the ability to filter revs to the C fast path. This speeds up histedit on repositories with filtered revs by 30% (13s to 9s). This could be improved further by sorting the filtered revs and walking the sorted list while we walk the changelog, but even this initial version that just calls __contains__ is still massively faster. The new C api is compatible for both new and old python clients, and the new python client can call both new and old C apis.

revset: simplify orderedlazyset creation in spanset method We can simply use the `self.isascending` value instead of more complex if/else clause. This get the code simpler. Benchmarks show no performances harmed in the process.

revset: use spanset.isdescending in multiple simple places We call the method directly instead of duplicating checks. Benchmarks show no performances harmed in the process.

revset: wider definition of ascending and descending for spanset Before this patches, empty spanset were seen as neither ascending nor descending. This is mathematically wrong and create some edges case. We put `isascending` and `isdescending` back on track so we can use them to simplify some of the spanset code. Benchmarks show no performances harmed in the process.

annotate: port to generic templater enabled by hidden -T option If the selected formatter is other than plainformatter, raw data are passed to the formatter. In this case, it isn't necessary (and not possible) to calculate column widths. Field names are substituted to be the same as "log" command. There are a few limitations: - "binary file" message is not included in formatted output. - no data structure for multiple files. all lines are packed to single list.

annotate: split functions to get data without applying text formatting This prepares for porting to generic templater API, where raw data should be passed to the formatter. makefunc() is necessary to build closure in list comprehension.

annotate: remove unused variable in calculation of column widths

annotate: build format string separately from annotation data This prepares for porting to generic templater API. Note that we cannot use '%*s' to pad white spaces because it doesn't take into account character widths, as described in 4f5a6df2af92.

formatter: convert float value to json It will be used to encode ctx.date().

formatter: have jsonformatter accept tuple as value This is necessary for "annotate" to encode ctx.date() in the same manner as jsonchangeset printer. It doesn't support list object because keeping mutable object in _item could be a source of hidden bugs. Also, I can't think of the use case.

formatter: extract function that encode values to json string This is the stub for tuple support, which will be used to encode ctx.date() in the same manner as jsonchangeset printer.

contrib/synthrepo: pass options to ctx.diff as kwargs, not a dict

contrib/synthrepo: only generate 2 parents if model contains merges If `hg analyze` is run on a revision set which contains no merges, then `hg synthesize` will raise IndexError trying to select from p2distance, which will be empty.

convert: add support to find git copies from all files in the working copy I couldn't think of a better name for this option, so I stole the Git one in the hope that anyone converting a Git repo knows what it means.

convert: add support to detect git renames and copies Git is fairly unique among VCSes in that it doesn't record copies and renames, instead choosing to detect them on the fly. Since Mercurial expects copies and renames to be recorded, it can be valuable to preserve this history while converting a Git repository to Mercurial. This patch adds a new convert option, called 'convert.git.similarity', which determines how similar files must be to be treated as renames or copies.

convert: for git, factor out code to add entries to a separate function We're going to call this for multiple files in one iteration in upcoming patches.

convert: for git's getchanges, always split entry line into components We always need to know whether the entry is a rename or copy, so split it up unconditionally.

convert: for git's getchanges, use explicit index for iteration Upcoming patches will add support for copies and renames, for which we'll need to access multiple lines of the difftree output at once.

convert: add initial docs for git sources Upcoming patches will add config options for git sources. This patch adds a place to document them.

color: document that changeset phases have labels It's very useful to be able to colourise csets according to their phases. There was no indication anywhere in the docs that this is possible. We use e.g. `changeset.secret = ` instead of `changeset.secret ='none'`, because otherwise this is a BC: it would nullify the effects given to log.changeset label that usually surrounds the changeset.{phase} labels. Specifying the label without any effect instead of 'none' is a true no-op change and purely documentation.

color: change the debug output format Before, the format was label(labeled text) # single label [label1 label2](labeled text) # multiple Now, it's [labels|labeled text] ..which should make things a bit more clear.

color: enable debug option to show labels This is a debug option for showing labels. This can be helpful for knowing which labels are available for colouring or to see the output when defining your own templates. A couple of tests are included.

color: document that labels are used for colorizing text It is a deeply hidden secret that it's possible to colorise so many things with so many different labels. This is an attempt to document this. The text is a bit long, but it seems as short as can be while documenting everything. Perhaps it should be hidden under a --verbose option.

contrib: add OS X p4merge to mergetools.hgrc

patch: enable diff.tab markup for the color extension The following patch splits up changed lines along tabs (using re.findall), and gives them a "diff.tab" label. This can be used by the color extension for colorising tabs, like it does right now with trailing whitespace. I also provide corresponding tests.

dirstate: copyedit exception for no beginparentchange call

revsetbenchmarks: add an additional roots() benchmark The existing roots(x - y) revset only considered the most recent 100 revisions. This was a good start. But expanding it to the full history of the repository can dramatically increase execution time and thus constitutes a useful benchmark.

merge with stable

mq: examine "pushable" of already applied patch correctly Before this patch, "hg qselect" with --pop/--reapply may pop patches unexpectedly, even when all of patches applied before "qselect" are still pushable. Strictly speaking about the condition of this issue: - before "qselect" - there are N applied patches - the index of the guarded patch X in the series is less than N - after "qselect" - X is still guarded, and - all of applied patched are still pushable In the case above, "hg qselect" should keep current status, but it actually tries to pop patches because of X. The index in "the series" should be used to examine "pushable" of a patch by "mq.pushablek()", but the index in "applied patches" is used, and this may cause unexpected examination of guarded patch. To examine "pushable" of already applied patch correctly, this patch uses "mq.applied[i].name": "pushable" is the function introduced by the previous patch, and it returns "mq.pushable(mq.applied[i].name)[0]".

mq: pop correct patches when changing pushable-ness of already applied ones Before this patch, "hg qselect" with --pop/--reapply may pop incorrect patches, because the index in "applied patches" is used to pop patches by "mq.pop()", even though the index in "the series" should be used. For example, when the already applied patch becomes guarded and it follows the already guarded (= not yet applied) one, "hg qselect" is aborted, because it tries to pop to guarded one. This patch uses "mq.applied[i - 1].name" to pop to the patch, of which the index in the "applied ones" is "i - 1".

mq: use "mq.applied[i].name" instead of "mq.appliedname(i)" for safety Before this patch, "hg qselect --reapply" is aborted when "--verbose" is specified, because "mq.appliedname()" returns "INDEX PATCHNAME" instead of "PATCHNAME" in such case and "mq.push" can't accept the former as the name of patch. This patch uses "mq.applied[i].name" instead of "mq.appliedname(i)" as the name of the patch to be pushed for safety. Now, there is no code path using "mq.appliedname()", and it should be removed to prevent developers from using it in the wrong way like this issue.

mq: report correct numbers for changing "number of guarded, applied patches" Before this patch, "hg qselect" may report incorrect numbers for "number of guarded, applied patches has changed", because it examines "pushable" of patches by the index not in "the series" but in "applied patches", even though "mq.pushable()" expects the former. To report correct numbers for changing "number of guarded, applied patches", this patch uses the name of applied patch to examine pushable-ness of it. This patch also changes the result of existing "hg qselect" tests, because they doesn't change pushable-ness of already applied patches. This patch assumes that "hg qselect" focuses on changing pushable-ness only of already applied patches, because: - the report message uses not "previous" (in the series) but "applied" - the logic to pop patches for --pop/--reapply examines pushable-ness only of already applied ones (in fact, there are some incorrect code paths)

annotate: remove redundant check for empty list of annotation data It isn't necessary because zip(*pieces) returns [] if pieces are empty, and pieces are empty only if lines are empty.

revset: lower weight for _intlist function The histedit command uses a revset like: (_intlist('1234\x001235')) and merge() Previously the optimizer gave a weight of 1.5 to the _intlist side (1 for the function, 0.5 for the string) which caused it to process the merge() side first. This caused it to evaluate merge against every commit in the repo, which took 2.5 seconds on a large repo. I changed the weight of _intlist to 0, since it's a trivial calculation, which makes it process intlist first, which makes merge apply only to the revs in the list. Which makes the revset take 0.15 seconds now. Cutting off 2.4 seconds off our histedit performance. >From the revset benchmark: revset #25: (_intlist('20000\x0020001')) and merge() 0) obsolete feature not enabled but 54243 markers found! ! wall 0.036767 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) 1) obsolete feature not enabled but 54243 markers found! ! wall 0.000198 comb 0.000000 user 0.000000 sys 0.000000 (best of 9084)

revset: make parents() O(number of parents) Strip executes a revset like this: max(parents(_intlist('1234\x001235')) - _intlist('1234\x001235')) Previously the parents() revset would do 'subset & parents' which iterates over each item in the subset and checks if it's in parents. subset is usually the entire repo (a spanset) so this takes a while. Reversing the parameters to be 'parents & subset' means the operation becomes O(number of parents) instead of O(size of repo). It also means the result gets evaluated immediately (since parents isn't a lazy set), but I think this is a win in most scenarios. This shaves 0.3 seconds off strip (amend/histedit/rebase/etc) for large repositories. revset #0: parents(20000) 0) obsolete feature not enabled but 54243 markers found! ! wall 0.006256 comb 0.010000 user 0.010000 sys 0.000000 (best of 289) 1) obsolete feature not enabled but 54243 markers found! ! wall 0.000391 comb 0.000000 user 0.000000 sys 0.000000 (best of 4323)

revset: make descendants() lazier Previously descendants() would force the provided subset to become a set. In the case of revsets like '(%ld::) - (%ld)' (as used by histedit) this would force the '- (%ld)' set to be evaluated, which produced a set containing every commit in the repo (except %ld). This takes 0.6s on large repos. This changes descendants to trust the subset to implement __contains__ efficiently, which improves the above revset to 0.16s. Shaving 0.4 seconds off of histedit. revset #27: (20000::) - (20000) 0) obsolete feature not enabled but 54243 markers found! ! wall 0.023640 comb 0.020000 user 0.020000 sys 0.000000 (best of 100) 1) obsolete feature not enabled but 54243 markers found! ! wall 0.019589 comb 0.020000 user 0.020000 sys 0.000000 (best of 100) This commit removes the final revset related perf hotspot from histedit. Combined with the previous two patches, they shave a little over 3 seconds off histedit on large repos.

check-code: look for misuse of __bool__

formatter: correct bool testing which should be __nonzero__ in Python 2

contrib/synthrepo: return None to delete files on commit, don't raise IOError The internal commit API was changed in 650b5b6e75ed to expect None from the filectx function when a file is to be deleted, not an IOError. This change keeps synthrepo up-to-date.

wix: contrib/sample.hgrc is no more

dockerrpm: create a yum/dnf repo from the generated rpms This gives "PPA" functionality where users easily can stay uptodate with latest nightly build.

docker: add CentOS 5 There is no official CentOS 5 docker template so we use one from saltstack.

docker: use stable tags for fedora and centos A moving target is rarely useful.

dockerrpm: prepare source outside docker and just run rpmbuild inside docker Simplifies the rpm build process. We will use platform specific rpmbuild directories and will not clean them and will drop the explicit copy to build directory.

dockerrpm: run docker build process as the current user, not as root Docker can be run by ordinary users if they are in the docker group. The build process would however be run as a root user, only protected by the sandboxing. That caused problems with the shared directory where rpmbuild would be picky about building from sources owned by less privileged users and producing files owned by root. Instead, add a build user with the right uid/gid to the image and run the docker process as that user.

dockerrpm: better handling of specification of docker name

dockerrpm: check that docker is running correctly before building

buildrpm: introduce --rpmdir instead of using hardcoded rpmbuild dir Used as rpm _topdir when preparing spec and source and building rpms.

buildrpm: introduce --withpython for building rpms that includes Python 2.7

buildrpm: introduce --prepare for preparing without actually building rpms

templater: add "diff" template function "diff" allows to embed changes in the target revision into template output, even if the command itself doesn't take "--patch" option Combination of "[committemplate]" configuration and "diff" template function can achieve the feature like issue231 ("option to have diff displayed in commit editor buffer") http://bz.selenic.com/show_bug.cgi?id=231 For example, templating below can be used to add each "diff" output lines "HG: " prefix:: {splitlines(diff) % 'HG: {line}\n'} This patch implements "diff" not as "a template keyword" but as "a template function" to take include/exclude patterns at runtime. It allows to specify target files of command (by -I/-X command line options) and "diff" separately.

locate: add pointer to files command in help

tests: fix breakage from locate deprecation

locate: deprecate in favor of files

formatter: add pickle format This gives convenient Python2 output. Python 3 users will need encoding=bytes.

commands: add hidden -T option for files/manifest/status/tags These commands have generic formatting support but no way to enable it yet. When this feature is more fully developed, this flag will be unhidden.

formatter: add json formatter

cmdutil: add json style to log-like commands Sadly, this can't be done with the normal templater like we do with XML due to JSON's inter-record comma rules.

encoding: add json escaping filter This ends up here because it needs to be somewhat encoding aware.

encoding: handle empty string in toutf8

formatter: make debug style match Python syntax

files: add new command unifying locate and manifest functionality