subrepo: add basic support to hgsubrepo for the files command Paths into the subrepo are not yet supported. The need to use the workingctx in the subrepo will likely be used more in the future, with the proposed working directory revset symbol. It is also needed with archive, if that code is to be reused to support 'extdiff -S'. Unfortunately, it doesn't seem possible to put the smarts in subrepo.subrepo(), as it breaks various status and diff tests. I opted not to pass the desired revision into the subrepo method explicitly, because the only ones that do pass an explicit revision are methods like status and diff, which actually operate on two contexts- the subrepo state and the explicitly passed revision.

mergecopies: reuse ancestry context when traversing file history (issue4537) Merge copies is traversing file history in search for copies and renames. Since 3.3 we are doing "linkrev adjustment" to ensure duplicated filelog entry does not confuse the traversal. This "linkrev adjustment" involved ancestry testing and walking in the changeset graph. If we do such walk in the changesets graph for each file, we end up with a 'O(<changesets>x<files>)' complexity that create massive issue. For examples, grafting a changeset in Mozilla's repo moved from 6 seconds to more than 3 minutes. There is a mechanism to reuse such ancestors computation between all files. But it has to be manually set up in situation were it make sense to take such shortcut. This changesets set this mechanism up and bring back the graph time from 3 minutes to 8 seconds. To do so, we need a bigger control on the way 'filectx' are instantiated during each 'checkcopies' calls that 'mergecopies' is doing. We add a new 'setupctx' that configure and return a 'filectx' factory. The function make sure the ancestry context is properly created and the factory make sure it is properly installed on returned 'filectx'.

adjustlinkrev: handle 'None' value as source When the source rev value is 'None', the ctx is a working context. We cannot compute the ancestors from there so we directly skip to its parents. This will be necessary to allow 'None' value for '_descendantrev' itself necessary to make all contexts used in 'mergecopies' reuse the same '_ancestrycontext'.

adjustlinkrev: prepare source revs for ancestry only once We'll need some more complex initialisation to handle workingfilectx case. We do this small change in a different patch for clarity.

subrepo: add the parent context to hgsubrepo This brings parity with gitsubrepo and svnsubrepo (which already reference their parent), and will be used in an upcoming patch. I'm a bit concerned that the parent context could get stale (consider what happens when the parent repo is reverted for example). I tried adding the parent context to the substate tuple so that the parent is available everywhere a state change is possible, but that made submerge() unhappy. Even with removing the parent context inside submerge(), I wasn't able to get all of the test diffs fixed. But since the other subrepos reference their parent too, if there is a problem, it is a preexisting one (that nobody seems to be running into). It can be fixed if/when it pops up.

fileset: add a fileset for portable filenames This has mostly the same semantics as the files that the 'ui.portablefilenames' config option would warn or abort about. The only difference is filenames that case-fold to the same string -- given a set of filenames we've already checked we can check whether a new one collides with them, but we don't have a way to tell which filename it collided with.

annotate: reuse ancestry context when adjusting linkrev (issue4532) The linkrev adjustment will likely do the same ancestry walking multiple time so we already have an optional mechanism to take advantage of this. Since 2896f53509a7, linkrev adjustment was done lazily to prevent too bad performance impact on rename computation. However, this laziness created a quadratic situation in 'annotate'. Mercurial repo: hg annotate mercurial/commands.py before: 8.090 after: 36.300 Mozilla repo: hg annotate layout/generic/nsTextFrame.cpp before: 1.190 after: 290.230 So we setup sharing of the ancestry context in the annotate case too. Linkrev adjustment still have an impact but it a much more sensible one. Mercurial repo: hg annotate mercurial/commands.py before: 36.300 after: 10.230 Mozilla repo: hg annotate layout/generic/nsTextFrame.cpp before: 290.230 after: 5.560

treemanifest: make hasdir() faster Same rationale as the previous change.

treemanifest: make filesnotin() faster Same rationale as the previous change.

treemanifest: make diff() faster Containment checking is slower in treemanifest than it is in manifestdict, making the current diff algorithm O(n log n). By traversing both treemanifests in parallel, we can make it O(n). More importantly, once we start lazily loading submanifests, we will be able to easily skip entire submanifest if they have the same nodeid.

treemanifest: store directory path in treemanifest nodes This leads to less concatenation while iterating, and it's useful for debugging.

treemanifest: add configuration for using treemanifest type This change adds boolean configuration option experimental.treemanifest. When the option is enabled, manifests are parsed into the new treemanifest type. Tests can be now run using treemanifest by switching the config option default in localrepo._applyrequirements(). Tests pass even when made to randomly choose between manifestdict and treemanifest, suggesting that the two types produce identical manifests (so e.g. a manifest revlog entry written from a treemanifest can be parsed by the manifestdict code).

treemanifest: create treemanifest class There are a number of problems with large and flat manifests. Copying from http://mercurial.selenic.com/wiki/ManifestShardingPlan: * manifest too large for RAM * manifest resolution too much CPU (long delta chains) * committing is slow because entire manifest has to be hashed * impossible for narrow clone to leave out part of manifest as all is needed to calculate new hash * diffing two revisions involves traversing entire subdirectories even if identical This is a first step in a series introducing a manifest revlog per directory. This change adds a new manifest class: treemanifest, which is a tree where each node has a dict of files (nodeids), a dict of flags, and a dict of subdirectories (treemanifests). So far, it behaves just like manifestdict, but it will later help us write one manifest revlog per directory. The new class is still unused; it will be used after the next change. The code is not yet optimized. Running with it (see below) makes most or all operations slower. Once we start storing manifest revlogs for every directory, it should be possible to make many of these operations much faster. The fastdelta() optimization has been intentionally not implemented for the treemanifests. We can implement it later if necessary. All tests pass when run with the following patch (and without, of couse): --- a/mercurial/manifest.py Thu Mar 19 11:08:42 2015 -0700 +++ b/mercurial/manifest.py Thu Mar 19 11:15:50 2015 -0700 @@ -596,7 +596,7 @@ class manifest(revlog.revlog): return None, None def add(self, m, transaction, link, p1, p2, added, removed): - if p1 in self._mancache: + if False and p1 in self._mancache: # If our first parent is in the manifest cache, we can # compute a delta here using properties we know about the # manifest up-front, which may save time later for the @@ -626,3 +626,5 @@ class manifest(revlog.revlog): self._mancache[n] = (m, arraytext) return n + +manifestdict = treemanifest

bookmarks: rewrite comparing bookmarks in commands.summary() by compare() This patch adds utility function "summary()", to replace comparing bookmarks in "commands.summary()". This replacement finishes centralizing the logic to compare bookmarks into "bookmarks.compare()". This patch also adds test to check summary output with incoming/outgoing bookmarks, because "hg summary --remote" is not tested yet on the repository with incoming/outgoing bookmarks. This test uses "(glob)" to ignore summary about incoming/outgoing changesets.

bookmarks: remove useless diff() Previous patches removed code paths referring it.

bookmarks: add outgoing() to replace diff() for outgoing bookmarks This replacement makes enhancement of "show outgoing bookmarks" easy, because "compare()" can detect more detailed difference of bookmarks between two repositories.

bookmarks: add incoming() to replace diff() for incoming bookmarks This replacement makes enhancement of "show incoming bookmarks" easy, because "compare()" can detect more detailed difference of bookmarks between two repositories.

manifest: avoid intersectfiles for matches > 100 files Previously we tried to avoid manifest.intersectfiles for exact matches with less than 100 files. However, when the left side of the "or" is false, the right side gets evaluated, of course, and the evaluation of "util.all(fn in self for fn in files)" is both costly in itself, and likely to be true, causing intersectfiles() to be called after all. Fix this by moving the check for less than 100 files outside of the "or" expression, thereby also making it apply for a non-exact matcher, should one be passed in.

convert: optimize convert of files that are unmodified from p2 in merges Conversion of a merge starts with p1 and re-adds the files that were changed in the merge or came unmodified from p2. Files that are unmodified from p1 will thus not be touched and take no time. Files that are unmodified from p2 would be retrieved and rehashed. They would end up getting the same hash as in p2 and end up reusing the filelog entry and look like the p1 case ... but it was slow. Instead, make getchanges also return 'files that are unmodified from p2' so the sink can reuse the existing p2 entry instead of calling getfile. Reuse of filelog entries can make a big difference when files are big and with long revlong chains so they take time to retrieve and hash, or when using an expensive custom getfile function (think http://mercurial.selenic.com/wiki/ConvertExtension#Customization with a code reformatter). This in combination with changes to reuse filectx entries in localrepo._filecommit make 'unchanged from p2' almost as fast as 'unchanged from p1'. This is so far only implemented for the combination of hg source and hg sink. This is a refactoring/optimization. It is covered by existing tests and show no changes - which is a good thing.

localrepo: reuse commit of parent filectx entries without rehashing It is currently only amend and debugbuilddag that will pass a filectx to localrepo._filecommit. Amend will usually not hit the case where a the filectx is a parent that just can be reused. Future convert changes will use it more.

obsolete: avoid infinite loop from obs-cycle in divergence (issue4126) As for other currently in place cycle detection, arbitrarily cut the first obsolescence link that create a cycle avoiding the infinite loop. This will have to be made more deterministic in the future but we do not really care right now.

tests: avoid deprecation warning

walkchangerevs: make followfilter a top-level class The class only depends on the 'repo' variable in the closure, so let's move the class out of the function and make it explicit that that (the repo) is all it needs.

patch._applydiff: resolve prefix with respect to the cwd This has several advantages compared to resolving it relative to the root: - '--prefix .' works as expected. - consistent with upcoming 'hg diff' option to produce relative patches (I made sure to put in the (glob) annotations this time!)

test-revert-interactive: eliminate bashism in wildcard pattern The test failed with Debian dash.

devel: also warn about transaction started without a lock Nobody should start a transaction on an unlocked repository. If developer warnings are enabled this will be reported. This use the same config as bad locking order since this is closely related.

contrib: remove the now useless lock-checker.py extension This feature is in core now, and we do not keep backward compability for contrib.

devel: move the lock-checking code into core If the developer warnings are enabled, bad locking order will be reported without the need for the contrib extension.

patch.pathtransform: prepend prefix even if strip is 0 Silly oversight by me.

walkchangerevs: simplify by using match.always() method Since "slowpath = match.anypats() or (match.files() and opts.get('removed'))", we can simplify the condition "not slowpath and not match.files()" to "not match.anypats() and not match.files()", which is equivalent to "match.always()".

localrepo: simplify by using match.always() method

localrepo: remove check for matcher object that's never None

context.walk: walk all files when file and '.' given When both '.' (the working copy root) and an explicit file (or files) are in match.files(), we only walk the explicitly listed files. This is because we remove the '.' from the set too early. Move later and add a test for it. Before this change, the last test would print only "3".

context.walk: call with util.all() a generator, not a list The file set can be large, so avoid going through the entire file set when a file happens not to be in the context.

obsolete: remove last instance of _enabled The _enabled bool has been replaced by obsolete.isenabled(...). This removes the last instance of it so I can remove the _enabled flag entirely shortly.

revbranchcache: write cache even during read operations Previously we would only actually write the revbranchcache to disk if we were in the middle of a write operation (like commit). Now we will also write it during any read operation. The cache knows how to invalidate itself, so it shouldn't become corrupt if multiple writers try at once (and the write-on-read behavior/risk is the same as all our other caches).

revbranchcache: move cache writing to the transaction finalizer Instead of writing the revbranchcache during updatecache (which often happens too early, before the cache is even populated), let's run it as part of the transaction finalizer. It still won't be written for read-only operations, but that's no worse than it is today. A future commit will remove the actual write that happens in updatecache(). This is also good prep for when all caches get moved into the transaction.

revbranchcache: populate cache incrementally Previously the cache would populate completely the first time it was accessed. This could take over a minute on larger repos. This patch changes it to update incrementally. Only values that are read will be written, and it will only rewrite as much of the file as strictly necessary. This adds a magic value of '\0\0\0\0' to represent an empty cache entry. The probability of this matching an actual commit hash prefix is tiny, so it's ok if that's always considered a cache miss. This is also BC safe since any existing entries with '\0\0\0\0' will just be considered misses. Perf numbers: Mozilla-central: hg --time log -r 'branch(mobile)' -T. Cold Cache: 14.7s -> 15.1s (3% worse) Warm Cache: 1.6s -> 2.1s (30% worse) Mozilla-cental: hg perfbranchmap 2s -> 2.4s (20% worse) hg: hg log -r 'branch(stable) & branch(default)' Cold Cache: 3.1s -> 1.9s (40% better - because the old code missed the cache on both branch() revset iterations, so it did twice the work) Warm Cache: 0.2 -> 0.26 (30% worse) internal huge repo: hg --time log -r 'tip & branch(default)' Cold Cache: 65.4s -> 0.2s (327x better) While this change introduces minor regressions when iterating over every commit in a branch, it massively improves the cold cache time for operations which touch a single commit. I feel the better O() is worth it in this case.

revbranchcache: move entry writing to a separate function This moves the actual writing of entries to the cache to a separate function. This will allow us to use it in multiple places. Ex: in one place we will write dummy entries, and in another place we will write real data.

revbranchcache: store repo on the object Previously we would instantiate the revbranchcache with a repo object, use it briefly, then require it be passed in every time we wanted to fetch any information. This seems unnecessary since it's obviously specific to that repo (since it was constructed with it). This patch stores the repo on the revbranchcache object, and removes the repo parameter from the various functions on that class. This has the other nice benefit of removing the double-revbranchcache-read that existed before (it was read once for the branch revset, and once for the repo.revbranchcache).

revbranchcache: move out of branchmap onto localrepo Previously the revbranchcache was a field inside the branchmap. This is bad for a couple reasons: 1) There can be multiple branchmaps per repo (one for each filter level). There can only be one revbranchcache per repo. In fact, a revbranchcache could only exist on a branchmap that was for the unfiltered view, so you could have branchmaps exist for which you couldn't have a revbranchcache. It was funky. 2) The write lifecycle for the revbranchcache is going to be different from the branchmap (branchmap is greedily written early on, revbranchcache should be lazily computed and written). This patch moves the revbranchcache to live as a field on the localrepo (alongside self._branchmap). This will allow us to handle it's lifecycle differently, which will let us move it to be lazily computed in future patches.

revbranchcache: add test for when the cache is not writable The revbranchecache code already handled the case when the cache file wasn't writable, but let's add a test as well so future changes don't regress this.

patch.trydiff: add a docstring It took me a bit to figure out what this function actually does.

keyword: monkeypatch patch.diff more generically This function doesn't need access to any of the args or kwargs, so make the monkeypatching more robust. (In upcoming patches we'll introduce another argument to patch.diff, and this function would break if it weren't for this patch.)

keyword: rename kw_diff to kwdiff in keeping with Mercurial style rules In an upcoming patch we'll change this function's signature. If the name is kept the same, test-check-commit-hg.t complains.

fetch: use an abort hint where appropriate

eol: replace "working copy" with "working directory" in extension help

revset: replace "working copy" with "working directory" in function help

commands: say "working directory" in full spelling

commands: replace "working copy" with "working directory" in help/messages "working directory" is the standard term, we should use it consistently. But I didn't touch the hint, "run 'hg update' to get a working copy", because "get a working directory" sounds a bit odd.

test-commit-interactive-curses: #require 'tic' When run on Windows, this test aborts: @@ -20,7 +20,8 @@ > X > EOF $ hg commit -i -m "a" -d "0 0" - no changes to record + abort: No module named fcntl! + [255] $ hg tip changeset: -1:000000000000 tag: tip Maybe there's another way to get the screen size on Windows (it dies in crecord.gethw()), but for now, quiet the test noise by skipping it if terminal info compiler is unavailable.

check-code: enforce the usage of 'seq.py' instead of 'seq'

tests: replace uses of 'seq' with portable 'seq.py'

tests: introduce 'seq.py' as a portable replacement for 'seq' OS X 10.6.8 doesn't have it, while 10.10 does. I'm not sure when it was added. It may be missing from other platforms as well. This currently doesn't handle the string manipulation options (-f, -s and -w in MinGW anyway), since there is currently no need for it. Since xrange defaults to starting at 0 instead of 1, and treats the end as exclusive instead of inclusive, the args need to be extracted instead of doing: xrange(*[int(a) for a in sys.argv[1:]]) Therefore, the step might as well be added, even though there is no current use.

revert: add flag to make revert interactive

record: move ui.write wrapping where it should be We have to do that for the coloring to work in interactive revert

record: refactor the filtering code

record: consolidate ui.write wrapping in a function

bookmarks: reuse @number bookmark, if it refers changeset referred remotely Before this patch, "@number" suffixed bookmark may be newly created at each "hg pull" from the remote repository, if the bookmark in remote repository diverges from one in local one. This causes unexpected increase of "@number" suffixed bookmarks. This patch reuses "@number" suffixed bookmark, if it refers the changeset which is referred by the same bookmark in the remote repository.

bookmarks: check @pathalias suffix before available @number for efficiency Before this patch, available "@number" suffix is searched before "@pathalias" suffix, even though the latter has higher priority than the former if the latter exits. This patch checks "@pathalias" suffix before available "@number" for efficiency. When an URL has multiple path definitions, the first one is used for "pathalias" after this patch, even though the last one is used before this patch, because: - this choice can terminate loop immediately for efficiency - such case seems to be rare

bookmarks: prevent divergent bookmark from being updated unexpectedly Before this patch, "@99" suffixed bookmark may be updated unexpectedly by the bookmark value on the remote side at "hg pull", if all of "@1" to "@99" suffixed bookmarks exist in the local repository, because variable "n" still refers "@99" suffixed bookmark after the loop to examine "@num" suffixes, even though it already exists in the local repository. This patch prevents divergent bookmark from being updated unexpectedly, and shows warning message in such situation. This patch uses original python script "seq.py" instead of "seq" command to create sequence numbers in the test, because "seq" command may not be available: it isn't defined in recent POSIX specification (POSIX.1-2001 2013 Edition or XPG7)

manifest: include Python.h before standard headers Python.h should be included before any standard headers according to the python docs: https://docs.python.org/2/c-api/intro.html#include-files

crecord: use colwidth instead of ucolwidth

manifest: speed up matches for large sets of files If the number of files being matched is large, the bisection overhead can dominate, which caused a performance regression for revert --all and histedit. This introduces a (fairly arbitrary) cross-over from using bisections to bulk search.

perf: add methods for timing changeset file list reading

censor: add exchange tests, via local push/pull and bundle/unbundle These tests exercise the edge cases of exchanging censored commits between censor-aware Mercurial clients. These tests do not extend to cover backwards-compatible exchange; such tests require a script which builds older versions of Mercurial. For more background on the censorship feature design, see: http://mercurial.selenic.com/wiki/CensorPlan

censor: add censor command to hgext with basic client-side tests The censor command is a core extension which can replace the contents of a historical file revision with a censor "tombstone" which can be exchanged with older clients in place of the real revision data. The command rewrites the filelog by copying revision-by-revision. Care must be taken to expand the fulltext of the children of the censored revision before copying them to the new filelog; they might be stored as deltas against the uncensored revision, and those deltas will be invalidated. For more background on the censorship feature design, see: http://mercurial.selenic.com/wiki/CensorPlan

record: add comparison methods for recordhunk class

record: minor refactoring of dorecord It prepares the way for introducing the flag to reverse hunk selection

record: add tests for the curses recording interface

record: enable curses recording logic with experimental flag

record: change arguments of curses recording function This way the arguments are the same for both curses and non-curses versions.

record: change interface of the filtering function This way filtering functions accept chunks and return chunks

hgweb: replace 'ctx._repo' with 'ctx.repo()'

pvec: replace 'ctx._repo' with 'ctx.repo()'

scmutil: replace 'ctx._repo' with 'ctx.repo()'

templater: replace 'ctx._repo' with 'ctx.repo()'

largefiles: replace 'ctx._repo' with 'ctx.repo()'

obsolete: replace 'ctx._repo' with 'ctx.repo()'

fileset: replace 'ctx._repo' with 'ctx.repo()'

filectx: add a repo accessor This is similar to a07314472a80 in motivation. All contexts now have this method, so the rest of the 'ctx._repo' uses can be converted without worrying about what type of context it is.

run-tests: ignore ENOENT failures when removing old .err results When the same test runs in multiple threads and the previous run was a failure, the threads can race to delete the error output. This fixes that.

run-tests: stop storing start/stop times in a dict by test name This resolves the last breakage in run-tests that prevented me from running a single test many times in several threads in parallel. This will be useful for testing potential fixes to flaky tests.

run-tests: avoid running the same test instance concurrently There's a fair amount of mutable state stored on test case instances. That causes many weird failures if you try to do something like `run-tests.py -j16 --loop test-help.t`. The quick fix is this slightly weird test-reloading dance, which ensures that every time a test is executed it runs on a fresh instance of the TestCase subclass.

run-tests: add --runs-per-test flag This is useful when you're working with a flaky test and want to run it (for example) 500 times to see if it'll false-fail. This currently breaks if you use it with more than one thread, but I'm looking into that now.

convert: adjust progress bar for octopus merges (issue4169) For merges, we walk the files N-1 times, where N is the number of parents. This means that for an octopus merge with 3 parents and 2 changed files, we actually fetch 6 files. This corrects the progress output of the convert command when such commits are encountered.

test-convert-git: show insane progress bar with octopus merge (issue4169) This is done as a separate change from the fix so that it's obvious what is corrected by the bugfix.

manifest: have context use self.hasdir() A couple places in context currently use "x in self._dirs" to check for the existence of the directory, but this requires that all directories be loaded into a dict. Calling hasdir() instead puts the work on the the manifest to check for the existence of a directory in the most efficient manner.

manifest: add hasdir() to context This is a convenience method that calls to its manifest's hasdir(). There are parts of context that check to see if a directory exists, and this method will let implementations of manifest provide an optimal way to find a particular directory.

manifest: add manifestdict.hasdir() method Allows for alternative implementations of manifestdict to decide if a directory exists in whatever way is most optimal.

manifest: have context's dirs() call its manifest's dirs() This lets the context's dirs() method be agnostic towards any alternate manifest implementations.

manifest: add dirs() to manifestdict Manifests should have a method of accessing its own dirs, not just the context that references the manifest. This makes it easier for other optimized versions of manifests to compute their own dirs in the most efficient way.

formatter: convert None to json null It will be used by "annotate" command to represent the workingctx revision.

localrepo: extend "changeid in repo" to return True for workingctx revision This is necessary to implement "wc" symbol for workingctx, that will be used as follows: $ hg annotate -r wc FILE In principle, "rev in repo" should be True if "repo[rev]" can return a context object. But when it was implemented by ea3acaae25bb, lookup() had a long logic to map all sorts of changeids to nodes, and "None in repo" did crash because lookup() could not accept None. So I assume that the case of changeid=None was not considered. Now "None in repo" doesn't crash, it should be True for workingctx revision. Behavior of "changeid in repo": revision "null" existing rev None (workingctx) ---------- ------ ------------ ----------------- original* True True TypeError current True True False this patch True True True (*original: ea3acaae25bb)

merge with stable

crecord: fix another underbar