tags: extract .hgtags filenodes cache to a standalone file Resolution of .hgtags filenodes values has historically been a performance pain point for large repositories, where reading individual manifests can take over 100ms. Multiplied by hundreds or even thousands of heads and resolving .hgtags filenodes becomes a performance issue. This patch establishes a standalone cache file holding the .hgtags filenodes for each changeset. After this patch, the .hgtags filenode for any particular changeset should only have to be computed once during the lifetime of the repository. The introduced hgtagsfnodes1 cache file is modeled after the rev branch cache: the cache is effectively an array of entries consisting of a changeset fragment and the filenode for a revision. The file grows in proportion to the length of the repository (24 bytes per changeset) and is truncated when the repository is stripped. The file is not written unless tag info is requested and tags have changed since last time. This patch partially addresses issue4550. Future patches will split the "tags" cache file into per-filter files and will refactor the cache format to not capture the .hgtags fnodes, as these are now stored in the hgtagsfnodes1 cache. This patch is capable of standing alone. We should not have to wait on the tags cache filter split and format refactor for this patch to land.

extensions: extract partial application into a bind() function We use partial function application for wrapping existing Mercurial functions, and it's implemented separately each time. This patch extracts the partial application into a new bind() function that can be used on its own by extensions when appropriate. In particular, the evolve extension needs to wrap functions in the various bundle2 processing dictionaries, which the pre-existing methods don't support.

obsolete: experimental flag to get debug about obsmarkers exchange The obsolescence markers exchange is still experimental. We (developer) need more information about what is going on. I'm adding an experimental flag to add display the amount of data exchanged during bundle2 exchanges.

bundle2: add an 'idx' argument to the 'getbundle2partsgenerator' This argument let extensions control in what order bundle2 part are generated server side during a pull. This is useful to ensure the transaction is in a proper state before some actions or hooks happens.

bundle2: add an 'idx' argument to the 'b2partsgenerator' This argument let extensions control in what order bundle2 part are generated client side during a push. This is useful to ensure the transaction is in a proper state before some actions or hooks happens.

rpms: create missing builds dir if it doesn't exist

rpms: for packages with their own python, put it in /opt/python-hg This is more compliant with the FHS and Fedora packaging guidelines.

branchcache: stay silent if failing to read cache files The warning has in some cases incorrectly attributed unrelated problems to rbc. Instead, just do like the branch head cache does and stay quiet when reading fails. The cache will be missing the first time a repo is used. It is a normal situation and there is no reason to make a note of that.

merge: better debug messages before/after invoking external merge tool

subrepo: use vfs.walk instead of os.walk "dirpath" in the tuple yielded by "vfs.walk()" is relative one from the root of specified vfs, and absolute path in the warning message is composed by "vfs.join()". On the other hand, target file "f" exists in "dirpath", and "reljoin()" is needed to unlink "f" by "vfs.unlink()".

vfs: add walk To eliminate "path prefix" (= "the root of vfs") part from "dirpath" yielded by "os.walk()" correctly, "path prefix" should have "os.sep" at the end of own string, but it isn't easy to ensure it, because: - examination by "path.endswith(os.sep)" isn't portable Some problematic encodings use 0x5c (= "os.sep" on Windows) as the tail byte of some multi-byte characters. - "os.path.join(path, '')" isn't portable With Python 2.7.9, this invocation doesn't add "os.sep" at the end of UNC path (see issue4557 for detail). Python 2.7.9 changed also behavior of "os.path.normpath()" (see *) and "os.path.splitdrive()" for UNC path. vfs root normpath splitdrive os.sep required =============== ============== =================== ============ z:\ z:\ z: + \ no z:\foo z:\foo z: + \foo yes z:\foo\ z:\foo z: + \foo yes [before Python 2.7.9] \\foo\bar \\foo\bar '' + \\foo\bar yes \\foo\bar\ \\foo\bar (*) '' + \\foo\bar yes \\foo\bar\baz \\foo\bar\baz '' + \\foo\bar\baz yes \\foo\bar\baz\ \\foo\bar\baz '' + \\foo\bar\baz yes [Python 2.7.9] \\foo\bar \\foo\bar \\foo\bar + '' yes \\foo\bar\ \\foo\bar\ (*) \\foo\bar + \ no \\foo\bar\baz \\foo\bar\baz \\foo\bar + \baz yes \\foo\bar\baz\ \\foo\bar\baz \\foo\bar + \baz yes If it is ensured that "normpath()"-ed vfs root is passed to "splitdrive()", adding "os.sep" is required only when "path" part of "splitdrive()" result isn't "os.sep" itself. This is just what "pathutil.nameasprefix()" examines. This patch applies "os.path.normpath()" on "self.join(None)" explicitly, because it isn't ensured that vfs root is already normalized: vfs itself is constructed with "realpath=False" (= avoid normalizing in "vfs.__init__()") in many code paths. This normalization should be much cheaper than subsequent file I/O for directory traversal.

subrepo: pass wvfs to _sanitize instead of absolute path to a subrepository As a preparation for vfs migration of "_sanitize()", this patch passes "wvfs" to "_sanitize()" and use "wvfs.base" instead of absolute path to a subrepository.

repoview: move function for computing filtered hash An upcoming patch will establish per-filter tags caches. We'll want to use the same cache validation logic as the branch cache. Prepare for that by moving the logic for computing a filtered view hash to somewhere central.

revset: don't import discovery at module level discovery.py imports a lot of the world. Pierre-Yves told me to move it to a function-level import to avoid an import cycle in a future patch.

transaction: add missing newline to message Add a missing newline to the "journal was created by a different version of Mercurial" message.

cat: disable optimization of single file case for workingctx It crashes because workingctx has no manifest.

committablectx: override manifestnode() to return None wctx.manifestnode() crashed before because it has no _changeset. Instead of crashing, just return None like wctx.node().

cat: use ctx.manifestnode() in place of ctx._changeset[0]

changegroup: flush the ui stdio buffers after adding a changegroup This eliminates the following test failure on Windows, as well as a similar one in evolve's test-wireproto.t. See the previous patch for details on the problem. --- e:/Projects/hg/tests/test-init.t +++ e:/Projects/hg/tests/test-init.t.err @@ -216,10 +216,10 @@ * test 0:08b9e9f63b32 $ hg clone -e "python \"$TESTDIR/dummyssh\"" local ssh://user@dummy/remote-bookmarks searching for changes + exporting bookmark test remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 1 changes to 1 files - exporting bookmark test $ hg -R remote-bookmarks bookmarks test 0:08b9e9f63b32

hook: forcibly flush stderr for Windows test stability There are a handful of SSH related test failures on Windows. --- c:/Users/Matt/Projects/hg/tests/test-bundle2-exchange.t +++ c:/Users/Matt/Projects/hg/tests/test-bundle2-exchange.t.err @@ -305,16 +305,16 @@ remote: added 1 changesets with 1 changes to 1 files remote: 1 new obsolescence markers updating bookmark book_5fdd + pre-close-tip:02de42196ebe draft book_02de + postclose-tip:02de42196ebe draft book_02de + txnclose hook: HG_SOURCE=push-response HG_TXNNAME=push-response + ssh://user@dummy/other HG_URL=ssh://user@dummy/other remote: pre-close-tip:5fddd98957c8 draft book_5fdd remote: pushkey: lock state after "bookmarks" remote: lock: free remote: wlock: free remote: postclose-tip:5fddd98957c8 draft book_5fdd remote: txnclose hook: (env vars truncated) - pre-close-tip:02de42196ebe draft book_02de - postclose-tip:02de42196ebe draft book_02de - txnclose hook: HG_SOURCE=push-response HG_TXNNAME=push-response - ssh://user@dummy/other HG_URL=ssh://user@dummy/other $ hg -R other log -G o 6:5fddd98957c8 draft Nicolas Dumazet <...> book_5fdd C | --- c:/Users/Matt/Projects/hg/tests/test-ssh.t +++ c:/Users/Matt/Projects/hg/tests/test-ssh.t.err @@ -438,12 +438,12 @@ $ hg push pushing to ssh://user@dummy/remote searching for changes + local stdout remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 1 changes to 1 files - remote: KABOOM - local stdout + remote: KABOOM\r (esc) $ cd .. What is happening is that no data is available in 'sshpeer.pipee' while the command is executing. As the command completes, local output is printed, and then sshpeer.cleanup() is called. When it calls 'self.pipeo.close()', the child process is shutdown, flushing stderr. As an experiment, I printed a line to stdout and another to stderr instead this flush(). The stdout data was immediately available to the hg client, and none of the stderr data was until the child exited. At that point, pipee has all of the buffered data, and it is read out and printed before the pipe is closed in sshpeer.cleanup(). This is probably a known issue, since ui.write_err() mentions that stderr may be buffered, and also flushes stderr. It would be nice if there was a more general fix (there is one more test that fails), but I'm not sure what it is. I've seen (ancient) references [1] to setvbuf() "crashing spectacularly" on some systems if any I/O has been done already, so it seems worth avoiding. https://sourceware.org/ml/gdb-patches/2013-08/msg00422.html [1] https://groups.google.com/forum/#!msg/comp.lang.python/JT8LiYzYDEY/Qg9d1HwyjScJ

json: implement {fileannotate} template

json: implement {comparison} template Similar to {filediff}, we abbreviate some property names to cut down on string bloat.

json: implement {filediff} template Single letter properties are used to keep payload size down, as diff representation can be quite large and longer property names can create a lot of extra work for parsers. Rename is not yet captured. This can be done in a follow-up.

hgweb: expose raw line numbers to templates Surpringly, the templates didn't receive an unmodified version of the line numbers. Expose it to make implementing the JSON templates easier. In theory, we could post-process an existing template variable. But extra string manipulation seems quite wasteful, especially on items that could occur hundreds or even thousands of times in output.

revert: stop marking files clean after interactive revert (issue4592) The goal of 'hg revert --interactive' is usually to keep some change in the revert file, so the files -must-not- be marked as clean. We want the status logic to do its usual job here. For unclear reasons (probably timing related), I was unable to build an automated test that reproduced issue4592 but manual testing shows this is fixed.

lazymanifest: prevent leak when updating an entry more than once __setitem__ on the lazymanifest C type wasn't checking to see if a line had previously been malloced before replacing it, leading to leaks if files got updated multiple times in the course of a task. I was able to reproduce the leak with this change to test-manifest.py: diff --git a/tests/test-manifest.py b/tests/test-manifest.py --- a/tests/test-manifest.py +++ b/tests/test-manifest.py @@ -456,6 +456,16 @@ class basemanifesttests(object): ['a/b/c/bar.txt', 'a/b/c/foo.txt', 'a/b/d/ten.txt'], m2.keys()) + def testManifestSetItem(self): + m = self.parsemanifest('') + for x in range(3): + m['file%d' % x] = BIN_HASH_1 + for x in range(3): + m['file%d' % x] = BIN_HASH_2 + import time + time.sleep(4) + + along with the commands: $ make local $ PYTHONPATH=. SILENT_BE_NOISY=1 python tests/test-manifest.py testmanifestdict.testManifestSetItem & $ sleep 4 $ leaks $(jobs -p | tee /dev/stderr | awk '{print $3}') $ wait in an interactive shell on OS X. As far as I can tell, it had to be an interactive shell so that I could get the pid of the test run using the jobs builtin. Prior to this change, I was leaking several strings, and after this change leaks reports no leaks. I thought there was a bug filed for this in bugzilla, but I can't find it either in bugzilla or by searching my email.

strip: properly clear resolve state with --keep (issue4593) Normal updates automatically clean up the resolve state, but strip --keep does a "manual" update that bypasses the normal machinery. This adds a mergestate reset.

revsets: more informative syntax error message I came across a case where an internal command was using a revset that I didn't immediately pass in and it was difficult to debug what was going wrong with the revset. This prints out the revset and informs the user that the error is with a rebset so it should be more obvious what and where the error is.

revsets: show current revset abort behavior

tests: move blackbox testing of tags to test-tags.t We're going to refactor tags cache shortly. It is easier to test the blackbox logging if these tests are in test-tags.t.

tests: move mock blackbox extension into own file Having all blackbox log testing in test-blackbox.t isn't scalable. Move the mock blackbox extension into its own file so we can start to move blackbox logging into other tests.

shelve: acquire lock in the right order Text book says that 'wlock' should be acquired before 'lock'. Caught through developer warning.

check-commit: be more picky about detection of wrong bug tag The check-commit script search for "bug" withing bracket and ask people to use (issueXXXX) instead. The test was too wide and matching any "(+b+u+g"sequence.

discovery: don't compute allfuturecommon when it won't be used In repos with many changesets, the computation of allfuturecommon can take a significant amount of time. Since it's only used if there's an obsstore, don't compute it otherwise.

treemanifest: separate flags for trees in memory and trees on disk When we start writing tree manifests with one manifest revlog per directory, it will still be nice to be able to run tests using tree manifests in memory but writing to a flat manifest to a single revlog. Let's break the current '_usetreemanifest' flag on the revlog into '_treeinmem' and '_treeondisk'. Both are populated from the same config, but after this change, one can temporarily hard-code _treeinmem=True to see that tests still pass.

manifestdict: drop empty-string argument when creating empty manifest manifestdict() creates an empty manifestdict, so let's consistently use that instead of explicitly parsing an empty string (which does result in an empty manifest).

lazymanifest: fix memory leak in lmiter_iterentriesnext() after 3d485727e45e

revert: do not requires '--all' if '--interative' is present The '--all' option have been introduced in 890e285c52a1 (August 2006), most probably to prevent user shooting themselves in the foot. As the record process will let you, view and select the set of files and change you want to revert, I feel like the '--all' flag is superfluous in the '--interactive' case.

bundle2: drop the experimental hooks The series at e9ede9b4c2f8::8e13cc0825f1 introduced generic transaction level hooking. This makes the experimental bundle2 specific hooks redundant, we drop them.

bundle2: advertise bundle2 by default That way, any new server will be ready to accept bundle2 payload. The decision for the client to use it is still off by default so this is not turning bundle2 everywhere. We introduce a new kill switch for this in case stuff goes wrong.

subrepo: convert the os.path references in git to vfs There are a handful of os.path references in the free functions at the top of the module that will be trickier to remove.

subrepo: use vfs.removedirs instead of os.removedirs This patch also removes useless composing absolute path by "repo.wjoin()".

vfs: add removedirs

util: add removedirs as platform depending function According to fa901423ac23 introducing "windows._removedirs()": If a hg repository including working directory is a reparse point (directory symlinked or a junction point), then using os.removedirs will remove the reparse point erroneously. "windows._removedirs()" should be used instead of "os.removedirs()" on Windows. This patch adds "removedirs" as platform depending function to replace "os.removedirs()" invocations for portability and safety

subrepo: use vfs.unlink instead of os.remove This patch also removes useless composing absolute path by "os.path.join()".

subrepo: use vfs.rmtree instead of shutil.rmtree This patch also removes useless "shutil" import.

vfs: add rmtree This duplicates "onerror()" function from "svnsubrepo.remove()" for equivalence of replacing in subsequent patch. This "onerror()" function for "shutil.rmtree()" was introduced by 92b0d669637f, which avoids failure of removing svn repository on Windows.

subrepo: use vfs.readdir instead of os.listdir to avoid expensive stat calls "kind" information given from "vfs.readdir()" makes expensive stat calls "os.path.isdir()" and "os.path.islink()" useless.

editor: prefer 'intermediate-source' extra to use for HGREVISION environment variable Revision 51930a7180bd ('graft: record intermediate grafts in extras') introduced 'intermediate-source' extra which refers to the closest graft source. As 'intermediate-source' extra provides more detailed information about the source changeset than 'source' one, it is better to prefer the first one to use as a value of HGREVISION environment variable for an editor.

bundle2: rename format, parts and config to final names It is finally time to freeze the bundle2 format! To do so we: - rename HG2Y to HG20, - drop "b2x:" prefix from all part names, - rename capability to "bundle2-exp" to "bundle2" - rename the hook flag from 'bundle2-exp' to 'bundle2'

manifestdict: extract condition for _intersectfiles() and use for walk() The condition on which manifestdict.matches() and manifestdict.walk() take the fast path of iterating over files instead of the manifest, is slightly different. Specifically, walk() does not take the fast path for exact matchers and it does not avoid taking the fast path when there are more than 100 files. Let's extract the condition so we don't have to maintain it in two places and so walk() can gain these two missing pieces of the condition (although there seems to be no current caller of walk() with an exact matcher).

manifestdict.walk: remove now-redundant check for match.files() When checking whether we can take the fast path of iterating over matcher files instead of manifest files, we check whether match.files() is non-empty. However, now that return early for match.always(), it can only be empty when there are only include/exclude patterns, but in that case anypats() will be True, so it's already covered. This makes manifestdict.walk() more similar to manifestdict.matches().

manifest.walk: special-case match.always() for speed This cuts down the run time of hg files -r . > /dev/null from ~0.850s to ~0.780s on the Firefox repo. Note that manifest.matches() already has the corresponding optimization.

manifest.walk: use return instead of StopIteration in generator Using "return" within a generator is supposedly more Pythonic than raising StopIteration.

archive: look for first visible revision to build repo identity (issue4591) No test for the case where all revisions are hidden because "archive" command aborts if the target revision is null.

largefiles: use common function to build content of .hg_archival.txt This fixes the missing "changessincelatesttag" field introduced by 242d11819c6c.