branchcache: stay silent if failing to read cache files The warning has in some cases incorrectly attributed unrelated problems to rbc. Instead, just do like the branch head cache does and stay quiet when reading fails. The cache will be missing the first time a repo is used. It is a normal situation and there is no reason to make a note of that.

merge: better debug messages before/after invoking external merge tool

subrepo: use vfs.walk instead of os.walk "dirpath" in the tuple yielded by "vfs.walk()" is relative one from the root of specified vfs, and absolute path in the warning message is composed by "vfs.join()". On the other hand, target file "f" exists in "dirpath", and "reljoin()" is needed to unlink "f" by "vfs.unlink()".

vfs: add walk To eliminate "path prefix" (= "the root of vfs") part from "dirpath" yielded by "os.walk()" correctly, "path prefix" should have "os.sep" at the end of own string, but it isn't easy to ensure it, because: - examination by "path.endswith(os.sep)" isn't portable Some problematic encodings use 0x5c (= "os.sep" on Windows) as the tail byte of some multi-byte characters. - "os.path.join(path, '')" isn't portable With Python 2.7.9, this invocation doesn't add "os.sep" at the end of UNC path (see issue4557 for detail). Python 2.7.9 changed also behavior of "os.path.normpath()" (see *) and "os.path.splitdrive()" for UNC path. vfs root normpath splitdrive os.sep required =============== ============== =================== ============ z:\ z:\ z: + \ no z:\foo z:\foo z: + \foo yes z:\foo\ z:\foo z: + \foo yes [before Python 2.7.9] \\foo\bar \\foo\bar '' + \\foo\bar yes \\foo\bar\ \\foo\bar (*) '' + \\foo\bar yes \\foo\bar\baz \\foo\bar\baz '' + \\foo\bar\baz yes \\foo\bar\baz\ \\foo\bar\baz '' + \\foo\bar\baz yes [Python 2.7.9] \\foo\bar \\foo\bar \\foo\bar + '' yes \\foo\bar\ \\foo\bar\ (*) \\foo\bar + \ no \\foo\bar\baz \\foo\bar\baz \\foo\bar + \baz yes \\foo\bar\baz\ \\foo\bar\baz \\foo\bar + \baz yes If it is ensured that "normpath()"-ed vfs root is passed to "splitdrive()", adding "os.sep" is required only when "path" part of "splitdrive()" result isn't "os.sep" itself. This is just what "pathutil.nameasprefix()" examines. This patch applies "os.path.normpath()" on "self.join(None)" explicitly, because it isn't ensured that vfs root is already normalized: vfs itself is constructed with "realpath=False" (= avoid normalizing in "vfs.__init__()") in many code paths. This normalization should be much cheaper than subsequent file I/O for directory traversal.

subrepo: pass wvfs to _sanitize instead of absolute path to a subrepository As a preparation for vfs migration of "_sanitize()", this patch passes "wvfs" to "_sanitize()" and use "wvfs.base" instead of absolute path to a subrepository.

repoview: move function for computing filtered hash An upcoming patch will establish per-filter tags caches. We'll want to use the same cache validation logic as the branch cache. Prepare for that by moving the logic for computing a filtered view hash to somewhere central.

revset: don't import discovery at module level discovery.py imports a lot of the world. Pierre-Yves told me to move it to a function-level import to avoid an import cycle in a future patch.

transaction: add missing newline to message Add a missing newline to the "journal was created by a different version of Mercurial" message.

cat: disable optimization of single file case for workingctx It crashes because workingctx has no manifest.

committablectx: override manifestnode() to return None wctx.manifestnode() crashed before because it has no _changeset. Instead of crashing, just return None like wctx.node().

cat: use ctx.manifestnode() in place of ctx._changeset[0]

changegroup: flush the ui stdio buffers after adding a changegroup This eliminates the following test failure on Windows, as well as a similar one in evolve's test-wireproto.t. See the previous patch for details on the problem. --- e:/Projects/hg/tests/test-init.t +++ e:/Projects/hg/tests/test-init.t.err @@ -216,10 +216,10 @@ * test 0:08b9e9f63b32 $ hg clone -e "python \"$TESTDIR/dummyssh\"" local ssh://user@dummy/remote-bookmarks searching for changes + exporting bookmark test remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 1 changes to 1 files - exporting bookmark test $ hg -R remote-bookmarks bookmarks test 0:08b9e9f63b32

hook: forcibly flush stderr for Windows test stability There are a handful of SSH related test failures on Windows. --- c:/Users/Matt/Projects/hg/tests/test-bundle2-exchange.t +++ c:/Users/Matt/Projects/hg/tests/test-bundle2-exchange.t.err @@ -305,16 +305,16 @@ remote: added 1 changesets with 1 changes to 1 files remote: 1 new obsolescence markers updating bookmark book_5fdd + pre-close-tip:02de42196ebe draft book_02de + postclose-tip:02de42196ebe draft book_02de + txnclose hook: HG_SOURCE=push-response HG_TXNNAME=push-response + ssh://user@dummy/other HG_URL=ssh://user@dummy/other remote: pre-close-tip:5fddd98957c8 draft book_5fdd remote: pushkey: lock state after "bookmarks" remote: lock: free remote: wlock: free remote: postclose-tip:5fddd98957c8 draft book_5fdd remote: txnclose hook: (env vars truncated) - pre-close-tip:02de42196ebe draft book_02de - postclose-tip:02de42196ebe draft book_02de - txnclose hook: HG_SOURCE=push-response HG_TXNNAME=push-response - ssh://user@dummy/other HG_URL=ssh://user@dummy/other $ hg -R other log -G o 6:5fddd98957c8 draft Nicolas Dumazet <...> book_5fdd C | --- c:/Users/Matt/Projects/hg/tests/test-ssh.t +++ c:/Users/Matt/Projects/hg/tests/test-ssh.t.err @@ -438,12 +438,12 @@ $ hg push pushing to ssh://user@dummy/remote searching for changes + local stdout remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 1 changes to 1 files - remote: KABOOM - local stdout + remote: KABOOM\r (esc) $ cd .. What is happening is that no data is available in 'sshpeer.pipee' while the command is executing. As the command completes, local output is printed, and then sshpeer.cleanup() is called. When it calls 'self.pipeo.close()', the child process is shutdown, flushing stderr. As an experiment, I printed a line to stdout and another to stderr instead this flush(). The stdout data was immediately available to the hg client, and none of the stderr data was until the child exited. At that point, pipee has all of the buffered data, and it is read out and printed before the pipe is closed in sshpeer.cleanup(). This is probably a known issue, since ui.write_err() mentions that stderr may be buffered, and also flushes stderr. It would be nice if there was a more general fix (there is one more test that fails), but I'm not sure what it is. I've seen (ancient) references [1] to setvbuf() "crashing spectacularly" on some systems if any I/O has been done already, so it seems worth avoiding. https://sourceware.org/ml/gdb-patches/2013-08/msg00422.html [1] https://groups.google.com/forum/#!msg/comp.lang.python/JT8LiYzYDEY/Qg9d1HwyjScJ

json: implement {fileannotate} template

json: implement {comparison} template Similar to {filediff}, we abbreviate some property names to cut down on string bloat.

json: implement {filediff} template Single letter properties are used to keep payload size down, as diff representation can be quite large and longer property names can create a lot of extra work for parsers. Rename is not yet captured. This can be done in a follow-up.

hgweb: expose raw line numbers to templates Surpringly, the templates didn't receive an unmodified version of the line numbers. Expose it to make implementing the JSON templates easier. In theory, we could post-process an existing template variable. But extra string manipulation seems quite wasteful, especially on items that could occur hundreds or even thousands of times in output.

revert: stop marking files clean after interactive revert (issue4592) The goal of 'hg revert --interactive' is usually to keep some change in the revert file, so the files -must-not- be marked as clean. We want the status logic to do its usual job here. For unclear reasons (probably timing related), I was unable to build an automated test that reproduced issue4592 but manual testing shows this is fixed.

lazymanifest: prevent leak when updating an entry more than once __setitem__ on the lazymanifest C type wasn't checking to see if a line had previously been malloced before replacing it, leading to leaks if files got updated multiple times in the course of a task. I was able to reproduce the leak with this change to test-manifest.py: diff --git a/tests/test-manifest.py b/tests/test-manifest.py --- a/tests/test-manifest.py +++ b/tests/test-manifest.py @@ -456,6 +456,16 @@ class basemanifesttests(object): ['a/b/c/bar.txt', 'a/b/c/foo.txt', 'a/b/d/ten.txt'], m2.keys()) + def testManifestSetItem(self): + m = self.parsemanifest('') + for x in range(3): + m['file%d' % x] = BIN_HASH_1 + for x in range(3): + m['file%d' % x] = BIN_HASH_2 + import time + time.sleep(4) + + along with the commands: $ make local $ PYTHONPATH=. SILENT_BE_NOISY=1 python tests/test-manifest.py testmanifestdict.testManifestSetItem & $ sleep 4 $ leaks $(jobs -p | tee /dev/stderr | awk '{print $3}') $ wait in an interactive shell on OS X. As far as I can tell, it had to be an interactive shell so that I could get the pid of the test run using the jobs builtin. Prior to this change, I was leaking several strings, and after this change leaks reports no leaks. I thought there was a bug filed for this in bugzilla, but I can't find it either in bugzilla or by searching my email.

strip: properly clear resolve state with --keep (issue4593) Normal updates automatically clean up the resolve state, but strip --keep does a "manual" update that bypasses the normal machinery. This adds a mergestate reset.

revsets: more informative syntax error message I came across a case where an internal command was using a revset that I didn't immediately pass in and it was difficult to debug what was going wrong with the revset. This prints out the revset and informs the user that the error is with a rebset so it should be more obvious what and where the error is.

revsets: show current revset abort behavior

tests: move blackbox testing of tags to test-tags.t We're going to refactor tags cache shortly. It is easier to test the blackbox logging if these tests are in test-tags.t.

tests: move mock blackbox extension into own file Having all blackbox log testing in test-blackbox.t isn't scalable. Move the mock blackbox extension into its own file so we can start to move blackbox logging into other tests.

shelve: acquire lock in the right order Text book says that 'wlock' should be acquired before 'lock'. Caught through developer warning.

check-commit: be more picky about detection of wrong bug tag The check-commit script search for "bug" withing bracket and ask people to use (issueXXXX) instead. The test was too wide and matching any "(+b+u+g"sequence.

discovery: don't compute allfuturecommon when it won't be used In repos with many changesets, the computation of allfuturecommon can take a significant amount of time. Since it's only used if there's an obsstore, don't compute it otherwise.

treemanifest: separate flags for trees in memory and trees on disk When we start writing tree manifests with one manifest revlog per directory, it will still be nice to be able to run tests using tree manifests in memory but writing to a flat manifest to a single revlog. Let's break the current '_usetreemanifest' flag on the revlog into '_treeinmem' and '_treeondisk'. Both are populated from the same config, but after this change, one can temporarily hard-code _treeinmem=True to see that tests still pass.

manifestdict: drop empty-string argument when creating empty manifest manifestdict() creates an empty manifestdict, so let's consistently use that instead of explicitly parsing an empty string (which does result in an empty manifest).

lazymanifest: fix memory leak in lmiter_iterentriesnext() after 3d485727e45e

revert: do not requires '--all' if '--interative' is present The '--all' option have been introduced in 890e285c52a1 (August 2006), most probably to prevent user shooting themselves in the foot. As the record process will let you, view and select the set of files and change you want to revert, I feel like the '--all' flag is superfluous in the '--interactive' case.

bundle2: drop the experimental hooks The series at e9ede9b4c2f8::8e13cc0825f1 introduced generic transaction level hooking. This makes the experimental bundle2 specific hooks redundant, we drop them.

bundle2: advertise bundle2 by default That way, any new server will be ready to accept bundle2 payload. The decision for the client to use it is still off by default so this is not turning bundle2 everywhere. We introduce a new kill switch for this in case stuff goes wrong.

subrepo: convert the os.path references in git to vfs There are a handful of os.path references in the free functions at the top of the module that will be trickier to remove.

subrepo: use vfs.removedirs instead of os.removedirs This patch also removes useless composing absolute path by "repo.wjoin()".

vfs: add removedirs

util: add removedirs as platform depending function According to fa901423ac23 introducing "windows._removedirs()": If a hg repository including working directory is a reparse point (directory symlinked or a junction point), then using os.removedirs will remove the reparse point erroneously. "windows._removedirs()" should be used instead of "os.removedirs()" on Windows. This patch adds "removedirs" as platform depending function to replace "os.removedirs()" invocations for portability and safety

subrepo: use vfs.unlink instead of os.remove This patch also removes useless composing absolute path by "os.path.join()".

subrepo: use vfs.rmtree instead of shutil.rmtree This patch also removes useless "shutil" import.

vfs: add rmtree This duplicates "onerror()" function from "svnsubrepo.remove()" for equivalence of replacing in subsequent patch. This "onerror()" function for "shutil.rmtree()" was introduced by 92b0d669637f, which avoids failure of removing svn repository on Windows.

subrepo: use vfs.readdir instead of os.listdir to avoid expensive stat calls "kind" information given from "vfs.readdir()" makes expensive stat calls "os.path.isdir()" and "os.path.islink()" useless.

editor: prefer 'intermediate-source' extra to use for HGREVISION environment variable Revision 51930a7180bd ('graft: record intermediate grafts in extras') introduced 'intermediate-source' extra which refers to the closest graft source. As 'intermediate-source' extra provides more detailed information about the source changeset than 'source' one, it is better to prefer the first one to use as a value of HGREVISION environment variable for an editor.

bundle2: rename format, parts and config to final names It is finally time to freeze the bundle2 format! To do so we: - rename HG2Y to HG20, - drop "b2x:" prefix from all part names, - rename capability to "bundle2-exp" to "bundle2" - rename the hook flag from 'bundle2-exp' to 'bundle2'

manifestdict: extract condition for _intersectfiles() and use for walk() The condition on which manifestdict.matches() and manifestdict.walk() take the fast path of iterating over files instead of the manifest, is slightly different. Specifically, walk() does not take the fast path for exact matchers and it does not avoid taking the fast path when there are more than 100 files. Let's extract the condition so we don't have to maintain it in two places and so walk() can gain these two missing pieces of the condition (although there seems to be no current caller of walk() with an exact matcher).

manifestdict.walk: remove now-redundant check for match.files() When checking whether we can take the fast path of iterating over matcher files instead of manifest files, we check whether match.files() is non-empty. However, now that return early for match.always(), it can only be empty when there are only include/exclude patterns, but in that case anypats() will be True, so it's already covered. This makes manifestdict.walk() more similar to manifestdict.matches().

manifest.walk: special-case match.always() for speed This cuts down the run time of hg files -r . > /dev/null from ~0.850s to ~0.780s on the Firefox repo. Note that manifest.matches() already has the corresponding optimization.

manifest.walk: use return instead of StopIteration in generator Using "return" within a generator is supposedly more Pythonic than raising StopIteration.

archive: look for first visible revision to build repo identity (issue4591) No test for the case where all revisions are hidden because "archive" command aborts if the target revision is null.

largefiles: use common function to build content of .hg_archival.txt This fixes the missing "changessincelatesttag" field introduced by 242d11819c6c.

test-largefiles: verify content of .hg_archival.txt

archive: extract metadata() closure to module-level function This function will be reused in largefiles.

archive: use ctx object consistently to build meta data

templatekw: have {manifest} use ctx.manifestnode() for consistency changeset_printer was updated at f57640bf10d4 to not access changeset by index.

subrepo: use vfs.reljoin instead of os.path.join

subrepo: inline reporelpath into abstractsubrepo._relpath to centralize logic "reporelpath()" is referred only from "abstractsubrepo._relpath()".

subrepo: add _relpath field to centralize subrelpath logic This patch adds propertycache-ed "_relpath" field to "abstractsubrepo", to centralize "subrelpath" logic into it. Now, "subrelpath()" can always return "_relpath" field of the specified subrepo object, because it is ensured that subrepo object has it. To reduce changes in this patch, "subrelpath()" itself is still kept, even though it seems to be redundant. This is also a part of eliminating "os.path.*" API invocations for "Windows UTF-8 Plan".

subrepo: add wvfs field to access the working directory via vfs This patch doesn't create vfs object in "abstractsubrepo.__init__()" but adds propertycache-ed "wvfs" field, because the latter can: - delay vfs instantiation until it is actually needed - allow to use "hgsubrepo._repo.wvfs" as "wvfs" "hgsubrepo._repo" is initialized after "abstractsubrepo.__init__()" invocation, and passing "hgsubrepo._repo.wvfs" to "abstractsubrepo.__init__()" is difficult.

subrepo: change arguments of abstractsubrepo.__init__ (API) This patch passes "ctx" and "path" instead of "ui" to "abstractsubrepo.__init__()" and stores them as "_ctx" and "_path" to use them in subsequent patches. This also removes redundant field initializations in the constructor of classes derived from "abstractsubrepo".

treemanifest: optimize treemanifest._walk() to skip directories This makes treemanifest.walk() not visit submanifests that are known not to have any matching files. It does this by calling match.visitdir() on submanifests as it walks. This change also updates largefiles to be able to work with this new behavior in treemanifests. It overrides match.visitdir(), the function that dictates how walk() and matches() skip over directories. The greatest speed improvements are seen with narrower scopes. For example, this commit speeds up the following command on the Mozilla repo from 1.14s to 1.02s: hg files -r . dom/apps/ Whereas with a wider scope, dom/, the speed only improves from 1.21s to 1.13s. As with similar a similar optimization to treemanifest.matches(), this change will bring out even bigger performance improvements once treemanifests are loaded lazily. Once that happens, we won't just skip over looking at submanifests, but we'll skip even loading them.

import-checker: force 'fcntl', 'grp', 'pwd', and 'termios' to stdlib modules These are Unix only, and caused these additional warnings on Windows if they aren't hardcoded as stdlib: mercurial/posix.py mixed imports stdlib: errno, getpass, os, socket, stat, sys, tempfile relative: grp, pwd, unicodedata mercurial/posix.py mixed imports stdlib: re relative: fcntl mercurial/posix.py mixed imports stdlib: array relative: termios Additionally, this was missing on Windows: mercurial/crecord.py mixed imports stdlib: fcntl, termios relative: curses https://docs.python.org/2/library/fcntl.html https://docs.python.org/2/library/grp.html https://docs.python.org/2/library/pwd.html https://docs.python.org/2/library/termios.html