lfs: only hardlink between the usercache and local store if the blob verifies This fixes the issue where verify (and other read commands) would propagate corrupt blobs. I originalled coded this to only hardlink if 'verify=True' for store.read(), but then good blobs weren't being linked, and this broke a bunch of tests. (The blob in repo5 that is being corrupted seems to be linked into repo5 in the loop running dumpflog.py prior to it being corrupted, but only if verify=False is handled too.) It's probably better to do a one time extra verification in order to create these files, so that the repo can be copied to a removable drive. Adding the same check to store.write() was only for completeness, but also needs to do a one time extra verification to avoid breaking tests.

lfs: verify lfs object content when transferring to and from the remote store This avoids inserting corrupt files into the usercache, and local and remote stores. One down side is that the bad file won't be available locally for forensic purposes after a remote download. I'm thinking about adding an 'incoming' directory to the local lfs store to handle the download, and then move it to the 'objects' directory after it passes verification. That would have the additional benefit of not concatenating each transfer chunk in memory until the full file is transferred. Verification isn't needed when the data is passed back through the revlog interface or when the oid was just calculated, but otherwise it is on by default. The additional overhead should be well worth avoiding problems with file based remote stores, or buggy lfs servers. Having two different verify functions is a little sad, but the full data of the blob is mostly passed around in memory, because that's what the revlog interface wants. The upload function, however, chunks up the data. It would be ideal if that was how the content is always handled, but that's probably a huge project. I don't really like printing the long hash, but `hg debugdata` isn't a public interface, and is the only way to get it. The filelog and revision info is nowhere near this area, so recommending `hg verify` is the easiest thing to do.

lfs: narrow the exceptions that trigger a transfer retry The retries were added to workaround TCP RESETs in fb-experimental fc8c131314a9. I have no idea if that's been debugged yet, but this wide net caught local I/O errors, bad hostnames and other things that shouldn't be retried. The next patch will validate objects as they are uploaded, and there's no need to retry those errors. The spec[1] does mention that certain http errors can be retried, including 500. But let's work through the corruption detection issues first. [1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md

test-lfs: add tests around corrupted lfs objects These are mostly tests against file:// based remote stores, because that's what we have the most control over. The test uploading a corrupt blob to lfs-test-server demonstrates an overly broad exception handler in the retry loop. A corrupt blob is actually transferred in a download, but eventually caught when it is accessed (only after it leaves the corrupt file in a couple places locally). I don't think we want to trust random 3rd party implementations, and this would be a problem if there were a `debuglfsdownload` command that simply cached the files. And given the cryptic errors, we should probably validate the file hash locally before uploading, and also after downloading.

lfs: add note messages indicating what store holds the lfs blob The following corruption related patches were written prior to adding the user level cache, and it took awhile to track down why the tests changed. (It generally made things more resilient.) But I think this will be useful to the end user as well. I didn't make it --debug level, because there can be a ton of info coming out of clone/push/pull --debug. The pointers are sorted for test stability. I opted for ui.note() instead of checking ui.verbose and then using ui.write() for convenience, but I see most of this extension does the latter. I have no idea what the preferred form is.

tests: teach `f` to handle sha256 checksums

tests: fix a bug in `f` that prevented calculating sha1sum on a file

templater: look up symbols/resources as if they were separated (issue5699) It wouldn't be easy to split the mapping dict into (symbols, resources). This patch instead rejects invalid lookup taking resources.keys() as source of truth. The doctest is updated since mapping['repo'] is now reserved for a repo object.

templater: move repo, ui and cache to per-engine resources

templater: keep default resources per template engine (API) This allows us to register a repo object as a resource in hgweb template, without loosing '{repo}' symbol: symbol('repo') -> mapping['repo'] (n/a) -> defaults['repo'] resource('repo') -> mapping['repo'] (n/a) -> resources['repo'] I'm thinking of redesigning the templatekw API to take (context, mapping) in place of **(context._resources + mapping), but that will be a big change and not implemented yet. props['templ'] is ported to the resources dict as an example. .. api:: mapping does not contain all template resources. use context.resource() in template functions.

templater: look up mapping table through template engine These functions are stub for symbol/resource separation. This series is intended to address the following problems: a) internal data may be exposed to user (issue5699) b) defaults['repo'] (a repository name) will conflict with mapping['repo'] (a repo object) in hgweb

debug: add newlines at the end of three locations that appear to need it Differential Revision: https://phab.mercurial-scm.org/D1720

debug: remove an 'if ui.debug()' that is not doing anything ui.debug() does not return a value. Differential Revision: https://phab.mercurial-scm.org/D1719

paper: minor adjustments to table styles Adding a bit of padding to table columns on e.g. /log means content and headers are better aligned: headers already have this padding. Right margin is removed from #changesetEntry th because elements with display: table-cell (such as <th>) ignore margins anyway.

filemerge: only raise InMemoryMergeConflictsError when running _xmerge The old code here was overly broad and would raise in cases when we didn't end up calling `xmerge` and resolved using an internal tool (such as when `premerge=True`). Instead, let's swap out _xmerge if IMM is enabled and have the new tool raise when called, which is the behavior we want. Differential Revision: https://phab.mercurial-scm.org/D1739

journal: use pager journal output is long and should use a pager. Differential Revision: https://phab.mercurial-scm.org/D1740

commandserver: unblock SIGCHLD This enables the SIGCHLD handler to work properly if some buggy program started chg server with SIGCHLD blocked. A test of this probably requires C code, but we don't have such kind of tests already. Since this is a simple and clear fix, I'm leaving it as "untested" but I did a manual test and there were no longer zombie workers. Differential Revision: https://phab.mercurial-scm.org/D1737

osutil: add a function to unblock signals Signals could be blocked by something like: #include <unistd.h> #include <signal.h> int main(int argc, char * const argv[]) { sigset_t set; sigfillset(&set); sigprocmask(SIG_BLOCK, &set, NULL); execv("/bin/hg", argv); return 0; } One of the problems is if SIGCHLD is blocked, chgserver would not reap zombie workers since it depends on SIGCHLD handler entirely. While it's the parent process to blame but it seems a good idea to just unblock the signal from hg. FWIW git does that for SIGPIPE already [1]. Unfortunately Python 2 does not reset or provide APIs to change signal masks. Therefore let's add one in osutil. Note: Python 3.3 introduced `signal.pthread_sigmask` which solves the problem. `sigprocmask` is part of POSIX [2] so there is no feature testing in `setup.py`. [1]: https://github.com/git/git/commit/7559a1be8a0afb10df41d25e4cf4c5285a5faef1 [2]: http://pubs.opengroup.org/onlinepubs/7908799/xsh/sigprocmask.html Differential Revision: https://phab.mercurial-scm.org/D1736

sshpeer: move docstring to top Also makes it use double quotes consistently.

log: make "slowpath" condition slightly more readable Before 8e0e334bad42 and 6c76c42a5893, the condition was "anypats() or (files() and --removed)". This can be read as "<match is actually slow> or <walk files including removed revs>". So "not always()" (i.e. walk file revs) seems more appropriate here. The logic should be unchanged: not anypats() => always() or isexact() or prefix() isexact() => not always() prefix() => not always()

completion: add support for new "amend" command The command is now shipped with Mercurial, but completion should be helpful (and accurate) for users of the amend command shipped with the evolve extension too. Differential Revision: https://phab.mercurial-scm.org/D1716

completion: don't suggest clean files to revert It looks like we used to suggest only modified, added, removed and deleted files to revert until a821ec835223 (completion: selectively use debugpathcomplete in bash_completion, 2013-03-21). The reasoning in that commit was that getting the status was too slow and the replacement (debugpathcomplete) seems to make sense for the other two commands (remove and forget), but I'm not sure it was intentional to change the behavior of completion for revert. Note that "add" and "diff" already use status-based completion. Differential Revision: https://phab.mercurial-scm.org/D1715

split: new extension to split changesets This diff introduces an experimental split extension to split changesets. The implementation is largely inspired by Laurent Charignon's implementation for mutable-history (changeset 9603aa1ecdfd54b0d86e262318a72e0a2ffeb6cc [1]) This version contains various improvements: - Rebase by default. This is more friendly for new users. Split won't lead to merge conflicts so a rebase won't give the user more trouble. This has been on by default at Facebook for months now and seems to be a good UX improvement. The rebase skips obsoleted or orphaned changesets, which can avoid issues like allowdivergence, merge conflicts, etc. This is more flexible because the user can decide what to do next (see the last test case in test-split.t) - Remove "Done split? [y/n]" prompt. That could be detected by checking `repo.status()` instead. - Works with obsstore disabled. Without obsstore, split uses strip to clean up old nodes, and it can even handle split a non-head changeset with "allowunstable" disabled, since it runs a rebase to solve the "unstable" issue in a same transaction. - More friendly editor text. Put what has been already split into the editor text so users won't lost track about where they are. [1]: https://bitbucket.org/marmoute/mutable-history/commits/9603aa1ecdfd54b Differential Revision: https://phab.mercurial-scm.org/D1082

merge with stable

worker: handle interrupt on windows After applying suggestions from https://phab.mercurial-scm.org/D1564 to catch all exceptions in the same way I actually broke the handling of KeyboardInterrupt on windows. The reason is that KeyboardInterrupt doesn't dervie from Exception, but BaseException: https://docs.python.org/2/library/exceptions.html starting from python 2.5 Test Plan: Run hg on windows and ctrl-c during a large update. No random exceptions from threads surface in the shell. Previously we'd nearly always get stack traces from some of threads Run tests ./run-tests.py [...] Failed test-convert-svn-encoding.t: output changed # Ran 622 tests, 41 skipped, 1 failed. python hash seed: 2962682116 The test failing seems to have nothing to do with the change and fails on base revision as well Differential Revision: https://phab.mercurial-scm.org/D1718

lfs: fix committing deleted files caused by e0a1b9ee93cd e0a1b9ee93cd (lfs: add a repo requirement for this extension once an lfs file is committed) introduced a regression that prevents committing file deletion. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D1717

tests: update expected output of svn encoding test The changes I see on the buildbot match the ones I see on my laptop, and all look reasonable. Differential Revision: https://phab.mercurial-scm.org/D1713

run-tests: use context managers for file descriptors I've seen the following error a few times recently when running the tests with `yes | ./run-tests.py --local -j9 -i`: Errored test-add.t: Traceback (most recent call last): File "./run-tests.py", line 821, in run self.runTest() File "./run-tests.py", line 910, in runTest if self._result.addOutputMismatch(self, ret, out, self._refout): File "./run-tests.py", line 1774, in addOutputMismatch rename(test.errpath, test.path) File "./run-tests.py", line 571, in rename os.remove(src) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\Users\\Matt\\projects\\hg\\tests\\test-add.t.err' This change doesn't fix the problem, but it seems like a simple enough improvement.

run-tests: add substitution patterns for common '\' path output on Windows The goal is to reduce the amount of hand tuning of new/changed tests that is required on Windows. Since the OS prints the proper paths everywhere else, this is limited to Windows. These are based on the check-code rules that were dropped in 5feb782c7a95. There are some minor tweaks, because those were trying to detect '/' paths without a '(glob)' at the end, whereas these detect '\' paths. Also, it looks like the 'no changes made to subrepo' one was broke, because the path to the subrepo has been getting output but was not in the pattern. End anchors are dropped because '(glob)' is no longer required, but '(feature !)' annotations are a possibility. The 'saved backup bundle' pattern dropped from run-tests.py was simply carrying over the first capture group. The replace() method runs prior to evaluating '\1', but it wasn't doing anything because of the 'r' prefix on '\\'. The 'not recording move' entry is new, because I stumbled upon it searching for some of these patterns. There are probably others.

debuginstall: add a line about re2 availability Using re2 engine can massively speed up regexp. We make it simpler to check if it is available in a given install.