lfs: verify lfs object content when transferring to and from the remote store This avoids inserting corrupt files into the usercache, and local and remote stores. One down side is that the bad file won't be available locally for forensic purposes after a remote download. I'm thinking about adding an 'incoming' directory to the local lfs store to handle the download, and then move it to the 'objects' directory after it passes verification. That would have the additional benefit of not concatenating each transfer chunk in memory until the full file is transferred. Verification isn't needed when the data is passed back through the revlog interface or when the oid was just calculated, but otherwise it is on by default. The additional overhead should be well worth avoiding problems with file based remote stores, or buggy lfs servers. Having two different verify functions is a little sad, but the full data of the blob is mostly passed around in memory, because that's what the revlog interface wants. The upload function, however, chunks up the data. It would be ideal if that was how the content is always handled, but that's probably a huge project. I don't really like printing the long hash, but `hg debugdata` isn't a public interface, and is the only way to get it. The filelog and revision info is nowhere near this area, so recommending `hg verify` is the easiest thing to do.

lfs: narrow the exceptions that trigger a transfer retry The retries were added to workaround TCP RESETs in fb-experimental fc8c131314a9. I have no idea if that's been debugged yet, but this wide net caught local I/O errors, bad hostnames and other things that shouldn't be retried. The next patch will validate objects as they are uploaded, and there's no need to retry those errors. The spec[1] does mention that certain http errors can be retried, including 500. But let's work through the corruption detection issues first. [1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md

test-lfs: add tests around corrupted lfs objects These are mostly tests against file:// based remote stores, because that's what we have the most control over. The test uploading a corrupt blob to lfs-test-server demonstrates an overly broad exception handler in the retry loop. A corrupt blob is actually transferred in a download, but eventually caught when it is accessed (only after it leaves the corrupt file in a couple places locally). I don't think we want to trust random 3rd party implementations, and this would be a problem if there were a `debuglfsdownload` command that simply cached the files. And given the cryptic errors, we should probably validate the file hash locally before uploading, and also after downloading.

lfs: add note messages indicating what store holds the lfs blob The following corruption related patches were written prior to adding the user level cache, and it took awhile to track down why the tests changed. (It generally made things more resilient.) But I think this will be useful to the end user as well. I didn't make it --debug level, because there can be a ton of info coming out of clone/push/pull --debug. The pointers are sorted for test stability. I opted for ui.note() instead of checking ui.verbose and then using ui.write() for convenience, but I see most of this extension does the latter. I have no idea what the preferred form is.

tests: teach `f` to handle sha256 checksums

tests: fix a bug in `f` that prevented calculating sha1sum on a file

templater: look up symbols/resources as if they were separated (issue5699) It wouldn't be easy to split the mapping dict into (symbols, resources). This patch instead rejects invalid lookup taking resources.keys() as source of truth. The doctest is updated since mapping['repo'] is now reserved for a repo object.

templater: move repo, ui and cache to per-engine resources

templater: keep default resources per template engine (API) This allows us to register a repo object as a resource in hgweb template, without loosing '{repo}' symbol: symbol('repo') -> mapping['repo'] (n/a) -> defaults['repo'] resource('repo') -> mapping['repo'] (n/a) -> resources['repo'] I'm thinking of redesigning the templatekw API to take (context, mapping) in place of **(context._resources + mapping), but that will be a big change and not implemented yet. props['templ'] is ported to the resources dict as an example. .. api:: mapping does not contain all template resources. use context.resource() in template functions.

templater: look up mapping table through template engine These functions are stub for symbol/resource separation. This series is intended to address the following problems: a) internal data may be exposed to user (issue5699) b) defaults['repo'] (a repository name) will conflict with mapping['repo'] (a repo object) in hgweb

debug: add newlines at the end of three locations that appear to need it Differential Revision: https://phab.mercurial-scm.org/D1720

debug: remove an 'if ui.debug()' that is not doing anything ui.debug() does not return a value. Differential Revision: https://phab.mercurial-scm.org/D1719

paper: minor adjustments to table styles Adding a bit of padding to table columns on e.g. /log means content and headers are better aligned: headers already have this padding. Right margin is removed from #changesetEntry th because elements with display: table-cell (such as <th>) ignore margins anyway.

filemerge: only raise InMemoryMergeConflictsError when running _xmerge The old code here was overly broad and would raise in cases when we didn't end up calling `xmerge` and resolved using an internal tool (such as when `premerge=True`). Instead, let's swap out _xmerge if IMM is enabled and have the new tool raise when called, which is the behavior we want. Differential Revision: https://phab.mercurial-scm.org/D1739

journal: use pager journal output is long and should use a pager. Differential Revision: https://phab.mercurial-scm.org/D1740

commandserver: unblock SIGCHLD This enables the SIGCHLD handler to work properly if some buggy program started chg server with SIGCHLD blocked. A test of this probably requires C code, but we don't have such kind of tests already. Since this is a simple and clear fix, I'm leaving it as "untested" but I did a manual test and there were no longer zombie workers. Differential Revision: https://phab.mercurial-scm.org/D1737

osutil: add a function to unblock signals Signals could be blocked by something like: #include <unistd.h> #include <signal.h> int main(int argc, char * const argv[]) { sigset_t set; sigfillset(&set); sigprocmask(SIG_BLOCK, &set, NULL); execv("/bin/hg", argv); return 0; } One of the problems is if SIGCHLD is blocked, chgserver would not reap zombie workers since it depends on SIGCHLD handler entirely. While it's the parent process to blame but it seems a good idea to just unblock the signal from hg. FWIW git does that for SIGPIPE already [1]. Unfortunately Python 2 does not reset or provide APIs to change signal masks. Therefore let's add one in osutil. Note: Python 3.3 introduced `signal.pthread_sigmask` which solves the problem. `sigprocmask` is part of POSIX [2] so there is no feature testing in `setup.py`. [1]: https://github.com/git/git/commit/7559a1be8a0afb10df41d25e4cf4c5285a5faef1 [2]: http://pubs.opengroup.org/onlinepubs/7908799/xsh/sigprocmask.html Differential Revision: https://phab.mercurial-scm.org/D1736

sshpeer: move docstring to top Also makes it use double quotes consistently.

log: make "slowpath" condition slightly more readable Before 8e0e334bad42 and 6c76c42a5893, the condition was "anypats() or (files() and --removed)". This can be read as "<match is actually slow> or <walk files including removed revs>". So "not always()" (i.e. walk file revs) seems more appropriate here. The logic should be unchanged: not anypats() => always() or isexact() or prefix() isexact() => not always() prefix() => not always()

completion: add support for new "amend" command The command is now shipped with Mercurial, but completion should be helpful (and accurate) for users of the amend command shipped with the evolve extension too. Differential Revision: https://phab.mercurial-scm.org/D1716

completion: don't suggest clean files to revert It looks like we used to suggest only modified, added, removed and deleted files to revert until a821ec835223 (completion: selectively use debugpathcomplete in bash_completion, 2013-03-21). The reasoning in that commit was that getting the status was too slow and the replacement (debugpathcomplete) seems to make sense for the other two commands (remove and forget), but I'm not sure it was intentional to change the behavior of completion for revert. Note that "add" and "diff" already use status-based completion. Differential Revision: https://phab.mercurial-scm.org/D1715

split: new extension to split changesets This diff introduces an experimental split extension to split changesets. The implementation is largely inspired by Laurent Charignon's implementation for mutable-history (changeset 9603aa1ecdfd54b0d86e262318a72e0a2ffeb6cc [1]) This version contains various improvements: - Rebase by default. This is more friendly for new users. Split won't lead to merge conflicts so a rebase won't give the user more trouble. This has been on by default at Facebook for months now and seems to be a good UX improvement. The rebase skips obsoleted or orphaned changesets, which can avoid issues like allowdivergence, merge conflicts, etc. This is more flexible because the user can decide what to do next (see the last test case in test-split.t) - Remove "Done split? [y/n]" prompt. That could be detected by checking `repo.status()` instead. - Works with obsstore disabled. Without obsstore, split uses strip to clean up old nodes, and it can even handle split a non-head changeset with "allowunstable" disabled, since it runs a rebase to solve the "unstable" issue in a same transaction. - More friendly editor text. Put what has been already split into the editor text so users won't lost track about where they are. [1]: https://bitbucket.org/marmoute/mutable-history/commits/9603aa1ecdfd54b Differential Revision: https://phab.mercurial-scm.org/D1082

merge with stable

worker: handle interrupt on windows After applying suggestions from https://phab.mercurial-scm.org/D1564 to catch all exceptions in the same way I actually broke the handling of KeyboardInterrupt on windows. The reason is that KeyboardInterrupt doesn't dervie from Exception, but BaseException: https://docs.python.org/2/library/exceptions.html starting from python 2.5 Test Plan: Run hg on windows and ctrl-c during a large update. No random exceptions from threads surface in the shell. Previously we'd nearly always get stack traces from some of threads Run tests ./run-tests.py [...] Failed test-convert-svn-encoding.t: output changed # Ran 622 tests, 41 skipped, 1 failed. python hash seed: 2962682116 The test failing seems to have nothing to do with the change and fails on base revision as well Differential Revision: https://phab.mercurial-scm.org/D1718