verify: use similar language for missing manifest and file revisions When a changeset refers to a manifest revision that's not found in the manifest log, we say "changeset refers to missing revision X", but when a manifest refers to file revision that's not found in the filelog, we say "X in manifests not found". The language used for missing manifest revisions seems clearer, so let's use that for missing filelog revisions too.

verify: include "manifest" prefix in a few more places We include the "manifest" prefix on most other errors, so it seems consistent to add them to the remaining messages too. Also, having the "manifest" prefix will be more consistent with having the directory prefix there when we add support for treemanifests. With the "manifest" at the beginning, let's remove the now-redundant "manifest" in the message itself.

verify: drop unnecessary check for nullid In eb914541a950 (verify: filter messages about missing null manifests (issue2900), 2011-07-13), we started ignoring nullid in the list of manifest nodeids to check. Then, in b32a30da608d (verify: do not choke on valid changelog without manifest, 2012-08-21), we stopped adding nullid to the list to start with. So let's drop the left-over check now.

verify: move cross-checking of changeset/manifest out of _crosscheckfiles() Reasons: * _crosscheckfiles(), as the name suggests, is about checking that the set of files files mentioned in changesets match the set of files mentioned in the manifests. * The "checking" in _crosscheckfiles() looked rather strange, as it just emitted an error for *every* entry in mflinkrevs. The reason was that these were the entries remaining after the call to _verifymanifest(). Moving all the processing of mflinkrevs into _verifymanifest() makes it much clearer that it's the remaining entries that are a problem. Functional change: progress is no longer reported for "crosschecking" of missing manifest entries. Since the crosschecking phase takes a tiny fraction of the verification, I don't think this is a problem. Also, any reports of "changeset refers to unknown manifest" will now come before "crosschecking files in changesets and manifests".

tests: add tests for missing revlogs and revlog entries The verify code is pretty poorly tested. It's easy to test missing revlogs and missing revlog entries, so let's add tests for that. Also add some more tests corrupting each type of revlog, so we test the messages presented when reading a revision fails. The pure and native implementations produce different error messages, so we have to use (glob) in the tests.

hook: don't crash on syntax errors in python hooks We had some real-world cases where syntax errors in Python hooks would crash the whole process and leave it in an indeterminate state. Handle those better.

hook: for python hook exceptions, add note to run with --traceback Just like with ImportErrors, it isn't obvious that --traceback will produce helpful debugging output here.

hook: add tests for failing post- python hooks I couldn't find any tests for this.

hook: even fewer parentheses for load errors Missed this one.

destutil: document various failure cases We document what various conditional branch mean and clarify that they are exclusive (since they all end up in with exception raised).

destutil: consistently retrieve 'p1' and 'branch' We already read p1 from the dirstate so let's read the branch from it too.

merge: give priority to "not at head" failures for bare 'hg merge' We refuse to pick a destination for a bare 'hg merge' if the working copy is not at head. This is meant to prevent strange merge from user who forget to update. (Moreover, such merge does not reduce actually the number of heads) However, we were doing that as the last possible failure type. So user were recommended to merge with an explicit head (from this bad location) if the branch had too many heads. We now make "not on branch heads" class of failure the first things to check and fail on. The one test that change was actually trying to check for these failure (and did not). The new test output is correct.

destutil: extract all 'mergedest' abort messages into a dictionary We plan to be able to reuse this function for rebase. The error message explicitly refers to "merge" in multiple places. So we'll need to be able to use different messages. The first step of that is to extract all messages in a dedicated dictionary and use them indirectly. As a side effect it clarifies the actual function and opens the way to various cleanups and fixes in future changesets.

tests: add an explicit destination in some rebase tests As we will make 'rebase' behave more like 'merge', it will no longer pick 'max(branch(.))' as the default destination. We have to hard code the expected destination is multiple tests where it matters. After a careful inspection none of theses tests really cares about the default destination behavior and just omitted one out of laziness.

tests: remove third head in some of the 'rebase-parameters' tests These tests do not care about that extra branch at all. In future changeset we will make rebase behave like merge and abort in case of an ambiguous destination (eg: multiple other heads) and that extra branch will make the command invocation breaks. We preventively remove this extra branch from the relevant tests in an independant changeset to reduce noise and increase confidence in the final change.

run-tests: allow to specify executable of any name by --with-hg If the executable is not named as "hg", TTest runner inserts alias. This way, we can run tests with chg. But it is still warned because the alias does not always work. We do "$BINDIR"/hg in a few places.

run-tests: drop redundant assignment to BINDIR We do it a few lines after.

run-tests: cast --with-hg option to bytes consistently at parseargs() parseargs() sets bytes to options.with_hg if --local is specified, so do the same for --with-hg.

run-tests: do not compare bytes with str while ordering tests It failed on Python 3.

clonebundles: use absolute_import

churn: use absolute_import

children: use absolute_import

censor: use absolute_import

bugzilla: use absolute_import

blackbox: use absolute_import

acl: use absolute_import Continuing the march towards Python 3.

checkunknown: audit path before checking if it's a file or link Previously we would lstat the file to see if it was a file or a link before attempting to process it. If the file happened to exist across a symlink, and if that symlink was pointing to a network file system, that check could be very expensive. The new logic audit's the path to avoid symlinks before performing the lstat on the file itself. In our situation, this shaved 10 minutes off of certain hg updates. 300 files * (2 seconds - the network filesystem lookup time)

pathauditor: change parts verification order to be root first Previously, when we verified the parts of a path in the auditor, we would validate the deepest directory first, then it's parent, and so on up to the root. If there happened to be a symlink in the chain, that meant our first check would likely traverse that symlink. In some cases that symlink might point to a network filesystem that is expensive, and therefore this simple check could be very slow. The fix is to check the path parts starting at the root and working our way down. This has a minor performance difference in that we used to be able to short circuit from the audit if we reached a directory that had already been checked. Now we can't, but the cost is N dictionary look ups, where N is the number of parts in the path, which should be fairly minor.

chg: forward job control signals to worker process (issue5051) This is necessary to suspend/resume long pulls, interactive curses session, etc. The implementation is based on emacsclient, but our version doesn't test if chg process is foreground or not before propagating SIGCONT. This is because chg isn't always an interactive session. If we copy the SIGTTIN/SIGTTOU emulation from emacsclient, non-interactive session can't be moved to a background job. $ chg pull ^Z suspended $ bg %1 [1] continued [1] suspended (tty input) # wrong https://github.com/emacs-mirror/emacs/blob/0e96320/lib-src/emacsclient.c#L1094

chg: verify return value of sigaction() and sigemptyset() They should never fail, but it couldn't hurt to be a paranoid.

chg: initialize sigaction fields more reliably It seems calling memset() and sigemptyset() is common pattern to initialize sigaction. And strictly speaking, sigset_t must be initialized by sigemptyset() or sigfillset(). I saw git and uwsgi do that way, so let's follow them.

tests: confirm that a badly documented extension doesn't cause a crash An external extension whose docstring doesn't conform to Mercurial standards used to cause crashes. Test that we omit such extensions when you do a keyword search.

bookmarks: avoid creating a nested repository during testing This helps the test to pass with hgwatchman, which would otherwise need to be taught about a nested .hg directory. hgwatchman already blacklists test-nested-repo.t which covers the actual usecase

dispatch: strip command line options like config file options Currently, whitespace in command line --config options are considered significant while whitespace in config files are not considered significant. This diff strips the leading and trailing whitespace from command line config options.

hook: for python hook ImportErrors, add note to run with --traceback I personally found it completely non-obvious that --traceback prints out stack traces for failed imports.

hook: fewer parentheses for hook load errors This matches 'hook failed' warnings. We're also going to add hints to some of the hook load errors. Without this change we'd have two pairs of parens for a single error message, which looks really cluttered.

hook: use sys.exc_info rather than the deprecated equivalents sys.exc_type etc have been deprecated since Python 1.5.

doc: describe full help document hierarchy to create a valid link in HTML For example, ":hg:`help config.default-push`" creates an invalid link to "hgrc.5.html#default-push" in HTML, but ":hg:`help config.paths.default-push`" creates a valid link to "hgrc.5.html#paths".

doc: translate from :hg:`help config.SECTION` to a valid link to hgrc.5.html Before this patch, ":hg:`help config.SECTION`" in online help text is translated to a link to "hg.1.html#config.SECTION" in HTML unintentionally. This patch translates from :hg:`help config.SECTION` in online help text to a valid link to "hgrc.5.html#SECTION" in HTML. This patch ignores element(s) under "SECTION" (e.g. "ITEM" of ":hg:`help config.SECTION.ITEM`"), because there is no way to refer directly to it in HTML, yet.

doc: translate from :hg:`help config` to a valid link to hgrc.5.html Before this patch, ":hg:`help config`" in online help text is translated to a link to "hg.1.html#config" in HTML, even though actual "hg help config" shows not help for "hg config" command but "config" help topic, and all of current ":hg:`help config`" expects the latter. This patch translates from ":hg:`help config`" in online help text to a link to "hgrc.5.html" in HTML as expected. This patch also allows ":hg:`help -c COMMAND`" style to link "hg.1.html#COMMAND" for readability.

i18n: calculate correct line number in source of messages to be translated Before this patch, line number in source of the message to be translated is wrong in hg.pot, if corresponded message is placed after ".. DIRECTIVE::", because number of lines related to such directive isn't added to variable "delta", which holds number of untranslated lines in given text. This patch always adds "2" to "delta", because text block is split into translation units by "\n\n".

revsetbenchmark: handle exception case If the revset being benchmarked has an exception, the handling code was encountering an error because the exception did not always have an "output" attribute (I think it's a python 2.7 thing).

merge: minimize conflicts when common base is not shown (issue4447) Previously, two changes that were nearly, but not quite, identical would result in large merge conflict regions that looked very similar, and were thus very confusing to users, and lead people used to other source control systems to claim that "mercurial's merge algorithms suck". In the relatively common case of a new file being introduced in two branches with very slight modifications, the old behavior would show the entire file as a conflict, and it would be very difficult for a user to determine what was going on. In the past, mercurial attempted to solve this with a "very smart" algorithm that would find all common lines, but this has significant problems as described in 2ea6d906cf9b. Instead, we use a "very dumb" algorithm introduced in the previous patch that simply matches lines at the periphery of conflict regions. This minimizes most conflict regions well, though there may still be some degenerate edge cases, like small modification to the beginning and end of a large file.

merge: introduce method to minimize merge regions In the next diff, we will use this to trim down the start and end of conflict regions where the A and B sides both made the same changes.

merge: add some useful documentation

encoding: backport paranoid escaping from templatefilters.jsonescape() This was introduced by 55c763926a28. It is required to embed JSON data in HTML page. Convince yourself here: http://escape.alf.nu/1

encoding: add option to escape non-ascii characters in JSON This is necessary for hgweb to embed JSON data in HTML. JSON data must be able to be embedded in non-UTF-8 HTML page so long as the page encoding is compatible with ASCII. According to RFC 7159, non-BMP character is represented as UTF-16 surrogate pair. This function first splits an input string into an array of UTF-16 code points. https://tools.ietf.org/html/rfc7159.html#section-7

encoding: initialize jsonmap when module is loaded This makes jsonescape() a thread-safe function, which is necessary for hgweb. The initialization stuff isn't that slow: $ python -m timeit -n1000 -s 'from mercurial import encoding as x' 'reload(x)' original: 1000 loops, best of 3: 158 usec per loop this patch: 1000 loops, best of 3: 214 usec per loop compared to loading the commands module: $ python -m timeit -n1000 -s 'from mercurial import commands as x' 'reload(x)' 1000 loops, best of 3: 1.11 msec per loop