keepalive: remove useless parentheses around exception type

chgserver: add a structure for confighash and mtimehash confighash and mtimehash are often used together. This patch adds a simple structure called hashstate to store them. hashstate also has a handly method called fromui to calculate the hashes from a ui object.

chgserver: add utilities to calculate mtimehash mtimehash is designed to detect file changes. These files include: - single file extensions (__init__.py for complex extensions) - mercurial/__version__.py - python (sys.executable) mtimehash only uses stat to check files so it's fast but not 100% accurate. However it should be good enough for our use case. For chgserver, once mtimehash changes, the server is considered outdated immediately and should no longer provide service.

tests: rename 'test-module-import.t' into 'test-check-module-import.t' This test is checking our source code to ensure style and correct behavior (eg: no cycle). Current convention is that such tests starts with 'test-check-' so we flock this on back with the others.

pull: deactivate a bookmark not matching with the destination of the update Before this patch, "hg pull -u" with a target doesn't deactivate a current active bookmark, which doesn't match with the explicit destination of the update, even though bare "hg update" does so. A "target" can be provided through: - option --rev ANOTHER - option --branch ANOTHER - source URL#ANOTHER

pull: activate a bookmark matching with the destination of the update (BC) Before this patch, "hg pull -u" with a target doesn't activate a bookmark, which matches with the explicit destination of the update, even though bare "hg update" does so. A "target" can be provided through: - option --rev BOOKMARK - source URL#BOOKMARK

revset: define "pat" variable unconditionally in subrepo() It's a source of UnboundLocalError to define and use local variables conditionally. As getstring() always returns a str, "pat" can be initialized to None.

revset: drop translation markers from error messages of internal _matchfiles They are a sort of debug messages, which should never be visible to end users.

templatekw: switch ctx of list expression to rev of {parents} (BC) This is the same semantics as revset() introduced at e4609ec959f8. Before this patch, {parents} provided nothing useful in new-style template. For example, '{parents % "{parent}"}' generated cryptic string like "rev12345node0123abcdef...". This patch drops {parent} variable since it was useless. We can get a revision number by '{parents % "{rev}"}'.

commands: add postincoming explicit brev argument (API) Before this patch, postincoming() initializes 'brev' with 'checkout', but this isn't useful to activate/deactivate bookmark after updating, because 'checkout' is not a string actually specified at command line, but an already node-nized byte sequence. This patch adds postincoming() explicit 'brev' argument, and makes 'pull()' pass appropriate value. This patch adds 'brev' argument instead of 'brev=None', because 'brev=None' isn't reasonable value if checkout is not None.

hgweb: add index template to json/map This template allows showing the list of all repos in an hgweb instance (in hgwebdir mode) as json. The test has "lastchange" globbed because hgweb uses here file modification time and not the last commit time.

merge: fix error message Obvious copy-and-paste error

destutil: show message about other branch heads, even if on a closed head Before this patch, bare "hg update" displays message below, if there is at least one non-closed branch head other than current parent: 1. 'XX other heads for branch "BRANCH"' message, if current parent is on a non-closed branch head This suggests user to invoke "hg heads" or so for merging them. 2. no message, if current parent is on a closed branch head At this patch, bare "hg update" might choose closed branch head as update destination, and it causes this situation easily. 3. no message, otherwise (= current parent isn't on any branch head) 'XX other heads for branch "BRANCH"' should be displayed also in #2 case above, because user might overlook other non-closed branch heads. This patch gets a list of all branch heads regardless of closed-ness of it, and uses it (= 'allheads') to distinguish #1/#2 from #3 above.

repoview: discard filtered changelog if index isn't shared with unfiltered Before this patch, revisions rollbacked at failure of previous transaction might be visible at subsequent operations unintentionally, if repoview object is reused even after failure of transaction: e.g. command server and HTTP server are typical cases. 'repoview' uses the tuple of values below of unfiltered changelog as "the key" to examine validity of filtered changelog cache. - length - tip node - filtered revisions (as hashed value) - '_delayed' field 'repoview' compares between "the key" of unfiltered changelog at previous caching and now, and reuses filtered changelog cache if no change is detected. But this comparison indicates only that there is no change between unfiltered 'repo.changelog' at last caching and now, but not that filtered changelog cache is valid for current unfiltered one. 'repoview' uses "shallow copy" of unfiltered changelog to create filtered changelog cache. In this case, 'index' buffer of unfiltered changelog is also referred by filtered changelog. At failure of transaction, unfiltered changelog itself is invalidated (= un-referred) on the 'repo' side (see 0a7610758c42 also). But 'index' of it still contains revisions to be rollbacked at this failure, and is referred by filtered changelog. Therefore, even if there is no change between unfiltered 'repo.changelog' at last caching and now, steps below makes rollbacked revisions visible via filtered changelog unintentionally. 1. instantiate unfiltered changelog as 'repo.changelog' (call it CL1) 2. make filtered (= shallow copy of) CL1 (call it FCL1) 3. cache FCL1 with "the key" of CL1 4. revisions are appended to 'index', which is shared by CL1 and FCL1 5. invalidate 'repo.changelog' (= CL1) at failure of transaction 6. instantiate 'repo.changelog' again at next operation (call it CL2) CL2 doesn't have revisions added at (4), because it is instantiated from '00changelog.i', which isn't changed while failed transaction. 7. compare between "the key" of CL1 and CL2 8. FCL1 cached at (3) is reused, because comparison at (7) doesn't detect change between CL1 at (1) and CL2 9. revisions rollbacked at (5) are visible via FCL1 unintentionally, because FCL1 still refers 'index' changed at (4) The root cause of this issue is that there is no examination about validity of filtered changelog cache against current unfiltered one. This patch discards filtered changelog cache, if its 'index' object isn't shared with unfiltered one. BTW, at the time of this patch, redundant truncation of '00changelog.i' at failure of transaction (see 0a7610758c42 for detail) often prevents "hg serve" from making already rollbacked revisions visible, because updating timestamps of '00changelog.i' by truncation makes "hg serve" discard old repoview object with invalid filtered changelog cache. This is reason why this issue is overlooked before this patch, even though test-bundle2-exchange.t has tests in similar situation: failure of "hg push" via HTTP by pretxnclose hook on server side doesn't prevent subsequent commands from looking up outgoing revisions correctly. But timestamp on the filesystem doesn't have enough resolution for recent computation power, and it can't be assumed that this avoidance always works as expected. Therefore, without this patch, this issue might appear occasionally.

chgserver: make _renewui load repo and command line configs Before this patch, there is no way to load repo config in chgserver. This patch revised _renewui to let it load repo config and take command line flags passed into consideration. The _renewui logic is not ideal at present because it's very tricky to know what should copy and what should not from the old ui object to the new one. This is partially because the current ui and config object design is not ideal. In the future, we may want to avoid all ui or config copies.

dispatch: add wd parameter to _getlocal Before this patch, _getlocal uses os.getcwd() to locate repo in current dir. chgserver needs it to load repo config and has to do chdir twice: the first is to set current directory and the second is to redo the side effect (in case hg --cwd some/relative/path, chdir will be called again in dispatch later), which is not pretty. This patch adds an optional wd parameter to make it possible to specify wd without chdir (and its side effect).

chgserver: add utilities to calculate confighash confighash is the hash of sensitive config items like [extensions], and sensitive environment variables like HG*, LD_*, etc. The config items can come from global, user, repo config, and command line flags. For chgserver, it is designed that once confighash changes, the server is not qualified to serve its client and should redirect the client to a new server. The server does not need to exit in this case, since it can still be valid (have a matched confighash) to serve other chg clients.

chg: detect chg started by chg Sometimes people may create a symbol link from hg to chg, or write a wrapper script named hg calling chg. Without $HG and $CHGHG set, this will lead to chg executes itself causing deadlock. The user will notice chg hangs for some time and aborts with a timed out message, without knowing the root cause and how to solve it. This patch sets a dummy environment variable before executing hg to detect this situation, and print a fatal message with some possible solutions. CHGINTERNALMARK is set by chg client to detect the situation that chg is started by chg. It is temporary and should be dropped to avoid possible side effects.

chg: fallback to original hg for some unsupported commands or flags There are some known unsupported commands or flags for chg, such as hg serve -d and hg foo --time. This patch detects these situations and transparently fall back to the original hg. So the users won't bother remembering what chg can and cannot do by themselves. The current detection is not 100% accurate since we do not have an equivalent command line parser in C. But it tries not to cause false positives that prevents people from using chg for legit cases. In the future we may want to implement a more accurate "unsupported" check server-side.

testing: add a 'continuous' profile This gives a good way of letting Hypothesis run until it finds an error, save that error, and be restarted without it picking up on the old bug. This lets you run long-running Hypothesis processes and then perform a manual deduplication task on the bugs found at the end. It's not an entirely satisfying way of using this, but anything much better would require extensive changes to Hypothesis itself.

testing: allow Hypothesis to enable extensions This adds support for testing extensions, including both tests that extensions don't change behaviour and test for specific commands. We use the precondition system to determine what commands are available to us. If we never use any commands enabled by an extension then that extension is *skippable* and should not have changed the behaviour of the test. We thus rerun the test with an environment variable which is designed to turn off the extension.

testing: test multiple repositories with Hypothesis This expands the Hypothesis based stateful testing so that rather than having a single repository under test, Hypothesis manages a family of repositories. Some of these are freshly created, some are clones of others.

testing: expand Hypothesis tests with branch commands This builds on the previous work to add Hypothesis based stateful testing to add branching commands to the model.

testing: generate tests operations using Hypothesis The idea of this patch is to expand the use of Hypothesis within Mercurial to use its concept of "stateful testing". The result is a test which runs a sequence of operations against a Mercurial repository. Each operation is given a set of allowed ways it can fail. Any other non-zero exit code is a test failure. At the end, the whole sequence is then reverified by generating a .t test and testing it again in pure mode (this is also useful for catching non-determinism bugs). This has proven reasonably effective at finding bugs, and has identified two problems in the shelve extension already (issue5113 and issue5112).

i18n-ja: synchronized with cb6a952efbf4

log: fix order of revisions filtered by multiple OR options (issue5100) This is the simplest workaround for the issue of the ordering of revset, which is that the expression "x or y" takes over the ordering specified by the input set (or the left-hand-side expression.) For example, the following expression A & (x | y) will be evaluated as if (A & x) | (A & y) That's wrong because revset has ordering. I'm going to fix this problem in the revset module, but that wouldn't fit to stable. So, this patch just works around the common log cases. Since this change might have some impact on performance, it is enabled only if the expression built from log options has ' or ' operation.

demandimport: add _imp to ignore list Mozilla is seeing an issue with demand importing of _imp failing in pkg_resources/__init__.py:fixup_namespace_packages. It strangely only reproduces when using a modern version of setuptools/pip in certain scenarios. Adding _imp to the demand import ignore list seems to make the problem go away.

tests: rename regression tests

zeroconf: fix crash in "hg paths" when zeroconf server is up Running "hg paths" with zeroconf enabled and when a zeroconf server is up and running gives a traceback with "ValueError: rawloc must be defined". This is because zeroconf needs to wrap ui.configsuboptions(), introduced in dccbebcff075.

zeroconf: fix setsockopt() call on Solaris to send payload of correct length The zeroconf extension has been broken on Solaris since the beginning, but no one noticed until the testsuite started poking it after changeset 72f2a19c5f88, when it started running "hg paths" with the extension enabled. Solaris requires that, for IP_MULTICAST_{TTL,LOOP}, the argument passed in be of length 1. With the original code here, it gets passed in as an int -- length 4 -- and so the system call fails with EINVAL. Thankfully, Python's socket.setsockopt() allows you to pass in a string instead of an integer, and it passes that string to libc's setsockopt() with the correct value and length.

blackbox: properly replace ui class Without this, anyone creating a ui object using: uimod.ui() skips the blackbox. Also, anyone doing ui.copy() skipped the blackbox. Unfortunately, the ui object lifestyle is a bit messy, the first one that's created is never actually initialized with subclasses, instead pieces of the subclass are adopted into the primal ui object. In order to handle this, a _partialinit method will be called to ensure that the blackboxui is properly initialized.

blackbox: store the blackbox ui object instead of the log file Without this, the last logged entry didn't have access to the repository, and thus couldn't report its version (and especially that an add or similar dirtied it). A side-effect is that one repo leaks until process exit...

blackbox: log dirty state If blackbox.dirty = True, use `+` to indicate dirty repositories.

blackbox: log working directory version Without this, while you could see the list of commands run, it wasn't possible to identify what they were doing, because commads could rely on revsets (including remote input which varies over time).

blackbox: rename fp variable

blackbox: avoid creating multiple file handles for a single log There are multiple ui objects in Mercurial that can relate to a repository, before this change, each one would have its own file pointer, which results in unfortunate logging behavior. Also, any log rotation results would be bad because only the active blackboxui object's file pointer would be refreshed. Note that this does not prevent two long running hg commands for the same repository from causing problems.

tests: mock getpid to reduce glob usage Updating tests based on ac49ecb2a897.

changegroup: drop special-casing of flat manifests Since c08814b48ae5 (changegroup: avoid iterating the whole manifest, 2015-12-04), the manifest linkrev callback iterates over only the files that were touched according the the changeset. Before that change, we iterated over all files returned in manifest.readfast(). That method returns the files in the delta, if the delta parent is a parent, otherwise it returns the full manifest. Most manifest revisions end up using one of the parents as its delta parent, so most of the time, the method returns a short manifest. It seems that that happens often enough that it doesn't really matter; I could not reproduce the timings reported in that change. Since the treemanifest code now works quite differently, and since that code also works correctly for flat manifests, let's drop the special-casing of flat manifests.

changegroup: fix treemanifests on merges The current code for generating treemanifest revisions takes the list of files in the changeset and finds the directories from them. This does not work for merges, since a merge may pick file A from one side and file B from another and neither of them would appear in the changeset's "files" list, but the manifest would still change. Fix this by instead walking the root manifest log for all needed revisions, storing all needed file and subdirectory revisions, then recursively visiting the subdirectories. This also turns out to be faster: cloning a version of hg core converted to treemanifests went from ~28s to ~19s (timing somewhat unfair: before this patch, timed until crash; after this patch, timed until manifests complete). The new algorithm is used only on treemanifest repos. Although it works equally well on flat manifests, we leave the iteration over files in the changeset for flat manifests for now.

templatekw: workaround for utf-8 round-trip of {desc} Though our encoding strategy is best effort, {desc} is a primitive keyword that should be worth enough to try hard to preserve UTF-8 bytes.

test: update test-bundle2-format.t comment Now that bundle2 is used by default for all exchanges, this comment is obviously out of date. Having deep testing of the API and expected behavior of the format and its processing is still valuable, so the comment is updated.

chg: extract gethgcmd logic to a function gethgcmd is to get original hg (not chg) binary name. This patch extracts the logic from execcmdserver to make it available for the following patch.

destutil: use cached branch information instead of query for efficiency Before this patch, calculation of "the tipmost branch head on current branch" uses revset query "max(.::(head() and branch(BRANCH)))", but this isn't efficiency, because: - head() predicate lists up heads on all branches, but - branch() predicate eliminates heads on other branches In addition to it, without "literal:" prefix for branch name, branch(BRANCH) tries to (1) look up BRANCH in "repo.branchmap()" and (2) look up BRANCH as symbol name again, if there is no branch matching against BRANCH. The latter looking up is obviously redundant. This patch uses repo.branchheads(closed=True) to get all branch heads on specified branch instead of "head() and branch(BRANCH)" revset query part. This patch also makes catching RepoLookupError meaningless, because it is only raised by revset predicate "branch()". But "currentbranch in repo.branchmap()" can detect whether currentbranch actually exists or not. Therefore, this patch replaces try/except for RepoLookupError by if/else for "currentbranch in repo.branchmap()".

destutil: replace wc.branch() invocations by cached value for efficiency

destutil: remove redundant examination Before this patch, "len(heads) != len(otherheads)" is examined to detect whether message should be displayed or not. But if "repo.revs('%ln and parents()', heads)", heads should contain "parents()" and otherheads is always less than heads.

destutil: add new local variable to increase readability Before this patch, local variable 'heads' is used not only for "all branch heads" but also for "branch heads other than current parent". This patch newly adds local variable 'otherheads' for the latter purpose, to increase readability.

changegroup: write root manifests and subdir manifests in a single loop This is another step towards making the manifest generation recurse along the directory trees. The loop over 'tmfnodes' now takes the form of a queue. At this point, we only add to the queue twice: we add the root manifests, and, while visiting the root manifest revisions, we add all subdirectory revisions (for treemanifest repos). Thus, any iterations over 'tmfnodes' after the first will not add any items and the "queue" will just keep shrinking.

changegroup: introduce makelookupmflinknode(dir) This is another step towards making the manifest generation recurse along the directory trees. It makes the two calls to _packmanifests() more similar.