destutil: show message about other branch heads, even if on a closed head Before this patch, bare "hg update" displays message below, if there is at least one non-closed branch head other than current parent: 1. 'XX other heads for branch "BRANCH"' message, if current parent is on a non-closed branch head This suggests user to invoke "hg heads" or so for merging them. 2. no message, if current parent is on a closed branch head At this patch, bare "hg update" might choose closed branch head as update destination, and it causes this situation easily. 3. no message, otherwise (= current parent isn't on any branch head) 'XX other heads for branch "BRANCH"' should be displayed also in #2 case above, because user might overlook other non-closed branch heads. This patch gets a list of all branch heads regardless of closed-ness of it, and uses it (= 'allheads') to distinguish #1/#2 from #3 above.

repoview: discard filtered changelog if index isn't shared with unfiltered Before this patch, revisions rollbacked at failure of previous transaction might be visible at subsequent operations unintentionally, if repoview object is reused even after failure of transaction: e.g. command server and HTTP server are typical cases. 'repoview' uses the tuple of values below of unfiltered changelog as "the key" to examine validity of filtered changelog cache. - length - tip node - filtered revisions (as hashed value) - '_delayed' field 'repoview' compares between "the key" of unfiltered changelog at previous caching and now, and reuses filtered changelog cache if no change is detected. But this comparison indicates only that there is no change between unfiltered 'repo.changelog' at last caching and now, but not that filtered changelog cache is valid for current unfiltered one. 'repoview' uses "shallow copy" of unfiltered changelog to create filtered changelog cache. In this case, 'index' buffer of unfiltered changelog is also referred by filtered changelog. At failure of transaction, unfiltered changelog itself is invalidated (= un-referred) on the 'repo' side (see 0a7610758c42 also). But 'index' of it still contains revisions to be rollbacked at this failure, and is referred by filtered changelog. Therefore, even if there is no change between unfiltered 'repo.changelog' at last caching and now, steps below makes rollbacked revisions visible via filtered changelog unintentionally. 1. instantiate unfiltered changelog as 'repo.changelog' (call it CL1) 2. make filtered (= shallow copy of) CL1 (call it FCL1) 3. cache FCL1 with "the key" of CL1 4. revisions are appended to 'index', which is shared by CL1 and FCL1 5. invalidate 'repo.changelog' (= CL1) at failure of transaction 6. instantiate 'repo.changelog' again at next operation (call it CL2) CL2 doesn't have revisions added at (4), because it is instantiated from '00changelog.i', which isn't changed while failed transaction. 7. compare between "the key" of CL1 and CL2 8. FCL1 cached at (3) is reused, because comparison at (7) doesn't detect change between CL1 at (1) and CL2 9. revisions rollbacked at (5) are visible via FCL1 unintentionally, because FCL1 still refers 'index' changed at (4) The root cause of this issue is that there is no examination about validity of filtered changelog cache against current unfiltered one. This patch discards filtered changelog cache, if its 'index' object isn't shared with unfiltered one. BTW, at the time of this patch, redundant truncation of '00changelog.i' at failure of transaction (see 0a7610758c42 for detail) often prevents "hg serve" from making already rollbacked revisions visible, because updating timestamps of '00changelog.i' by truncation makes "hg serve" discard old repoview object with invalid filtered changelog cache. This is reason why this issue is overlooked before this patch, even though test-bundle2-exchange.t has tests in similar situation: failure of "hg push" via HTTP by pretxnclose hook on server side doesn't prevent subsequent commands from looking up outgoing revisions correctly. But timestamp on the filesystem doesn't have enough resolution for recent computation power, and it can't be assumed that this avoidance always works as expected. Therefore, without this patch, this issue might appear occasionally.

chgserver: make _renewui load repo and command line configs Before this patch, there is no way to load repo config in chgserver. This patch revised _renewui to let it load repo config and take command line flags passed into consideration. The _renewui logic is not ideal at present because it's very tricky to know what should copy and what should not from the old ui object to the new one. This is partially because the current ui and config object design is not ideal. In the future, we may want to avoid all ui or config copies.

dispatch: add wd parameter to _getlocal Before this patch, _getlocal uses os.getcwd() to locate repo in current dir. chgserver needs it to load repo config and has to do chdir twice: the first is to set current directory and the second is to redo the side effect (in case hg --cwd some/relative/path, chdir will be called again in dispatch later), which is not pretty. This patch adds an optional wd parameter to make it possible to specify wd without chdir (and its side effect).

chgserver: add utilities to calculate confighash confighash is the hash of sensitive config items like [extensions], and sensitive environment variables like HG*, LD_*, etc. The config items can come from global, user, repo config, and command line flags. For chgserver, it is designed that once confighash changes, the server is not qualified to serve its client and should redirect the client to a new server. The server does not need to exit in this case, since it can still be valid (have a matched confighash) to serve other chg clients.

chg: detect chg started by chg Sometimes people may create a symbol link from hg to chg, or write a wrapper script named hg calling chg. Without $HG and $CHGHG set, this will lead to chg executes itself causing deadlock. The user will notice chg hangs for some time and aborts with a timed out message, without knowing the root cause and how to solve it. This patch sets a dummy environment variable before executing hg to detect this situation, and print a fatal message with some possible solutions. CHGINTERNALMARK is set by chg client to detect the situation that chg is started by chg. It is temporary and should be dropped to avoid possible side effects.

chg: fallback to original hg for some unsupported commands or flags There are some known unsupported commands or flags for chg, such as hg serve -d and hg foo --time. This patch detects these situations and transparently fall back to the original hg. So the users won't bother remembering what chg can and cannot do by themselves. The current detection is not 100% accurate since we do not have an equivalent command line parser in C. But it tries not to cause false positives that prevents people from using chg for legit cases. In the future we may want to implement a more accurate "unsupported" check server-side.

testing: add a 'continuous' profile This gives a good way of letting Hypothesis run until it finds an error, save that error, and be restarted without it picking up on the old bug. This lets you run long-running Hypothesis processes and then perform a manual deduplication task on the bugs found at the end. It's not an entirely satisfying way of using this, but anything much better would require extensive changes to Hypothesis itself.

testing: allow Hypothesis to enable extensions This adds support for testing extensions, including both tests that extensions don't change behaviour and test for specific commands. We use the precondition system to determine what commands are available to us. If we never use any commands enabled by an extension then that extension is *skippable* and should not have changed the behaviour of the test. We thus rerun the test with an environment variable which is designed to turn off the extension.

testing: test multiple repositories with Hypothesis This expands the Hypothesis based stateful testing so that rather than having a single repository under test, Hypothesis manages a family of repositories. Some of these are freshly created, some are clones of others.

testing: expand Hypothesis tests with branch commands This builds on the previous work to add Hypothesis based stateful testing to add branching commands to the model.

testing: generate tests operations using Hypothesis The idea of this patch is to expand the use of Hypothesis within Mercurial to use its concept of "stateful testing". The result is a test which runs a sequence of operations against a Mercurial repository. Each operation is given a set of allowed ways it can fail. Any other non-zero exit code is a test failure. At the end, the whole sequence is then reverified by generating a .t test and testing it again in pure mode (this is also useful for catching non-determinism bugs). This has proven reasonably effective at finding bugs, and has identified two problems in the shelve extension already (issue5113 and issue5112).

i18n-ja: synchronized with cb6a952efbf4

log: fix order of revisions filtered by multiple OR options (issue5100) This is the simplest workaround for the issue of the ordering of revset, which is that the expression "x or y" takes over the ordering specified by the input set (or the left-hand-side expression.) For example, the following expression A & (x | y) will be evaluated as if (A & x) | (A & y) That's wrong because revset has ordering. I'm going to fix this problem in the revset module, but that wouldn't fit to stable. So, this patch just works around the common log cases. Since this change might have some impact on performance, it is enabled only if the expression built from log options has ' or ' operation.

demandimport: add _imp to ignore list Mozilla is seeing an issue with demand importing of _imp failing in pkg_resources/__init__.py:fixup_namespace_packages. It strangely only reproduces when using a modern version of setuptools/pip in certain scenarios. Adding _imp to the demand import ignore list seems to make the problem go away.

tests: rename regression tests

zeroconf: fix crash in "hg paths" when zeroconf server is up Running "hg paths" with zeroconf enabled and when a zeroconf server is up and running gives a traceback with "ValueError: rawloc must be defined". This is because zeroconf needs to wrap ui.configsuboptions(), introduced in dccbebcff075.

zeroconf: fix setsockopt() call on Solaris to send payload of correct length The zeroconf extension has been broken on Solaris since the beginning, but no one noticed until the testsuite started poking it after changeset 72f2a19c5f88, when it started running "hg paths" with the extension enabled. Solaris requires that, for IP_MULTICAST_{TTL,LOOP}, the argument passed in be of length 1. With the original code here, it gets passed in as an int -- length 4 -- and so the system call fails with EINVAL. Thankfully, Python's socket.setsockopt() allows you to pass in a string instead of an integer, and it passes that string to libc's setsockopt() with the correct value and length.

blackbox: properly replace ui class Without this, anyone creating a ui object using: uimod.ui() skips the blackbox. Also, anyone doing ui.copy() skipped the blackbox. Unfortunately, the ui object lifestyle is a bit messy, the first one that's created is never actually initialized with subclasses, instead pieces of the subclass are adopted into the primal ui object. In order to handle this, a _partialinit method will be called to ensure that the blackboxui is properly initialized.

blackbox: store the blackbox ui object instead of the log file Without this, the last logged entry didn't have access to the repository, and thus couldn't report its version (and especially that an add or similar dirtied it). A side-effect is that one repo leaks until process exit...

blackbox: log dirty state If blackbox.dirty = True, use `+` to indicate dirty repositories.

blackbox: log working directory version Without this, while you could see the list of commands run, it wasn't possible to identify what they were doing, because commads could rely on revsets (including remote input which varies over time).

blackbox: rename fp variable

blackbox: avoid creating multiple file handles for a single log There are multiple ui objects in Mercurial that can relate to a repository, before this change, each one would have its own file pointer, which results in unfortunate logging behavior. Also, any log rotation results would be bad because only the active blackboxui object's file pointer would be refreshed. Note that this does not prevent two long running hg commands for the same repository from causing problems.

tests: mock getpid to reduce glob usage Updating tests based on ac49ecb2a897.

changegroup: drop special-casing of flat manifests Since c08814b48ae5 (changegroup: avoid iterating the whole manifest, 2015-12-04), the manifest linkrev callback iterates over only the files that were touched according the the changeset. Before that change, we iterated over all files returned in manifest.readfast(). That method returns the files in the delta, if the delta parent is a parent, otherwise it returns the full manifest. Most manifest revisions end up using one of the parents as its delta parent, so most of the time, the method returns a short manifest. It seems that that happens often enough that it doesn't really matter; I could not reproduce the timings reported in that change. Since the treemanifest code now works quite differently, and since that code also works correctly for flat manifests, let's drop the special-casing of flat manifests.

changegroup: fix treemanifests on merges The current code for generating treemanifest revisions takes the list of files in the changeset and finds the directories from them. This does not work for merges, since a merge may pick file A from one side and file B from another and neither of them would appear in the changeset's "files" list, but the manifest would still change. Fix this by instead walking the root manifest log for all needed revisions, storing all needed file and subdirectory revisions, then recursively visiting the subdirectories. This also turns out to be faster: cloning a version of hg core converted to treemanifests went from ~28s to ~19s (timing somewhat unfair: before this patch, timed until crash; after this patch, timed until manifests complete). The new algorithm is used only on treemanifest repos. Although it works equally well on flat manifests, we leave the iteration over files in the changeset for flat manifests for now.

templatekw: workaround for utf-8 round-trip of {desc} Though our encoding strategy is best effort, {desc} is a primitive keyword that should be worth enough to try hard to preserve UTF-8 bytes.

test: update test-bundle2-format.t comment Now that bundle2 is used by default for all exchanges, this comment is obviously out of date. Having deep testing of the API and expected behavior of the format and its processing is still valuable, so the comment is updated.

chg: extract gethgcmd logic to a function gethgcmd is to get original hg (not chg) binary name. This patch extracts the logic from execcmdserver to make it available for the following patch.

destutil: use cached branch information instead of query for efficiency Before this patch, calculation of "the tipmost branch head on current branch" uses revset query "max(.::(head() and branch(BRANCH)))", but this isn't efficiency, because: - head() predicate lists up heads on all branches, but - branch() predicate eliminates heads on other branches In addition to it, without "literal:" prefix for branch name, branch(BRANCH) tries to (1) look up BRANCH in "repo.branchmap()" and (2) look up BRANCH as symbol name again, if there is no branch matching against BRANCH. The latter looking up is obviously redundant. This patch uses repo.branchheads(closed=True) to get all branch heads on specified branch instead of "head() and branch(BRANCH)" revset query part. This patch also makes catching RepoLookupError meaningless, because it is only raised by revset predicate "branch()". But "currentbranch in repo.branchmap()" can detect whether currentbranch actually exists or not. Therefore, this patch replaces try/except for RepoLookupError by if/else for "currentbranch in repo.branchmap()".

destutil: replace wc.branch() invocations by cached value for efficiency

destutil: remove redundant examination Before this patch, "len(heads) != len(otherheads)" is examined to detect whether message should be displayed or not. But if "repo.revs('%ln and parents()', heads)", heads should contain "parents()" and otherheads is always less than heads.

destutil: add new local variable to increase readability Before this patch, local variable 'heads' is used not only for "all branch heads" but also for "branch heads other than current parent". This patch newly adds local variable 'otherheads' for the latter purpose, to increase readability.

changegroup: write root manifests and subdir manifests in a single loop This is another step towards making the manifest generation recurse along the directory trees. The loop over 'tmfnodes' now takes the form of a queue. At this point, we only add to the queue twice: we add the root manifests, and, while visiting the root manifest revisions, we add all subdirectory revisions (for treemanifest repos). Thus, any iterations over 'tmfnodes' after the first will not add any items and the "queue" will just keep shrinking.

changegroup: introduce makelookupmflinknode(dir) This is another step towards making the manifest generation recurse along the directory trees. It makes the two calls to _packmanifests() more similar.

changegroup: prune subdirectory dirlogs too We already prune changesets, root manifests and files whose linkrev is in the set of common revisions. We should do the same for dirlogs.

changegroup: include subdirectory manifests in verbose size When verbose logging is one, we report the size in bytes of the manifest data in the changegroup. For files, we report the size per file, but I'm not sure we need that level of detail (i.e. size per directory manifest). Instead, report a single figure for the size of root manifest plus submanifests.

changegroup: make _packmanifests() dumber The next few patches will rewrite the manifest generation code to work with merges. We will then walk dirlogs recursively. This prepares for that by moving much of the treemanifest code out of _packmanifests() and into generatemanifests(). For this to work, it also adds _manifestsdone() method that returns the "end of manifests" close chunk for cg3 and an empty string for cg1 and cg2.

changegroup: extract generatemanifests() The changegroup.generate() function is pretty long, so let's extract the manifest generation part of it.

merge: use any() instead of for loop when checking for dirty subrepos I think it's both simpler and clearer to use any() than the current for loop. While at it, also drop the call to sorted(), since it doesn't matter which order we iterate over subrepos.

templater: fix list templating bug High-level use case: printing a list of objects with formatter when each object in turn contains a list of properties (like when % template symbol is used in {things % '{thing}'} Let the top-level list contain one thing with two properties: objs = [{ 'props': [ { 'value': 1, 'show': 1 }, { 'value': 2 }] }] (please note that second property does not have 'show' key) If a templateformatter is used to print this with template "{props % '{if(show, value)}'}" current implementation will print value for both properties, which is a bug. This happens because in `templater.runmap` function we only rewrite mapping values with existing new values for each item. If some mapping value is missing in the item, it will not be removed.

histedit: also handle locally missing nodes when reading obsolescence The previous version of the code was interpreting markers to a missing node as a prune in all cases. The expected way to handle such situation is to keep reading markers, only turning successors into "prune" if they are at the end of a chain. We update the code and add a test for this.

chgserver: auto exit after being idle for too long or lose the socket file This is a part of the one server per config series. In multiple-server setup, new server may be started for a temporary config change like in command line, --config extensions.foo=bar.py. This may end up with a lot of not so useful server processes. Other questions are about socket file and process management, How to stop these processes? What if a new server wants to listen on a same address, replacing the old one? This patch introduces AutoExitMixIn, which will: 1. Exit after being idle for too long. So useless servers won't run forever. 2. Periodically check the ownership of socket file, exit if it is no longer owned. This brings strong consistency between the filesystem and the process, and handles some race conditions neatly. Since rename is atomic, a new server can just have a same server address with an old one and won't worry about how to make sure the old server is killed, address conflict, service downtime issues. The user can safely stop all servers by simply removing the socket files, without worrying about outdated or accidentally removed pidfiles.

unionrepo: properly handle hidden linkrev in revlog (issue5070) The unionrepository have to do some special magic to handle linkrev of the unioned filerev and manifestrev. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

bundlerepo: properly handle hidden linkrev in manifestlog (issue4945) The bundlerepository have to do some special magic to handle linkrev of the bundled manifest. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

bundlerepo: properly handle hidden linkrev in filelog (issue4945) The bundlerepository have to do some special magic to handle linkrev of the bundlerepo filerev. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

check-code: allow old style class with special comments The following chgserver change will use an old style class to comply with SocketServer's code style. This patch made it possible to pass check-code.

merge with stable

revset: use smartset minus operator Previously, revsets like 'X - Y' were translated to be 'X and not Y'. This can be expensive, since if Y is a single commit then 'not Y' becomes a huge set and sometimes the query optimizer doesn't account for it well. This patch changes revsets to use the built in smartset minus operator, which is often smarter than 'X and not Y'. On a large repo this saves 2.2 seconds on rebase and histedit because "X:: - X" becomes almost instant. Relevant performance numbers from revsetbenchmark.py revset #13: roots((tip~100::) - (tip~100::tip)) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.001080 0.001107 0.001102 0.001118 0.001121 0.001114 0.001141 0.001123 0.001099 0.001123 0.001137 1) 0.000708 65% 0.000738 66% 0.000735 66% 0.000739 66% 0.000784 69% 0.000780 70% 0.000807 70% 0.000756 67% 0.000727 66% 0.000759 67% 0.000808 71% revset #14: roots((0::) - (0::tip)) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.131304 0.079168 0.133129 0.076560 0.048179 0.133349 0.049153 0.077097 0.129689 0.076212 0.048543 1) 0.065066 49% 0.036941 46% 0.066063 49% 0.034755 45% 0.048558 0.071091 53% 0.047679 0.034984 45% 0.064572 49% 0.035680 46% 0.048508 revset #22: (not public() - obsolete()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.000139 0.000133 0.000133 0.000138 0.000134 0.000155 0.000157 0.000152 0.000157 0.000156 0.000153 1) 0.000108 77% 0.000129 0.000129 0.000134 0.000132 0.000127 81% 0.000151 0.000147 0.000127 80% 0.000152 0.000149 revset #25: (20000::) - (20000) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.050560 0.045513 0.022593 0.043588 0.021909 0.045517 0.021822 0.044660 0.049740 0.044227 0.021819 1) 0.018614 36% 0.000171 0% 0.019659 87% 0.000168 0% 0.015543 70% 0.021069 46% 0.015623 71% 0.000180 0% 0.018658 37% 0.000186 0% 0.015750 72%

histedit: make histedit aware of obsolescense not stored in state (issue4800) Before this change, when histedit exited to interactive session (during edit command for example), user could introduce obsolescence markers that would not be known to histedit. For example, user could've amended one of the commits. The fact of this amendment would not be stored in histedit's state file and later, when histedit would try to process all the replacements, one of the final successors (in histedit's opinion) would turn out to be hidden. This behavior is described in issue4800. This commit fixes it.

treemanifest: allow setting flag to 't' When using treemanifests, an on-disk manifest entry with the 't' flag set means that that entry is a directory and not a file. When read into memory, these become instances of the treemanifest class. The 't' flag should therefore never be visible to outside of manifest.py, so setflag() checks that it is not called with the 't' flag. However, it turns out that it will be useful for the narrowhg extension to expose the 't' flag to the user (see below), so let's drop the assertion. The narrowhg extension allows cloning only a given set of files and directories. Filelogs and dirlogs that don't match that set will not be included in the clone. The extension currently doesn't work with treemanifests. I plan on changing it so directories outside the narrow clone appear in the manifest. For example, if a directory 'outside/' is not part of the narrow clone, it will look like a file 'outside' with the 't' flag set. That will make e.g. manifestmerge() just work in most cases (and make it well prepared to handle the other cases).

treemanifest: use "cp xyz/." instead of "cp xyz/*" This is more similar to cp -T because it covers hidden files.

templatefilters: drop old jsonescape() function It's been superseded by encoding.jsonescape(paranoid=True).

templatefilters: make json filter be byte-transparent (BC) (issue4926) This is necessary to preserve filename encoding over JSON. Instead, this patch inserts "|utf8" where non-ascii local-encoding texts can be passed to "|json". See also the commit that introduced "utf8" filter.

hgweb: add option to convert encoding of graphdata() Because future patches will change "|json" filter to handle input bytes transparently, i.e. use UTF-8b encoding, "{jsdata}" must keep data in UTF-8 bytes, whereas "{nodes}" are text. This patch inserts encodestr() where localstr is likely to survive.

hgweb: remove unused argument from graphdata() factory As graphdata() is wrapped by lambda, there's no reason to pass unused arguments to it.

templatefilters: add "utf8" to get utf-8 bytes from local-encoding text This will be applied prior to "|json" filter. This sounds like odd, but it is necessary to handle local-encoding text as well as raw filename bytes. Because filenames are bytes in Mercurial and Unix world, {filename|json} should preserve the original byte sequence, which implies {x|json} -> '"' toutf8b(x) '"' On the other hand, most template strings are in local encoding. Because "|json" filter have to be byte-transparent to filenames, we need something to annotate an input as a local string, that's what "|utf8" will do. {x|utf8|json} -> '"' toutf8b(fromlocal(x)) '"' "|utf8" is an explicit call, so aborts if input bytes can't be converted to UTF-8.

templatefilters: drop broken "jsonescape" from filters table (BC) It's been unused, undocumented and flawed in that it expects a unicode input, never works correctly if an input has non-ascii character. We should use "json" filter instead.

treemanifest: rewrite text() using iterentries() This simplifies a bit. Note that the function is only used when manually testing with _treeinmem=True.

treemanifest: implement iterentries() To make tests pass with _treeinmem manually set to True, we need to implement the recently added iterentries() on the treemanifest class too.

verify: show progress while verifying dirlogs In repos with treemanifests, the non-root-directory dirlogs often have many more total revisions than the root manifest log has. This change adds progress out to that part of 'hg verify'. Since the verification is recursive along the directory tree, we don't know how many total revisions there are at the beginning of the command, so instead we report progress in units of directories, much like we report progress for verification of files today. I'm not very happy with passing both 'storefiles' and 'progress' into the recursive calls. I tried passing in just a 'visitdir(dir)' callback, but the results did not seem better overall. I'm happy to update if anyone has better ideas.

verify: check for orphaned dirlogs We already report orphaned filelogs, i.e. revlogs for files that are not mentioned in any manifest. This change adds checking for orphaned dirlogs, i.e. revlogs that are not mentioned in any parent-directory dirlog. Note that, for fncachestore, only files mentioned in the fncache are considered, there's not check for files in .hg/store/meta that are not mentioned in the fncache. This is no different from the current situation for filelogs.

verify: check directory manifests In repos with treemanifests, there is no specific verification of directory manifest revlogs. It simply collects all file nodes by reading each manifest delta. With treemanifests, that's means calling the manifest._slowreaddelta(). If there are missing revlog entries in a subdirectory revlog, 'hg verify' will simply report the exception that occurred while trying to read the root manifest: manifest@0: reading delta 1700e2e92882: meta/b/00manifest.i@67688a370455: no node This patch changes the verify code to load only the root manifest at first and verify all revisions of it, then verify all revisions of each direct subdirectory, and so on, recursively. The above message becomes b/@0: parent-directory manifest refers to unknown revision 67688a370455 Since the new algorithm reads a single revlog at a time and in order, 'hg verify' on a treemanifest version of the hg core repo goes from ~50s to ~14s. As expected, there is no significant difference on a repo with flat manifests.

hg: perform update after pulling during clone with share (issue5103) When pooled storage is enabled, `hg clone` will initialize a repo from a local repo using the store sharing mechanism then pull from the originally requested repo. Before this patch, the working directory update occurred between these steps. This meant that we would only update to revisions that were already present in the local pooled storage. This patch moves the update to after we pull from the originally requested repository so we may check out a revision that didn't yet exist locally. In other words, it makes the behavior like normal `hg clone`.

hg: extract post share update logic into own function A future patch will introduce a new caller that needs to perform an update. Extract the code so we don't duplicate it.

merge: perform background file closing in batchget As 2fdbf22a1b63 demonstrated with stream clones, closing files on background threads on Windows can yield a significant speedup because closing files that have been created/appended to is slow on Windows/NTFS. Working directory updates can write thousands of files. Therefore it is susceptible to excessive slowness on Windows due to slow file closes. This patch enables background file closing when performing working directory file writes. The impact when performing an `hg up tip` on mozilla-central (136,357 files) from an empty working directory is significant: Before: 535s (8:55) After: 133s (2:13) Delta: -402s (6:42) That's a 4x speedup! By comparison, that same machine can perform the same operation in ~15s on Linux. So Windows went from ~35x to ~9x slower. Not bad but there's still work to do. As a reminder, background file closing is only activated on Windows because it is only beneficial on that platform. So this patch shouldn't change non-Windows behavior at all. It's worth noting that non-Windows systems perform working directory updates with multiple processes. Unfortunately, worker.py doesn't yet support Windows. So, there is still plenty of room for making working directory updates faster on Windows. Even if multiple processes are used on Windows, I believe background file closing will still provide a benefit, as individual processes will still be slowed down by the file close bottleneck (assuming the I/O system isn't saturated).

merge: indent code in batchget() To make the next patch easier to read.

localrepo: support background closing for wwrite() So working copy update can pass it in.

scmutil: support background closing for write() Upcoming patches will add background file closer support to working copy update. This patch adds some plumbing to prepare for that.

chg: hold a lock file before connected to server This is a part of the one server per config series. In multiple-server setup, multiple clients may try to start different servers (on demand) at the same time. The old lock will not guarantee a client to connect to the server it just started, and is not crash friendly. This patch addressed above issues by using flock and does not release the lock until the client actually connects to the server or times out.

serve: allow --daemon-postexec to be 'unlink:path' or 'none' This patch changes the format of value of --daemon-postexec. Now it can be either to unlink a file, or a no-op. The following patch will make chg to use the no-op option.

serve: rename --daemon-pipefds to --daemon-postexec (BC) Initially we use --daemon-pipefds to pass file descriptors for synchronization. Later, in order to support Windows, --daemon-pipefds is changed to accept a file path to unlink instead. The name is outdated since then. chg client is designed to use flock, which will be held before starting a server and until the client actually connects to the server it started. The unlink synchronization approach is not so helpful in this case. To address the issues, this patch renames pipefds to postexec and the following patch will allow the value of --daemon-postexec to be things like 'unlink:/path/to/file' or 'none'.

largefiles: don't explicitly list optional parameters that are not used This makes it easier for changes to the API.

revert: properly revert to ancestor of p2 during merge (issue5052) During merge, added (from one perspective) file can be reported as "modified". To work around that, revert was testing if modified file were present in the parent manifest and marking them as "added" in this case. However, we should be checking against the target revision manifest instead. Otherwise see file as "newly added" even if they exist in the target revision. That revert behavior regressed in 06fbd9518bc5.

help: hg.intevation.de is new primary name of hg.intevation.de (and new cert) Adjust the examples (prefix and hostfingerprints) in help/config.txt https://hg.intevation.de/ is now served with a new certificate that is signed by a commercial CA, so all nearly all browsers will accept it automatically. Listing both names, hg.intevation.de and hg.intevation.org, in the section [hostfingerprints] allows using both without configuring web.cacerts and without changing existing https URLs in the [paths] section.

rebase: explicitly test abort from ambiguous destination We want to explicitly test the next behavior. We add this test in its own changeset to make the test change from the behavior change as clean as possible.

rebase: choose default destination the same way as 'hg merge' (BC) This changeset finally make 'hg rebase' choose its default destination using the same logic as 'hg merge'. The previous default was "tipmost changeset on the current branch", the new default is "the other head if there is only one". This change has multiple consequences: - Multiple tests which were not rebasing anything (rebasing from tipmost head) are now rebasing on the other "lower" branch. This is the expected new behavior. - A test is now explicitly aborting when there is too many heads on the branch. This is the expected behavior. - We gained a better detection of the "nothing to rebase" case while performing 'hg pull --rebase' so the message have been updated. Making clearer than an update was performed and why. This is beneficial side-effect. - Rebasing from an active bookmark will behave the same as 'hg merge' from a bookmark.

rebase: add potential divergent commit hashes to error message (issue5086)

hgwebdir_wsgi: update script and expand help I've updated the script to reflect changes in Mercurial and to include a much more through installation guide with configuration examples and details on how to configure IIS. I've used the script to set up a working server from scratch.

bundlerepo: properly handle hidden linkrev in filelog (issue4945) The bundlerepository have to do some special magic to handle linkrev of the bundlerepo filerev. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

rebase: adds storing collapse message (issue4792) Before this patch collapse message wasn't stored so when you ran into the merge conflict while rebasing, running rebase --continue didn't remember the message and always opened editor to fill commit message. This patch adds saving collapse message in .hg/last-message.txt and restoring it later when needed.

test-automv: fix inline config settings for 5ec1ce8fdf0a

automv: use 95 as the default similarity threshold The motivation for the change from 100 to 95 is included in a comment. * Updated the tests to include a change to a moved file that still should be caught as a move. * Use ui.configint() to non-integer configuration entries more gracefully. Also complain if a similarity outside of the acceptable range is set.

bookmarks: add 'hg push -B .' for pushing the active bookmark (issue4917)

worker: change partition strategy to every Nth element The only consumer of the worker pool code today is `hg update`. Previously, the algorithm to partition work to each worker process preserved input list ordering. We'd take the first N elements, then the next N elements, etc. Measurements on mozilla-central demonstrate this isn't an optimal partitioning strategy. I added debug code to print when workers were exiting. When performing a working copy update on a previously empty working copy of mozilla-central, I noticed that process lifetimes were all over the map. One worker would complete after 7s. Many would complete after 12s. And another worker would often take >16s. This behavior occurred for many worker process counts and was more pronounced on some than others. What I suspect is happening is some workers end up with lots of small files and others with large files. This is because the update code passes in actions according to sorted filenames. And, directories under tend to accumulate similar files. For example, test directories often consist of many small test files and media directories contain binary (often larger) media files. This patch changes the partitioning algorithm to select every Nth element from the input list. Each worker thus has a similar composition of files to operate on. The result of this change is that worker processes now all tend to exit around the same time. The possibility of a long pole due to being unlucky and receiving all the large files has been mitigated. Overall execution time seems to drop, but not by a statistically significant amount on mozilla-central. However, repositories with directories containing many large files will likely show a drop. There shouldn't be any regressions due to partial manifest decoding because the update code already iterates the manifest to determine what files to operate on, so the manifest should already be decoded.

run-tests: allow run-tests.py to run tests outside current directory When reloading tests, run-tests.py was assuming that it could look up the test by the basename, which only works if you are running tests which are in the current directory. This patch changes that lookup to use the full path. This is all that was needed, and does not appear to cause any problems for any of the existing testing work flows based on running the suggested commands at the top of run-tests.py. Motivation: In order to test Mercurial with Hypothesis (according to https://www.mercurial-scm.org/wiki/HypothesisPlan) it is useful to be able to generate temporary test files and execute them. Generating temporary files in the tests/ directory leads to a lot of suboptimal clutter.

histedit: unifying the way replacements are computed for abort and success This is a part of a bigger refactoring effort with the ultimate goal of better understanding of how histedit works and fixing issue4800.

templater: factor out type conversion of revset() result This makes it clear why we have to do repo[int(x)].

templater: evaluate each item of revset() as integer revision Because templater.revset() returns a list of strings, repo["-1"] was mapped to the tipmost revision. Ideally, we should make revset() return a list of integer revisions, but it turned out not simple. If revset() is a list of integers, "{ifcontains(rev, revset(), ...)}" would fail because "ifcontains" casts "rev" to a string. So this patch just converts a string back to an integer revision.

demandimport: blacklist sqlalchemy.events as it has side effects (issue5085) Importing sqlalchemy.events cannot be delayed as it registers classes to their event mechanism. It worked fine before 4f1144c3c72b, since they use new-style imports. But now we have to blacklist it because our demandimport can handle new-style imports. This patch series isn't intended for stable as we don't guarantee API compatibility with 3rd-party extensions. They can temporarily disable the demand importer to work around the issue.

demandimport: enforce ignore list while processing modules in fromlist If a module is loaded as "from . import x" form, there has been no way to disable demand loading for that module because name is ''. This patch makes it possible to prevent demand loading by '<package-name>.x'. We don't use _hgextimport(_origimport) here since attr is known to be a sub-module name. Adding hgext_ to attr wouldn't make sense.

phases: use constants for phase values Magic numbers hinder readability.

test-paths: fix up json output to match $TESTTMP on Windows Without this, the paths were in the form "C:\\Users\\Matt\\AppData...". The test runner wouldn't collapse it down to $TESTTMP, which uses single backslashes.

test-obsolete: conditionalize a test dependent on chmod

progress: display progress bar when HGPLAINEXCEPT contains "progress" This patch changes "progress.shouldprint()" so a feature name is provided to "ui.plain()" to determine if there is an exception specificed in HGPLAINEXCEPT for the progress extension. This will allow user-facing scripts to provide progress output while HGPLAIN is enabled.

run-tests: stop allocating HGPORT3+HGPORT4 The only consumer was test-treemanifest.t, which has been fixed. In general, you should be able to use killdaemons.py to recycle ports instead of going over 3 ports (HGPORT, HGPORT1, HGPORT2). In the future, if you want to add a port, be sure to change portneeded in _getport.

run-tests: refactor port allocation into functions Adding a port reservation was too hard and someone did it wrong. By refactoring, such reservations can be managed more safely. This also adds documentation so that the next person who tries is more likely to update all the places correctly. Note that in this commit the reservation and consumers do not match, that will be fixed in the next commit.

tests: put test-treemanifest.t on a port diet test-treemanifest.t had introduced HGPORT3 and HGPORT4, which were improperly added to run-tests.py. It also was not using HGPORT1. This recycles HGPORT, and shifts everything into HGPORT1 + HGPORT2.

chg: pass sensitive command line flags to server We are going to make chgserver load repo config, remember what it is, and load repo config again to detect config change. This is the first step that passes config, repo, cwd options to server. Traceback is passed as well to cover errors before hitting chgserver.runcommand.

chg: use mallocx and reallocx in hgclient This patch simplifies the code a bit, and reduces the binary size a little.

chg: add utility functions mallocx, reallocx They are like malloc and realloc but will abort the program on error. A lot of places use {m,re}alloc and check their results. This patch can simplify them.

localrepo: move new repo requirements into standalone function (API) This patch extracts the code for determining requirements for a new repo into a standalone function. By doing so, future code that will perform an in-place repository upgrade (e.g. to generaldelta) can examine the set of proposed new requirements and possibly take additional actions (such as adding dotencode or fncache) when performing the upgrade. This patch is marked as API because _baserequirements (which was added in b090601a80d1 so extensions could override it) has been removed and will presumably impact whatever extension it was added for. Consumers should be able to monkeypatch the new function to achieve the same functionality. The "create" argument has been dropped because the function is only called in one location and "create" is always true in that case. While it makes logical sense for this code to be a method so extensions can implement a custom repo class / method to override it, this won't actually work. This is because requirements determination occurs during localrepository.__init__ and this is before the "reposetup" "callback" is fired. So, the only way for extensions to customize requirements would be to overwrite localrepo.localrepository or to monkeypatch a function on a module during extsetup(). Since we try to keep localrepository small, we use a standalone function. There is probably room to offer extensions a "hook" point to alter repository creation. But that is scope bloat.

localrepo: isolate requirements determination from side effects In preparation for moving requirements determination to its own function.

localrepo: use local variable for requirements assignment A future patch will refactor requirements determination into a standalone function. To prepare for this, refactor the requirements code to assign to a local variable instead of to self.requirements.

destutil: ensure we offer 'hg update' hint when not at head in all cases In the merge case, we abort if the working copy is not at head, offering to run 'hg update' in the hint instead. In the rebase case, we do not abort in that case. Yet if no rebase destination are found, it still make sense to hint the user about running 'hg update'. So we re-introduce a conditional using this branch in the 'onheadcheck == False' case. This will get used on rebase use this function.

chg: forward umask from client to server This is necessary to make chg test pass on test-inherit-mode.t.

chgserver: add setumask method Before this patch, the server won't inherit umask from the client, which will fail test-inherit-mode.t. This patch provides a way for the client to change umask of the server, similar to chdir and setenv.

chgserver: use _readlist and _readstr Use _readlist and _readstr to make the code shorter.

commandserver: use _readlist Use _readlist introduced in previous commit to make the code shorter.

commandserver: add _readstr and _readlist Reading a string or a list are common operations for commandserver and chgserver. This patch adds _readstr and _readlist to avoid duplication.

extensions: add notloaded method to return extensions failed to load Before this patch, there is no easy way to detect if there are extensions failed to load. While this is okay for most situations, chgserver is designed to preload all extensions specified in config and does need the information. This patch adds extensions.notloaded() to return names of extensions failed to load.

histedit: renaming parts to which _histedit was split I recently broke _histedit into multiple functions, each of which had 'action' in its name. This is a little bit confusing since 'action' is the word for an individual histedit plan item (such as edit or pick). To avoid this confusion, these functions are now renamed to contain 'histedit' as part of their names.

histedit: break _histedit into smaller pieces (add _finishaction) This is a part of a bigger effort to refactor histedit with ultimate goal of understanding it.

automv: switch to specifying the similarity as an integer (0-100) This is consistent with the addremove --similarity option.

automv: do not release lock between marking files and the actual commit

automv: reuse existing scutil._markchanges() function

automv: improve function docstrings

automv: remove test for 'developer mode' The tests no longer use this mode, the test is entirely redundant.

automv: add some context for the tests