changegroup: prune subdirectory dirlogs too We already prune changesets, root manifests and files whose linkrev is in the set of common revisions. We should do the same for dirlogs.

changegroup: include subdirectory manifests in verbose size When verbose logging is one, we report the size in bytes of the manifest data in the changegroup. For files, we report the size per file, but I'm not sure we need that level of detail (i.e. size per directory manifest). Instead, report a single figure for the size of root manifest plus submanifests.

changegroup: make _packmanifests() dumber The next few patches will rewrite the manifest generation code to work with merges. We will then walk dirlogs recursively. This prepares for that by moving much of the treemanifest code out of _packmanifests() and into generatemanifests(). For this to work, it also adds _manifestsdone() method that returns the "end of manifests" close chunk for cg3 and an empty string for cg1 and cg2.

changegroup: extract generatemanifests() The changegroup.generate() function is pretty long, so let's extract the manifest generation part of it.

merge: use any() instead of for loop when checking for dirty subrepos I think it's both simpler and clearer to use any() than the current for loop. While at it, also drop the call to sorted(), since it doesn't matter which order we iterate over subrepos.

templater: fix list templating bug High-level use case: printing a list of objects with formatter when each object in turn contains a list of properties (like when % template symbol is used in {things % '{thing}'} Let the top-level list contain one thing with two properties: objs = [{ 'props': [ { 'value': 1, 'show': 1 }, { 'value': 2 }] }] (please note that second property does not have 'show' key) If a templateformatter is used to print this with template "{props % '{if(show, value)}'}" current implementation will print value for both properties, which is a bug. This happens because in `templater.runmap` function we only rewrite mapping values with existing new values for each item. If some mapping value is missing in the item, it will not be removed.

histedit: also handle locally missing nodes when reading obsolescence The previous version of the code was interpreting markers to a missing node as a prune in all cases. The expected way to handle such situation is to keep reading markers, only turning successors into "prune" if they are at the end of a chain. We update the code and add a test for this.

chgserver: auto exit after being idle for too long or lose the socket file This is a part of the one server per config series. In multiple-server setup, new server may be started for a temporary config change like in command line, --config extensions.foo=bar.py. This may end up with a lot of not so useful server processes. Other questions are about socket file and process management, How to stop these processes? What if a new server wants to listen on a same address, replacing the old one? This patch introduces AutoExitMixIn, which will: 1. Exit after being idle for too long. So useless servers won't run forever. 2. Periodically check the ownership of socket file, exit if it is no longer owned. This brings strong consistency between the filesystem and the process, and handles some race conditions neatly. Since rename is atomic, a new server can just have a same server address with an old one and won't worry about how to make sure the old server is killed, address conflict, service downtime issues. The user can safely stop all servers by simply removing the socket files, without worrying about outdated or accidentally removed pidfiles.

unionrepo: properly handle hidden linkrev in revlog (issue5070) The unionrepository have to do some special magic to handle linkrev of the unioned filerev and manifestrev. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

bundlerepo: properly handle hidden linkrev in manifestlog (issue4945) The bundlerepository have to do some special magic to handle linkrev of the bundled manifest. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

bundlerepo: properly handle hidden linkrev in filelog (issue4945) The bundlerepository have to do some special magic to handle linkrev of the bundlerepo filerev. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

check-code: allow old style class with special comments The following chgserver change will use an old style class to comply with SocketServer's code style. This patch made it possible to pass check-code.

merge with stable

revset: use smartset minus operator Previously, revsets like 'X - Y' were translated to be 'X and not Y'. This can be expensive, since if Y is a single commit then 'not Y' becomes a huge set and sometimes the query optimizer doesn't account for it well. This patch changes revsets to use the built in smartset minus operator, which is often smarter than 'X and not Y'. On a large repo this saves 2.2 seconds on rebase and histedit because "X:: - X" becomes almost instant. Relevant performance numbers from revsetbenchmark.py revset #13: roots((tip~100::) - (tip~100::tip)) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.001080 0.001107 0.001102 0.001118 0.001121 0.001114 0.001141 0.001123 0.001099 0.001123 0.001137 1) 0.000708 65% 0.000738 66% 0.000735 66% 0.000739 66% 0.000784 69% 0.000780 70% 0.000807 70% 0.000756 67% 0.000727 66% 0.000759 67% 0.000808 71% revset #14: roots((0::) - (0::tip)) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.131304 0.079168 0.133129 0.076560 0.048179 0.133349 0.049153 0.077097 0.129689 0.076212 0.048543 1) 0.065066 49% 0.036941 46% 0.066063 49% 0.034755 45% 0.048558 0.071091 53% 0.047679 0.034984 45% 0.064572 49% 0.035680 46% 0.048508 revset #22: (not public() - obsolete()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.000139 0.000133 0.000133 0.000138 0.000134 0.000155 0.000157 0.000152 0.000157 0.000156 0.000153 1) 0.000108 77% 0.000129 0.000129 0.000134 0.000132 0.000127 81% 0.000151 0.000147 0.000127 80% 0.000152 0.000149 revset #25: (20000::) - (20000) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.050560 0.045513 0.022593 0.043588 0.021909 0.045517 0.021822 0.044660 0.049740 0.044227 0.021819 1) 0.018614 36% 0.000171 0% 0.019659 87% 0.000168 0% 0.015543 70% 0.021069 46% 0.015623 71% 0.000180 0% 0.018658 37% 0.000186 0% 0.015750 72%

histedit: make histedit aware of obsolescense not stored in state (issue4800) Before this change, when histedit exited to interactive session (during edit command for example), user could introduce obsolescence markers that would not be known to histedit. For example, user could've amended one of the commits. The fact of this amendment would not be stored in histedit's state file and later, when histedit would try to process all the replacements, one of the final successors (in histedit's opinion) would turn out to be hidden. This behavior is described in issue4800. This commit fixes it.

treemanifest: allow setting flag to 't' When using treemanifests, an on-disk manifest entry with the 't' flag set means that that entry is a directory and not a file. When read into memory, these become instances of the treemanifest class. The 't' flag should therefore never be visible to outside of manifest.py, so setflag() checks that it is not called with the 't' flag. However, it turns out that it will be useful for the narrowhg extension to expose the 't' flag to the user (see below), so let's drop the assertion. The narrowhg extension allows cloning only a given set of files and directories. Filelogs and dirlogs that don't match that set will not be included in the clone. The extension currently doesn't work with treemanifests. I plan on changing it so directories outside the narrow clone appear in the manifest. For example, if a directory 'outside/' is not part of the narrow clone, it will look like a file 'outside' with the 't' flag set. That will make e.g. manifestmerge() just work in most cases (and make it well prepared to handle the other cases).

treemanifest: use "cp xyz/." instead of "cp xyz/*" This is more similar to cp -T because it covers hidden files.

templatefilters: drop old jsonescape() function It's been superseded by encoding.jsonescape(paranoid=True).

templatefilters: make json filter be byte-transparent (BC) (issue4926) This is necessary to preserve filename encoding over JSON. Instead, this patch inserts "|utf8" where non-ascii local-encoding texts can be passed to "|json". See also the commit that introduced "utf8" filter.

hgweb: add option to convert encoding of graphdata() Because future patches will change "|json" filter to handle input bytes transparently, i.e. use UTF-8b encoding, "{jsdata}" must keep data in UTF-8 bytes, whereas "{nodes}" are text. This patch inserts encodestr() where localstr is likely to survive.

hgweb: remove unused argument from graphdata() factory As graphdata() is wrapped by lambda, there's no reason to pass unused arguments to it.

templatefilters: add "utf8" to get utf-8 bytes from local-encoding text This will be applied prior to "|json" filter. This sounds like odd, but it is necessary to handle local-encoding text as well as raw filename bytes. Because filenames are bytes in Mercurial and Unix world, {filename|json} should preserve the original byte sequence, which implies {x|json} -> '"' toutf8b(x) '"' On the other hand, most template strings are in local encoding. Because "|json" filter have to be byte-transparent to filenames, we need something to annotate an input as a local string, that's what "|utf8" will do. {x|utf8|json} -> '"' toutf8b(fromlocal(x)) '"' "|utf8" is an explicit call, so aborts if input bytes can't be converted to UTF-8.

templatefilters: drop broken "jsonescape" from filters table (BC) It's been unused, undocumented and flawed in that it expects a unicode input, never works correctly if an input has non-ascii character. We should use "json" filter instead.

treemanifest: rewrite text() using iterentries() This simplifies a bit. Note that the function is only used when manually testing with _treeinmem=True.

treemanifest: implement iterentries() To make tests pass with _treeinmem manually set to True, we need to implement the recently added iterentries() on the treemanifest class too.

verify: show progress while verifying dirlogs In repos with treemanifests, the non-root-directory dirlogs often have many more total revisions than the root manifest log has. This change adds progress out to that part of 'hg verify'. Since the verification is recursive along the directory tree, we don't know how many total revisions there are at the beginning of the command, so instead we report progress in units of directories, much like we report progress for verification of files today. I'm not very happy with passing both 'storefiles' and 'progress' into the recursive calls. I tried passing in just a 'visitdir(dir)' callback, but the results did not seem better overall. I'm happy to update if anyone has better ideas.

verify: check for orphaned dirlogs We already report orphaned filelogs, i.e. revlogs for files that are not mentioned in any manifest. This change adds checking for orphaned dirlogs, i.e. revlogs that are not mentioned in any parent-directory dirlog. Note that, for fncachestore, only files mentioned in the fncache are considered, there's not check for files in .hg/store/meta that are not mentioned in the fncache. This is no different from the current situation for filelogs.

verify: check directory manifests In repos with treemanifests, there is no specific verification of directory manifest revlogs. It simply collects all file nodes by reading each manifest delta. With treemanifests, that's means calling the manifest._slowreaddelta(). If there are missing revlog entries in a subdirectory revlog, 'hg verify' will simply report the exception that occurred while trying to read the root manifest: manifest@0: reading delta 1700e2e92882: meta/b/00manifest.i@67688a370455: no node This patch changes the verify code to load only the root manifest at first and verify all revisions of it, then verify all revisions of each direct subdirectory, and so on, recursively. The above message becomes b/@0: parent-directory manifest refers to unknown revision 67688a370455 Since the new algorithm reads a single revlog at a time and in order, 'hg verify' on a treemanifest version of the hg core repo goes from ~50s to ~14s. As expected, there is no significant difference on a repo with flat manifests.

hg: perform update after pulling during clone with share (issue5103) When pooled storage is enabled, `hg clone` will initialize a repo from a local repo using the store sharing mechanism then pull from the originally requested repo. Before this patch, the working directory update occurred between these steps. This meant that we would only update to revisions that were already present in the local pooled storage. This patch moves the update to after we pull from the originally requested repository so we may check out a revision that didn't yet exist locally. In other words, it makes the behavior like normal `hg clone`.

hg: extract post share update logic into own function A future patch will introduce a new caller that needs to perform an update. Extract the code so we don't duplicate it.