exchange: move code for generating a streaming clone into exchange Streaming clones are fast because they are essentially tar files. On mozilla-central, a streaming clone only consumes ~55s CPU time on clients as opposed to ~340s CPU time for a regular clone or gzip bundle unbundle. Mozilla is deploying static file "lookaside" support to our Mercurial server. Static bundles are pre-generated and uploaded to S3. When a clone is performed, the static file is fetched, applied, and then an incremental pull is performed. Unfortunately, on an ideal network connection this still takes as much wall and CPU time as a regular clone (although it does save significant server resources). We like the client-side wall time wins of streaming clones. But we want to leverage S3-based pre-generated files for serving the bulk of clone data. This patch moves the code for producing a "stream bundle" into its own standalone function, away from the wire protocol. This will enable stream bundle files to be produced outside the context of the wire protocol. A bikeshed on whether exchange is the best module for this function might be warranted. I selected exchange instead of changegroup because "stream bundles" aren't changegroups (yet).

dirstate: avoid match.files() in walk()

match: introduce boolean prefix() method tl;dr: This is another step towards a (previously unstated) goal of eliminating match.files() in conditions. There are four types of matchers: * always: Matches everything, checked with always(), files() is empty * exact: Matches exact set of files, checked with isexact(), files() contains the files to match * patterns: Matches more complex patterns, checked with anypats(), files() contains roots of the matched patterns * prefix: Matches simple 'path:' patterns as prefixes ('foo' matches both 'foo' and 'foo/bar'), no single method to check, files() contains the prefixes to match For completeness, it would be nice to have a method for checking for the "prefix" type of matcher as well, so let's add that, making it return True simply when none of the others do. The larger goal here is to eliminate uses of match.files() in conditions (i.e. bool(match.files())). The reason for this is that there are scenarios when you would like to create a "prefix" matcher that happens to match no files. One example is for 'hg files -I foo bar'. The narrowmatcher also restricts the set of files given and it would not surprise me if have bugs caused by that already. Note that 'if m.files() and not m.anypats()' and similar is sometimes used to catch the "exact" and "prefix" cases above.

hgweb: descend empty directories in monoblue The ability to "skip" a chain of empty directories in hgweb was added in c21d236ca897, but monoblue style wasn't updated. This block is copied from gitweb/map file and just works.

match: have visitdir() consider includes and excludes match.visitdir() used to only look at the match's primary pattern roots to decide if a treemanifest traverser should descend into a particular directory. This change logically makes visitdir also consider the match's include and exclude pattern roots (if applicable) to make this decision. This is especially important for situations like using narrowhg with multiple treemanifest revlogs.

hgweb: remove artificial width constraint from header in monoblue This width property comes from the beginning of the monoblue theme itself, and was used to stop the action header ("summary", "shortlog", "changelog", etc) from clashing with the search form. But it still was happening (on smaller screens, and with more actions added to hgweb over time). Effectively, the hardcoded width was preventing the header from fitting into the available screen space, since it always tried to be 900px wide, even if that meant horizontal scroll on smaller screens and having the actions on two lines where one should've been enough. For example, http://selenic.com/hg/log?style=monoblue has the last two actions ("gz" and "help") in the header on the second line, even when there seems to be enough space on the first. This patch makes the form float, which prevents it from overlaying/clashing with the action header, and allows the latter to resize itself in the best possible way.

merge with stable

files: recurse into subrepos automatically with an explicit path

dirstate: use open/read of vfs(opener) explicitly instead of read This simplifies changes in subsequent patch, which tries to open `.pending` file when HG_PENDING environment variable is defined.

dirstate: use self._filename instead of immediate string `dirstate` This prevents immediate string `dirstate` from multiplying. This is also a preparation for making dirstate aware of PENDING mechanism in subsequent patches.

revset: drop translation marker from error message of _notpublic() It is a kind of an internal error. End user won't see it.

revset: drop docstring from internal _notpublic() function It shouldn't be listed in "hg help revset".

record: make hg record always use the non curses interface Before this patch, hg record was running hg commit -i, therefore with the experimental.crecord=True flag, hg record was actually launching the curses record interface. Some of our users could be confused by that. This patch makes the hg record command set this flag to False, ensuring that hg record never shows the curses interface. commit -i, shelve -i and revert -i remain unchanged and use the curses interface if the experimental.crecord flag is set.

treemanifest: lazily load manifests Most operations on treemanifests already visit only relevant submanifests. Notable examples include __getitem__, __contains__, walk/matches with matcher, diff. By making submanifests lazily loaded, we speed up all these operations. The lazy loading is achieved by adding a _load() method that gets defined where we currently eagerly parse the manifest. We make sure to call it before any access to _dirs, _files or _flags. Some timings on the Mozilla repo (with flat manifest timings for reference): hg cat -r . README.txt: 1.644s -> 0.096s (0.255s) hg diff -r .^ -r . : 1.746s -> 0.137s (0.431s) hg files -r . python : 1.508s -> 0.146s (0.335s) hg files -r . : 2.125s -> 2.203s (0.712s)

treemanifest: speed up commit using dirty flag We currently avoid saving a treemanifest revision if it's the same as one of it's parents. This is checked by comparing the generated text for all three versions. Let's avoid that when possible by comparing the nodeids for clean (not dirty) nodes. On the Mozilla repo, this speeds up commit from 2.836s to 2.343s.

treemanifest: speed up diff by keeping track of dirty nodes Since tree manifests have a nodeid per directory, we can avoid diffing entire directories if they have the same nodeid. The comparison is only valid for unmodified treemanifest instances, of course, so we need to keep track of which have been modified. Therefore, let's add a dirty flag to treemanifest indicating whether its nodeid can be trusted. We set it when _files or _dirs is modified, and make diff(), and its cousin filesnotin(), not descend into subdirectories that are the same on both sides. On the Mozilla repo, this speeds up 'hg diff -r .^ -r .' from 1.990s to 1.762s. The improvement will be much larger when we start lazily loading subdirectory manifests.

localrepo: use correct argument name for pretxnclose hooks (BC) Before this patch, "the reason for the transaction" is passed to `pretxnclose` hooks via wrong name argument `xnname` (`HG_XNNAME` for external hooks)

localrepo: rename hook argument from TXNID to txnid (BC) From the first (3.4 or d283517b260b), `TXNID` is passed to Python hooks without lowering its name, but it is wrong.

test-walk: add more tests for -I/-X We had very limited testing of -I and -X, especially when combined with plain file patterns and with each other. This adds some more protection against regressions as upcoming patches modify the matcher code. (Originally meant for my own upcoming patches, but now I know drgott will be sending some patches soon.) The only noteworthy cases seems to be that both of hg debugwalk -Xbeans/black beans/black hg debugwalk -Xbeans beans/black walk the file. I would personally have expected the -X to trump. I don't care enough to change it, but I also think it's fair if some future commit changes the behavior.

ignore: use 'include:' rules instead of custom syntax Now that the matcher supports 'include:' rules, let's change the dirstate.ignore creation to just create a matcher with a bunch of includes. This allows us to completely delete ignore.py. I moved some of the syntax documentation over to readpatternfile in match.py so we don't lose it.

match: add 'include:' syntax This allows the matcher to understand 'include:path/to/file' style rules. The files support the standard hgignore syntax and any rules read from the file are included in the matcher without regard to the files location in the repository (i.e. if the included file is in somedir/otherdir, all of it's rules will still apply to the entire repository).

match: add optional warn argument Occasionally the matcher will want to print warning messages instead of throwing exceptions (like if it encounters a bad syntax parameter when parsing files). Let's add an optional warn argument that can provide this. The next patch will actually use this argument.

match: add source to kindpats list Future patches will be adding the ability to recursively include pattern files in a match rule expression. Part of that behavior will require tracking which file each pattern came from so we can report errors correctly. Let's add a 'source' arg to the kindpats list to track this. Initially it will only be populated by listfile rules.

check-code: reintroduce str.format() ban for 3.x porting In their infinite wisdom, the Python maintainers stripped bytes of its % and format() methods for 3.x. They've now added % back to 3.5, but format() is still missing. Since we don't have any particular need for it, we should keep avoiding it.