exchange: introduce a '_canusebundle2' function This function refactors the logic that decides to use 'bundle2' during an exchange (pull/push). This will help being consistent while transitioning from the experimental protocol to the final frozen version. I do not expect this function to survive on the long run when using 'bundle2' will become a simple capability check. This is also necessary to allow HG2Y support in an extension to ease transition of companies using the experimental protocol in production (yeah...). Such extension will be able to wrap this function to use the experimental protocol in some case.

bundle2: detect bundle2 stream/request on /HG2./ instead of /HG2Y/ To support more bundle2 formats, we need a wider detection of bundle2-family streams. The various places what were explicitly detecting the full magic string are now matching on the first three characters of it.

unbundle20: allow generic dispatch between unbundlers We now take full advantage of the 'getunbundler' function by using a '{version -> unbundler-class}' mapping. This map currently contains a single entry but will make it easy to support more versions from an extension/the future. At some point, this map will probably contain bundler-class information too, in the same fashion the packer map does. However, this is not critically required right now so it will happen by itself when needed. The main target is to allow HG2Y support in an extension to ease transition of companies using the experimental protocol in production (yeah...) But I've no doubt this will be useful when playing with a future HG21.

treemanifest: refactor treemanifest.walk() This refactor is a preparation for an optimization in the next commit. This introduces a recursive element that recurses each submanifest. By using a recursive function, the next commit can avoid walking over some subdirectories altogether.

manifest: move changectx.walk() to manifests The logic of walking a manifest to yield files matching a match object is currently being done by context, not the manifest itself. This moves the walk() function to both manifestdict and treemanifest. This separate implementation will also permit differing, optimized implementations for each manifest.

subrepo: precisely identify the missing subrepo spec file It isn't obvious which file is the problem with deep subrepos, so provide the path. Since the parsing is done with a ctx and not a subrepo object, it isn't possible to display a path from the root subrepo. Therefore, the path shown is relative to cwd. There's no test coverage for the first abort, and I couldn't figure out how to trigger it, but it is changed for consistency.

graft: record intermediate grafts in extras Previously the extra field for a graft only contained the original commit hash. This made it impossible to use graft to copy a commit more than once, because the extras fields did not change after the second graft. The fix is to add an extra.intermediate-source field that records the immediate predecessor to graft. This changes hashes for commits that have been grafted twice, which is why the test was affected.

graft: allow creating sibling grafts Previously it was impossible to graft a commit onto it's own parent (i.e. create a copy of the commit). This is useful when wanting to create a backup of the commit before continuing to amend it. This patch enables that behavior. The change to the histedit test is because histedit uses graft to apply commits. The test in question moves a commit backwards onto an ancestor. Since the graft logic now more explicitly supports this, it knows to simply accept the incoming changes (since they are more recent), instead of prompting.

unbundle20: move header parsing into the 'getunbundler' function The dispatching will be based on the header content, so we need to move this logic into the factory function.

unbundle20: retrieve unbundler instances through a factory function To support multiple bundle2 formats, we will need a function returning the proper unbundler according to the header. We introduce such aa function and change the usage in the code base. The function will get smarter in later changesets. This is somewhat similar to the dispatching we do for 'HG10' and 'HG11'. The main target is to allow HG2Y support in an extension to ease transition of companies using the experimental protocol in production (yeah...) But I've no doubt this will be useful when playing with a future HG21.

bundle20: move magic string into the class This makes it easy to create a new bundler class that inherits from the core one. This matches the way 'changegroup' packers work. The main target is to allow HG2Y support in an extension to ease transition of companies using the experimental protocol in production (yeah...) But I've no doubt this will be useful when playing with a future HG21.

localrepo.getbundle: drop unused 'format' argument The 'format' argument was not used even when it was added in 60ad2ea5b106 (getbundle: pass arbitrary arguments all along the call chain, 2014-04-17). Note that by removing the argument, if any caller did pass a named 'format' argument, we will now pass that along to exchange.getbundle() via the kwargs. If the idea was to remove such a key, that should have been done explicitly.

exchange: remove check for 'format' key When the 'kwargs' variable was added in 12f161f08d74 (bundle2: allow pulling changegroups using bundle2, 2014-04-01), it could contain only 'bundlecaps', 'common' and 'heads', so the check for 'format' would always be false. Since then, _pullbundle2extraprepare() has been added for hooks, but it seems unlikely that they would a 'format' key.

templates-default: do not show description or summary if empty changeset_printer shows description only if ctx.description().strip() is not empty. The default template should do the same way.

treemanifest: further optimize treemanifest.matches() The matches function was previously traversing all submanifests to look for matching files, even though it was possible to know if a submanifest won't contain any matches. This change adds a visitdir function on the match object to decide quickly if a directory should be visited when traversing. The function also decides if _all_ subdirectories should be traversed. Adding this logic as methods on the match object also makes the logic modifiable by extensions, such as largefiles. An example of a command this speeds up is running hg status --rev .^ python/ on the Mozilla repo with the treemanifest experiment enabled. It goes from 2.03s to 1.85s. More improvements to speed from this change will happen when treemanifests are lazily loaded. Because a flat manifest is still loaded and then converted into treemanifests, speed improvements are limited. This change has no negative effect on speed. For a worst-case example, this command is not negatively impacted: hg status --rev .^ 'relglob:*.js' on the Mozilla repo. It goes from 2.83s to 2.82s.

util: move dirs() and finddirs() from scmutil to util An upcoming commit requires that match.py be able to call scmutil.dirs(), but when match.py imports scmutil, a dependency cycle is created. This commit avoids the cycle by moving dirs() and its related finddirs() function from scmutil to util, which match.py already depends on.

parsers: remove unused dependency on util Parsers.py had a reference to util.sha1 which was unused. This commit removes this reference as well as the unused import of util to simplify the dependency graph. This is important for the next commit which actually relocates part of a module to eliminate a cycle.

changectx.walk: drop unnecessary call to match function If all the files in match.files() are in the context/manifest, we already know that the matcher will match each file.

merge with stable

largefiles: use the share source as the primary local store (issue4471) The benefit of retargeting the local store to the share source is that all shares will always have access to the largefiles any one of them commit, even if the user cache is deleted (which is documented to be OK to do). Further, any push into the source (and now any shares), will likewise make the largefile(s) visible to all related repositories. In order to maintain compatibility with existing repos, where the largefiles would be cached only in the local share, fallback to searching the local share if it isn't found at the share source. The unshare command should probably be taught to copy the source store into the store for the repo being unshared to complete the loop. This patch changes the test like this: @@ -159,6 +159,5 @@ $ hg share -q src share_dst --config extensions.share= $ hg -R share_dst update -r0 getting changed largefiles - large: largefile $HASH not available from file:///$TESTTMP\share_dst - 0 largefiles updated, 0 removed + 1 largefiles updated, 0 removed 1 files updated, 0 files merged, 0 files removed, 0 files unresolved The issue writeup mentions pushing a largefile from a remote repo to the main local repo, and the largefile is then not available in any shares. Since the push doesn't cache the largefile in $USERCACHE, the trashed $USERCACHE in this test is equivalent.

largefiles: use lfutil.findstorepath() when verifying a local repo

largefiles: introduce lfutil.findstorepath() The handful of direct uses of lfutil.storepath() merely need a single path to read from or write to the largefile, whether or not it exists. Most callers that care about the file existing call lfutil.findfile(), in order to fallback from the store to the user cache. localstore._verify() doesn't call lfutil.findfile(). This prevents redirecting the store to the share source because the largefiles for existing repos may not be in the source's store, so verification may fail. It can't be changed to call findfile(), because findfile() links the file from the usercache to the local store[1], and because it returns None instead of a path if the file doesn't exist. For now, this method is just a cover for lfutil.storepath(), but it will be filled out in an upcoming patch. [1] Maybe we shouldn't care? But on a filesystem that doesn't support hardlinks, then verify will take a lot longer, and start to consume disk space.

vfs: make it possible to pass multiple path elements to join os.path.join(), localrepo.join() and localrepo.wjoin() allow passing multiple path elements; vfs.join() should be as convenient.

largefiles: drop os.path reference in lfutil.storepath() localrepo.join() can concatenate multiple parts on its own.