hg: write narrow patterns after repo creation Now that hg.clone() knows when a narrow clone is requested, it makes sense to have it update the narrow patterns for the repo soon after the repo is created, before any exchange occurs. Previously, the narrow extension was monkeypatching an exchange function to do this. The old code is redundant and has been removed. Differential Revision: https://phab.mercurial-scm.org/D4541

narrow: don't wrap exchange.pull() during clone The wrapped version was setting up the narrow repo requirement when a narrow clone was requested. Previous commits taught hg.clone() and repo creation to add the narrow requirement when a narrow clone was requested. So this requirement should already be set up for us and this code is no longer necessary. Differential Revision: https://phab.mercurial-scm.org/D4540

exchange: support defining narrow file patterns for pull This commit teaches exchange.pull() about the desire to perform a narrow file pull. We simply pass include and exclude patterns to the function. The values are validated and stored on the pulloperation instance. hg.clone() has been taught to pass these arguments to exchange.pull(). If the arguments are not passed to exchange.pull(), the active narrow patterns from the repository will automatically be used. We /could/ always use the narrow patterns from the repo. However, allowing explicit values to be passed in allows us to perform data fetching that doesn't necessarily align with the repo configuration. This provides more flexibility. Differential Revision: https://phab.mercurial-scm.org/D4539

commands: pass include and exclude options to hg.clone() These arguments are defined by the narrow extension. Let's teach core to recognize them so we can delete some code from the narrow extension and start to exercise the in-core code for performing a narrow clone. We have no way of easily testing it, but this change should result in .hg/requires having the narrow requirement from the time the file is written rather than added as part of pull. We'll confirm this when we delete some monkeypatched functions from the narrow extension in later commits. Test output changed because hg.clone() is now receiving patterns and validation of those values is occurring sooner, before the exchange code runs and prints the message that was deleted. Differential Revision: https://phab.mercurial-scm.org/D4538

localrepo: add requirement when narrow files creation option present The previous commit taught hg.clone() to define a creation option when file include or exclude patterns are passed. This commit teaches the new repo creation code to convert that creation option into a repository requirement. While not yet used by the narrow extension, the eventual side-effect of this change is that newly-created repositories will have the narrow requirement from their creation onset. Currently, the requirement is added to the repo at exchange.pull() time via a wrapped function in the narrow extension. Differential Revision: https://phab.mercurial-scm.org/D4537

hg: recognize include and exclude patterns when cloning This commit teaches clone() to accept arguments defining file patterns to clone. This is the first step in teaching core code about the existence of a narrow clone. Right now, we only perform validation of the arguments and pass additional options into createopts to influence repository creation. Nothing of consequence happens with that creation option yet, however. For now, arbitrary restrictions exist, such as not allowing patterns for shared repos and disabling local copies when patterns are defined. We can potentially lift these restrictions in the future once partial clone/storage support is more flushed out. I figure it is best to reduce the surface area for bugs for the time being. It may seem weird to prefix these arguments with "store." However, clone is effectively pull + update and file patterns could apply to both the store and the working directory. The prefix is there to disambiguate in the future when this function may want to use different sets of patterns for the store and working directory. Differential Revision: https://phab.mercurial-scm.org/D4536

hg: allow extra arguments to be passed to repo creation (API) Currently, repository creation is influenced by consulting the ui instance and turning config options into requirements. This means that in order to influence repository creation, you need to define and set a config option and that the option must translate to a requirement stored in the .hg/requires file. This commit introduces a new mechanism to influence repository creation. hg.repository() and hg.peer() have been taught to receive a new optional argument defining extra options to apply to repository creation. This value is passed along to the various instance() functions and can be used to influence repository creation. This will allow us to pass rich data directly to repository creation without having to go through the config layer. It also allows us to be more explicit about the features requested during repository creation and provides a natural point to detect unhandled options influencing repository creation. The new code detects when unknown creation options are present and aborts in that case. .. api:: options can now be passed to influence repository creation The various instance() functions to spawn new peers or repository instances now receive a ``createopts`` argument that can be a dict defining additional options to influence repository creation. localrepo.newreporequirements() also receives this argument. Differential Revision: https://phab.mercurial-scm.org/D4535

localrepo: move repo creation logic out of localrepository.__init__ (API) It has long bothered me that local repository creation is handled as part of localrepository.__init__. Upcoming changes I want to make around how repositories are initialized and instantiated will make the continued existence of repository creation code in localrepository.__init__ even more awkward. localrepository instances are almost never constructed directly: instead, callers are supposed to go through hg.repository() to obtain a handle on a repository. And hg.repository() calls localrepo.instance() to return a new repo instance. This commit teaches localrepo.instance() to handle the create=True logic. Most of the code for repo construction has been moved to a standalone function. This allows extensions to monkeypatch the function to further customize freshly-created repositories. A few calls to localrepo.localrepository.__init__ that were passing create=True were converted to call localrepo.instance(). .. api:: local repo creation moved out of constructor ``localrepo.localrepository.__init__`` no longer accepts a ``create`` argument to create a new repository. New repository creation is now performed as part of ``localrepo.instance()`` and the bulk of the work is performed by ``localrepo.createrepository()``. Differential Revision: https://phab.mercurial-scm.org/D4534

subrepo: mask out passwords embedded in the messages displaying a URL I noticed the password in maintenance logs for the "no changes since last push" and "pushing to" messages when pushing with an explicit path. But the test case here with :pushurl was also affected. I didn't see that cloning or pulling subrepos on demand had this problem, but it seems safer to just mask that too. There's a bit of a disconnect here, because it looks like clone is slicing off the password (makes sense not to store it in the hgrc in cleartext). But not shearing it off of an explicit path causes the subrepo not to realize that it already pushed the latest stuff. This is the easiest fix, however.

localrepo: pass ui to newreporequirements() (API) newreporequirements() is called as part of creating a new repository. It doesn't make much sense for it to receive a repo instance as part of determining what requirements for new repos should be. .. api:: localrepo.newreporequirements() receives a ui instead of a repo Differential Revision: https://phab.mercurial-scm.org/D4533

narrow: set opts['narrow'] instead of local variable This will allow the command function in core to infer the presence of the option without duplicating logic. Differential Revision: https://phab.mercurial-scm.org/D4532

narrow: drop support for remote expansion (BC) Previous patches to validate narrow patterns accidentically dropped support for the include: syntax that allows patterns to be expanded from a remote. This feature was never implemented in core and is only implemented on Google's custom server. Per @martinvonz's review comment in D4522, it is OK to drop this feature since it isn't used. The concept of this feature does seem useful. I anticipate it making a comeback some day in some shape or form. But for now, let's jettison the dead code. Differential Revision: https://phab.mercurial-scm.org/D4530

fastannotate: use repo.local() This is the proper way to check whether we're dealing with a local repository, since extensions should be coding to an interface and not testing for exact types. Differential Revision: https://phab.mercurial-scm.org/D4542

tests: drop extra "file:" prefix from paths in narrow test It looks like these were added by mistake in f4d4bd8c8911 (narrow: add a --narrowspec flag to clone command, 2018-08-08). Differential Revision: https://phab.mercurial-scm.org/D4531

narrow: validate spec files are well-formed during clone (BC) Previously, specfiles would get read then normalized. We want specfiles to be normalized on read so there is no confusion about what the format of specfiles should be. This commit validates the parsed result of --specfile. If entries aren't prefixed, an error is raised. Previously, validation would occur at exchange time, hence why we dropped a line of test output related to server iteraction. Differential Revision: https://phab.mercurial-scm.org/D4526

narrow: validate patterns on incoming bundle2 part The remote data is untrusted and needs to be validated for pattern conformance. Differential Revision: https://phab.mercurial-scm.org/D4525

narrowspec: validate patterns when loading and saving spec file Patterns should be normalized and validated before being passed into narrowspec.save(). Let's assert that by checking immediately before writing the narrow spec file. And let's assert that patterns loaded from the spec file also conform. Differential Revision: https://phab.mercurial-scm.org/D4524

ancestor: use heapreplace() in place of heappop/heappush() This should be slightly faster. Overall perfancestors result:: cpython nginx mercurial ------------- ---------------- ---------------- ---------------- b6db2e80a9ce^ 0.103461 0.006303 0.035716 8eb2145ff0fb 0.192307 (x1.86) 0.012115 (x1.92) 0.052135 (x1.46) this patch 0.139986 (x1.35) 0.006389 (x1.01) 0.037176 (x1.04)

ancestor: rename local aliases of heapq functions in _lazyancestorsiter() The original names no longer look pretty. Just call them as heap*() instead.

ancestor: optimize _lazyancestorsiter() for contiguous chains If there's no revision between p1 and current, p1 must be the next revision to visit. In this case, we can get around the overhead of heappop/push operations. Note that this is faster than using heapreplace(). 'current - p1 == 1' could be generalized as 'all(r not in seen for r in xrange(p1, current)', but Python is too slow to do such thing.

ancestor: unroll loop of parents in _lazyancestorsiter() This change itself isn't major performance win, but it helps optimizing the visit loop for contiguous chains. See the next patch.

ancestor: return early from _lazyancestorsiter() when reached to stoprev There's no need to empty the heap.

ancestor: remove alias of initrevs from _lazyancestorsiter() It's just redundant and less comprehensible.

narrow: validate patterns returned by expandnarrow Remotes could supply malicious or invalid patterns. We should validate them as soon as possible. Differential Revision: https://phab.mercurial-scm.org/D4523

narrowspec: limit patterns to path: and rootfilesin: (BC) Some matcher patterns are computationally expensive and may even have security issues (e.g. evaluating some file sets). For these reasons, we want to limit the types of matcher patterns that can be used in narrow specs and by command line arguments used for defining narrow specs. This commit teaches ``narrowspec.parsepatterns()`` to validate the pattern types against "safe" patterns. Surprisingly, no existing tests broke. So tests for the feature have been added. We also added a function to validate a patterns data structure. This will be used in future commits. Differential Revision: https://phab.mercurial-scm.org/D4522

narrow: mark wire proto capability names experimental and versioned We already plan to add a "widen" wire protocol command to the "narrow" capability, so let's version the capabilities as "exp-narrow-1" and "exp-ellipses-1". When we add the "widen" command, we will then add a "exp-narrow-2" capability to indicate support for that command. Differential Revision: https://phab.mercurial-scm.org/D4529

narrow: move wire proto capabilities to narrowwirepeer These are not bundle2 capabilities (they just happened to share the name "narrow"), so they seem to belong with the wirepeer overrides. Differential Revision: https://phab.mercurial-scm.org/D4528

narrow: check "narrow" wire protocol capability, not bundle2 capability It seems like the new "narrow" wire protocol capability should be what determines if the server supports the "narrow" and "{,old}{in,ex}cludepats" arguments to the getbundle request. Differential Revision: https://phab.mercurial-scm.org/D4527

sparse: add local files to temporaryfiles if they exist out of sparse We get the f1 from args if it's merge and check that whether that exists in sparse checkout or not. If that does not, we add that for merging. The error comes from very low-level where we try to read data of a working-filectx which does not exists in the working directory. It will be extremely ugly to plug in logic to update sparse copy with new file at such a low level. We already have logic related to updating the checkout with required files in calculateupdates() and let's handle this case there only. calculateupdates() call sparse.filterupdatesactions() and the logic is added into the latter function. To get the exact traceback, this patch can be backed out and test-sparse-merges.t can be run with ui.traceback=True. Differential Revision: https://phab.mercurial-scm.org/D4341

tests: show that merging with sparse is broken when rename is involved This patch adds test to show that merging with sparse is broken when you have a rename on one side and just modification on another side. Differential Revision: https://phab.mercurial-scm.org/D4340