fix: remove unused imports Differential Revision: https://phab.mercurial-scm.org/D2977

infinitepush: introduce server option to route every push to bundlestore This patch introduces a new config option for server `infinitepush.pushtobundlestore` which if sets to True, the server will route each incoming push to the bundlestore and store all the parts i.e. changegroups, phases, obsmarkers in the bundlestore and won't be applied to the revlog. This config option does not need any client side wrapping and does not need any custom bundle2 part or stream level parameter to decide where the push should go. This is very useful for Mozilla CI use case where they have a central server that recieves pushes to trigger code-reviews, trigger a test run of CI, run static analysis etc. The server using the new config option can stash standalone bundles to the bundlestore and server can get access to individual revisions on demand. A new test file which has related tests are added for the config option. Differential Revision: https://phab.mercurial-scm.org/D2958

infinitepush: don't wrap bundle2.processparts while calling `hg unbundle` This patch adds dirty logic to check whether we are processing `hg unbundle` instead of an `hg incoming` to prevent the wrapping of bundle2.processparts function. Differential Revision: https://phab.mercurial-scm.org/D2957

inifnitepush: use utils.{stringutil|procutil}.* instead of util.* Recently in core, util.py is splitted into various modules in mercurial/utils/. Differential Revision: https://phab.mercurial-scm.org/D2956

infinitepush: delete the non-forward-move flag for hg push Differential Revision: https://phab.mercurial-scm.org/D2955

infinitepush: drop the wrapping of update command on client side The extension wraps update command to client side to add functionality to pull and checkout to a revision if it does not present locally. There is a twist to that, only changesets which can resolve to remotenames can be pulled using this. Differential Revision: https://phab.mercurial-scm.org/D2954

infinitepush: delete infinitepushcommands.py and related tests This patch drops infinitepushcommands.py and the tests which were related to the command `debugfillinfinitepushmetadata` introduced in the commit. The patch also drops a config option which was related to debuginfinitepushmetadata command. Differential Revision: https://phab.mercurial-scm.org/D2953

infinitepush: delete infinitepush.fillmetadatabranchpattern config option This patch deletes the config option infinitepush.fillmetadatabranchpattern which if set to true sets a background process which will save metadata in infinitepush index. This series is meant to have a state where we can use it for CI purposes. Differential Revision: https://phab.mercurial-scm.org/D2952

infinitepush: drop the `--to` flag to push and use `-B` instead The extension added a `--to` flag to specify the bookmark to which revs should be pushed. This patch deletes that flag and instead uses the `-B` flag. After this patch, bookmark passed as `-B` is parsed and if it matches the infinitepush bookmark pattern, we consider that push as infinitepush. This is still not the best of what we can do. Later patches in the series will drop the use of `-B` flag and will instead handle things at bookmark bundle2 part. Plugging these logic to bookmark bundle2 part will also get rid of the scratchbranchparttype bundle2 part. Differential Revision: https://phab.mercurial-scm.org/D2108

infinitepush: drop the `--create` flag to push command The extension added a `--create` flag for creating a new bookmark on the bundlestore. This patch changes the bahviour to create a bookmark if it does not exists and removes the requirement of the `--create` and then drop the logic around the `--create` flag. Tests are changed to drop the usage of `--create` flag. Differential Revision: https://phab.mercurial-scm.org/D2107

infinitepush: drop logic related to treemanifest extension treemanifest is another facebook's extension which is not in core. Let's drop the logic related to infinitepush extension which is now ported to core. Differential Revision: https://phab.mercurial-scm.org/D2106

infinitepush: drop hack related to --to, --create flags of remotenames-ext The remotenames extension has --to, --create. --allow-anon flags which won't be ported to core. This patch drops the hack which fixes the interaction with remotenames-ext during push. In upcoming patches, --to and --create extensions will be removed from infinitepush also. Differential Revision: https://phab.mercurial-scm.org/D2105

infinitepush: drop error handling while pushing to svn server This is something which is internal to Facebook and we don't want in core. Differential Revision: https://phab.mercurial-scm.org/D2104

infinitepush: drop `--list-remote`, `--remote-path` flags to bookmark cmd The extension adds these two flags to list remote bookmarks using the `hg bookmark` command. These are not required in core currently and needs some discussion before getting them in, so let's drop them for now. This makes us deleting the wrapping of `hg bookmark` command on client side. The end goal here is to have minimal or no wrapping at client side. Differential Revision: https://phab.mercurial-scm.org/D2103

infinitepush: drop the scratchbookmarksparttype bundle2 part The scratchbookmarksparttype bundle2 part was used in backupcommands.py which we deleted in an earlier changeset. We don't need this part anymore. Moreover we now have bookmarks bundle2 part in core which we can use. This patch also drops the related encoding and decoding functions. Differential Revision: https://phab.mercurial-scm.org/D2102

infinitepush: remove wrapping around bundle2._addpartsfromopts() This wrapping around bundle2._addpartsfromopts() was added by me while hacking on an alternate way to mark a push as infinitepush or not. However the wrapping is wrong as the push command does not go through the code path which was wrapped on the client side. Differential Revision: https://phab.mercurial-scm.org/D2101

infinitepush: delete the bundle2 part ordering related to pushrebase pushrebase is another Facebook extension which does not live in core yet. So let's remove the logic which make sures infinitepush part is moved before the pushrebase one. Differential Revision: https://phab.mercurial-scm.org/D2100

infinitepush: delete the hack to load the extension after remotenames The remotenames extension which circumvents the default push implementation lives outside the core, so let's delete logic related to loading of extension on the basis of remotenames ext. Differential Revision: https://phab.mercurial-scm.org/D2099

infinitepush: drop the logic related to inhibit extension The inhibit extension lives in hg-experimental and is not a core extension. Differential Revision: https://phab.mercurial-scm.org/D2098

infinitepush: remove backupcommands.py This file contains command pushbackup and pullbackup which are used to store things to infinitepush. This is an advanced functionality which we don't require as of now. Also this uses shareutil.py from fb-hgext/ which is not there in core. Therefore this patch deletes the file and the config option which belongs to the backup thing. If we need this functionality in future, we can always move this file back. Differential Revision: https://phab.mercurial-scm.org/D2097

infinitepush: move the extension to core from fb-hgext This patch moves the infinitepush extension from fb-hgext to core. The extension is used to store incoming bundles during a push in bundlestore rather than applying them to the revlog. The extension was copied from the repository revision at f27f094e91553d3cae5167c0b1c42ae940f888d5 and following changes were made: * added `from __future__ import absolute_import` where missing * fixed module imports to follow the core style * minor fixes for test-check-code.t * registered the configs * adding the testedwith value to match core's convention * removed double newlines to make test-check-commit.t happy * added one line doc about extension and marked it as experimental Only one test file test-infinitepush-bundlestore.t is moved to core and following changes are made to file: * remove dependency of library.sh * split the tests into two tests i.e. test-infinitepush.t and test-infinitepush-bundlestore.t * removed testing related to other facebook's extensions pushrebase, inhibit, fbamend library-infinitepush.sh is also copied from fb-hgext from the same revision and following changes are made: * change the path to infinitepush extension as it's in core with this patch * removed sql handling from the file as we are not testing that initially Currently at this revision, test-check-module-imports.t does not pass as there is import of a module from fb/hgext in one the of the file which will be removed in the next patch. This extension right now has a lot of things which we don't require in core like `--to`, `--create` flags to `hg bookmark`, logic related to remotenames extension and another facebook's extensions, custom bundle2parts which can be prevented by using bookmarks bundle part and also logic related to sql store which is probably we don't want initially. The next patches in this series will remove all the unwanted and unrequired things from the extension and will make this a nice one. The end goal is to have a very lighweight extension with no or very less wrapping on the client side. Differential Revision: https://phab.mercurial-scm.org/D2096

narrow: use repo.local() instead of isinstance() Differential Revision: https://phab.mercurial-scm.org/D2973

tests: add zope.interface to clang-format ignore list Differential Revision: https://phab.mercurial-scm.org/D2975

contrib: rename clang-format-blacklist to clang-format-ignorelist "blacklist" is racially charged. Let's rename it to something that isn't. Differential Revision: https://phab.mercurial-scm.org/D2974

fix: new extension for automatically modifying file contents This change implements most of the corresponding proposal as discussed at the 4.4 and 4.6 sprints: https://www.mercurial-scm.org/wiki/AutomaticFormattingPlan This change notably does not include parallel execution of the formatter/fixer tools. It does allow for implementing that without affecting other areas of the code. I believe the test coverage to be good, but this is a hotbed of corner cases. Differential Revision: https://phab.mercurial-scm.org/D2897

tests: ignore zope packages when running pyflakes Differential Revision: https://phab.mercurial-scm.org/D2972

repository: define interface for local repositories Per discussions on the mailing list and at the 4.4 and 4.6 sprints, we want to start defining interfaces for local repository primitives so that we a) have a better idea of what the formal interface for various types is b) can more easily introduce alternate implementations of various components (e.g. in Rust). We have previously implemented interfaces that declare the peer and wire protocol APIs using the abc module. This commit introduces a monolithic interface for the localrepository class. It uses zope.interface - not abc - for defining and declaring the interface. The newly defined "completelocalrepository" interface is objectively horrible. It is based on what is actually in localrepository and doesn't represent a reasonable interface definition IMO. There's lots of... unwanted garbage in the interface. In other words, it reflects the horrible state of the localrepository "god object." But this is fine: a goal of this commit is to get the interface defined so that we have an interface. Future commits can refactor the interface into sub-interfaces, remove unwanted public attributes, etc. I attempted to define reasonable docstrings for the various interface members. But there are so many of them and I didn't know what some are used for. So I was lazy in a number of places and didn't write docstrings or detailed usage docs. Also, the members of the interface are defined in the order they are declared in localrepo.py. This revealed that the grouping of things in localrepo.py is... odd. The localrepository class now declares that it implements our newly defined interface. Unlike abc, zope.interface doesn't check interface conformance at type creation time (abc uses __metaclass__ magic to validate interface conformance when a type is created - usually at module import time). It does provide some functions for validating class and object conformance with declared interfaces. We add these checks to test-check-interfaces.py. We /could/ validate at run-time. But we hold off for now. (I'm a bit scared of doing that because of the various ways extensions monkeypatch repo instances.) After this commit, test-check-interfaces.py will fail if the set of public attributes on the localrepository class or instances change without corresponding updates to the interface. This is by design. Differential Revision: https://phab.mercurial-scm.org/D2933

setup: register zope.interface packages and compile C extension With this change, we should be able to use zope.interface in core! Differential Revision: https://phab.mercurial-scm.org/D2932

thirdparty: allow zope.interface.advice to be lazily imported The symbol from this module is only used in functions. Let's access that symbol through its imported module so importing zope.interface.advice can be lazy. Differential Revision: https://phab.mercurial-scm.org/D2931

thirdparty: port zope.interface to relative imports By using relative imports, we're guaranteed to get modules vendored with Mercurial rather than other random modules that might be in sys.path. My editor strips trailing whitespace on save. So some minor source code cleanup was also performed as part of this commit. # no-check-commit because some modified lines have double newlines Differential Revision: https://phab.mercurial-scm.org/D2930

thirdparty: don't make zope a namespace package There are a gazillion zope.* packages in the wild. So zope/__init__.py needs to be a namespace package. But in Mercurial, we have 1 zope package. And even if we had multiple packages, they'd all be in thirdparty/zope/. So we don't need a namespace package. Differential Revision: https://phab.mercurial-scm.org/D2929

thirdparty: vendor zope.interface 4.4.3 I've been trying to formalize interfaces for various components of Mercurial. So far, we've been using the "abc" package. This package is "good enough" for a lot of tasks. But it quickly falls over. For example, if you declare an @abc.abstractproperty, you must implement that attribute with a @property or the class compile time checking performed by abc will complain. This often forces you to implement dumb @property wrappers to return a _ prefixed attribute of the sane name. That's ugly. I've also wanted to implement automated checking that classes conform to various interfaces and don't expose other "public" attributes. After doing a bit of research and asking around, the general consensus seems to be that zope.interface is the best package for doing interface-based programming in Python. It has built-in support for verifying classes and objects conform to interfaces. It allows an interface's properties to be defined during __init__. There's even an "adapter registry" that allow you to register interfaces and look up which classes implement them. That could potentially be useful for places where our custom registry.py modules currently facilitates central registrations, but at a type level. Imagine extensions providing alternate implementations of things like the local repository interface to allow opening repositories with custom requirements. Anyway, this commit vendors zope.interface 4.4.3. The contents of the source tarball have been copied into mercurial/thirdparty/zope/ without modifications. Test modules have been removed because they are not interesting to us. The LICENSE.txt file has been copied so it lives next to the source. The Python modules don't use relative imports. zope/__init__.py defines a namespace package. So we'll need to modify the source code before this package is usable inside Mercurial. This will be done in subsequent commits. # no-check-commit for various style failures Differential Revision: https://phab.mercurial-scm.org/D2928

context: set repo property in basectx It seems like a good practice to call the super constructor. Let's start by passing the repo along to basectx so it can assign it to a private attribute. We should perhaps pass the rev and node along as well, but that requires more work before it can be done. Differential Revision: https://phab.mercurial-scm.org/D2970

context: move reuse of context object to repo.__getitem__ (API) As an example of how weird the basectx.__new__ is: whenever you create a workingctx, basectx.__new__ gets called first. Since our __new__ has a "changeid" argument as second parameter, when create the workingctx(repo, text="blah"), the text gets bound to "changeid". Since a string isn't a basectx, our __new__ ends up not doing anything funny, but that's still very confusing code. Another case is metadataonlyctx.__new__(), which I think exists in order to prevent metadataonlyctx.__init__'s third argument (originalctx) from being interpreted as a changeid in basectx.__new__(), thereby getting reused. Let's move this to repo.__getitem__ instead, where it will be pretty obvious what the code does. After this patch, changectx(ctx) will be an error (it will fail when trying to see if it's a 20-byte string). Differential Revision: https://phab.mercurial-scm.org/D2969

memctx: create parent contexts using "repo[p]" syntax I want to reduce dependence on basectx.__new__() and move that code over to repo.__getitem__(). Differential Revision: https://phab.mercurial-scm.org/D2968

context: avoid using a context object as a changeid I find it misleading to pass changeid=changectx. It currently works to do that because there's weird (IMO) handling of it in basectx.__new__. I'm planning on removing that code. Passing changeid as "changeid" and context as "context" makes it more readable. Note that the documentation of filectx.__init__ doesn't even seem to be aware that a changeid can be a context ("changeset revision, node, or tag"). Differential Revision: https://phab.mercurial-scm.org/D2967

context: remove unwanted assignments in basectx.__new__() (API) The two subclasses in core apparently didn't trust __new__() to do the job anyway (they both reassigned all the properties after). Differential Revision: https://phab.mercurial-scm.org/D2966

streamclonebundle: make sure we accept new stream clone bundle spec When asked specifically, the code do a sanity check on the clone bundle to ensure it's a stream bundle. As we introduced a new stream bundle spec, update the logic to support it. With this final changeset, we can now announce safely a stream v2 clone bundle and old clients would not crash trying to process it. This changeset address the last comment from Gregory Szorc on the stream v2 bundle series. Differential Revision: https://phab.mercurial-scm.org/D1957

streamclonebundle: add a test for stream clone bundle v2 Differential Revision: https://phab.mercurial-scm.org/D1956

bundlespec: add support for some variants This way the stream v2 bundle spec can disable the changegroup part while enabling the stream v2 part. Differential Revision: https://phab.mercurial-scm.org/D1955

bundle: add the possibility to bundle a stream v2 part Differential Revision: https://phab.mercurial-scm.org/D1954

streambundlev2: add a new test-file Add the new test file in a separate changeset before supporting the new format so we better see the differences. Differential Revision: https://phab.mercurial-scm.org/D1953

bundlespec: move computing the bundle contentops in parsebundlespec We will introduce a new bundlespec for stream bundle which will influence the contentops. Differential Revision: https://phab.mercurial-scm.org/D1952

bundlespec: introduce an attr-based class for bundlespec We will add support of contentops in the next patch, introduce a class instead of returning a 4-items tuple. Differential Revision: https://phab.mercurial-scm.org/D2971

templater: factor out unwrapastype() from evalastype() So ParseError of unwrapastype() can be caught reliably.

templater: extract unwrapinteger() function from evalinteger()

templater: extract type conversion from evalfuncarg() Needed by the subsequent patches.

templater: drop bool support from evalastype() Future patches will split evalastype() into two functions, evalrawexp() and unwrapastype(), so we can catch the exception of type conversion. # evaluating function may bubble up inner ParseError thing = evalrawexp(context, mapping, arg) try: return unwrapastype(context, thing) except ParseError: # add hint and reraise However, evalboolean() can't be factored out in this way since it has to process boolean-like symbols as non keyword. Fortunately, it's unlikely that we'll need a general type converter supporting bool, so this patch drops it from the table.

templater: do not use stringify() to concatenate flattened template output

templateutil: reimplement stringify() using flatten()

templateutil: move flatten() from templater It's the same kind of utility as stringify().

server: ensure the incoming request falls under the prefix value Prior to this, the first test asserted in wsgiref.validate.check_environ() saying PATH didn't start with '/', but the second test served up the repo. The assertion was just added in this cycle (though the value of PATH is still wrong without the assertion). Allowing access to the repo at any URL outside of the prefix is a long standing bug. This also affected hgwebdir, at least when used via --subrepo. Paths are not being canonicalized, so accesses to things like 'foo/../bar' will get tossed out here, unless the prefix also matches.

stringutil: move person function from templatefilters Move the person function from template filters to the stringutil module, so it can be reused in the mailmap template function. Differential Revision: https://phab.mercurial-scm.org/D2960

stringutil: add isauthorwellformed function The regular expression for this function formerly lived at https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hghooks/mozhghooks/author_format.py#l13 Differential Revision: https://phab.mercurial-scm.org/D2959

test-lfs-test-server: add a testcase for `hg serve` I haven't figured out yet how to make the authentication checks work for a specific list of users, so the 'web.allow-push' list is wildcarded. (It appears that the client doesn't react to a 401 by sending authentication data, which may be caused in part by not having all of the headers in httpbasicauthhandler's http_error_auth_reqed(), compared to a run of test-http.t. But in any case, we should probably have a separate set of tests for various authentication scenarios. As it is, without the wildcard, no push access is granted.) There are several deviations from the `lfs-test-server` case: - `hg serve` emits a Server header. I think Gregory indicated that this isn't easily suppressed. - `hg serve` names the "basic" transfer handler in the Batch API response. Not having to specify it was for backwards compatability, so this seems like the right thing to do. (`lfs-test-server` doesn't name it, whether it was explicitly requested by the client or not.) - PUT status for a newly created file is 201, per RFC-2616 [1]. The Basic Transfer API [2] shows an example upload transcript with a 200 response. It doesn't make much sense to re-upload a file (unless it is corrupt) in an example, but I wouldn't be surprised if some other implementations also expect 200 because of this. But the RFC says MUST use 201 for creation. - The Content-Type for the file transfers is "application/octet-stream", like the sample transcript (though I don't see it explicitly called out in the text elsewhere). Using "text/plain" seems clearly wrong. - `lfs-test-server` isn't removing the action property and sending back an error code like the spec calls out when a file is missing or corrupt. Doing so on the `hg serve` side reveals a bug in our client code when handling the response- it indicates the remote file is missing instead of corrupt around line 452. I'll probably glob over the Content-Length differences once this settles down. Prior to the recent hgweb refactoring, the Batch API response was using chunked encodings instead. Back to the RFC, I have no idea if the python framework handles the "MUST NOT ignore any Content-* (e.g. Content-Range) headers that it does not understand or implement and MUST return a 501" for a PUT request. [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6 [2] https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md#uploads

push: use "repo['.']" instead of old form "repo['']" Note that this does not conflict with the commit message of my previous patch: I found this after writing the previous patch (besides, I very easily forget things). Differential Revision: https://phab.mercurial-scm.org/D2965

context: change default changeid from old form '' to '.' I don't think I've seen repo[''] in the codebase at least in the last two years. Differential Revision: https://phab.mercurial-scm.org/D2964

remove: add dry-run functionality

lfs: add support for serving blob files

lfs: add server side support for the Batch API