streamclonebundle: make sure we accept new stream clone bundle spec When asked specifically, the code do a sanity check on the clone bundle to ensure it's a stream bundle. As we introduced a new stream bundle spec, update the logic to support it. With this final changeset, we can now announce safely a stream v2 clone bundle and old clients would not crash trying to process it. This changeset address the last comment from Gregory Szorc on the stream v2 bundle series. Differential Revision: https://phab.mercurial-scm.org/D1957

streamclonebundle: add a test for stream clone bundle v2 Differential Revision: https://phab.mercurial-scm.org/D1956

bundlespec: add support for some variants This way the stream v2 bundle spec can disable the changegroup part while enabling the stream v2 part. Differential Revision: https://phab.mercurial-scm.org/D1955

bundle: add the possibility to bundle a stream v2 part Differential Revision: https://phab.mercurial-scm.org/D1954

streambundlev2: add a new test-file Add the new test file in a separate changeset before supporting the new format so we better see the differences. Differential Revision: https://phab.mercurial-scm.org/D1953

bundlespec: move computing the bundle contentops in parsebundlespec We will introduce a new bundlespec for stream bundle which will influence the contentops. Differential Revision: https://phab.mercurial-scm.org/D1952

bundlespec: introduce an attr-based class for bundlespec We will add support of contentops in the next patch, introduce a class instead of returning a 4-items tuple. Differential Revision: https://phab.mercurial-scm.org/D2971

templater: factor out unwrapastype() from evalastype() So ParseError of unwrapastype() can be caught reliably.

templater: extract unwrapinteger() function from evalinteger()

templater: extract type conversion from evalfuncarg() Needed by the subsequent patches.

templater: drop bool support from evalastype() Future patches will split evalastype() into two functions, evalrawexp() and unwrapastype(), so we can catch the exception of type conversion. # evaluating function may bubble up inner ParseError thing = evalrawexp(context, mapping, arg) try: return unwrapastype(context, thing) except ParseError: # add hint and reraise However, evalboolean() can't be factored out in this way since it has to process boolean-like symbols as non keyword. Fortunately, it's unlikely that we'll need a general type converter supporting bool, so this patch drops it from the table.

templater: do not use stringify() to concatenate flattened template output

templateutil: reimplement stringify() using flatten()

templateutil: move flatten() from templater It's the same kind of utility as stringify().

server: ensure the incoming request falls under the prefix value Prior to this, the first test asserted in wsgiref.validate.check_environ() saying PATH didn't start with '/', but the second test served up the repo. The assertion was just added in this cycle (though the value of PATH is still wrong without the assertion). Allowing access to the repo at any URL outside of the prefix is a long standing bug. This also affected hgwebdir, at least when used via --subrepo. Paths are not being canonicalized, so accesses to things like 'foo/../bar' will get tossed out here, unless the prefix also matches.

stringutil: move person function from templatefilters Move the person function from template filters to the stringutil module, so it can be reused in the mailmap template function. Differential Revision: https://phab.mercurial-scm.org/D2960

stringutil: add isauthorwellformed function The regular expression for this function formerly lived at https://hg.mozilla.org/hgcustom/version-control-tools/file/tip/hghooks/mozhghooks/author_format.py#l13 Differential Revision: https://phab.mercurial-scm.org/D2959

test-lfs-test-server: add a testcase for `hg serve` I haven't figured out yet how to make the authentication checks work for a specific list of users, so the 'web.allow-push' list is wildcarded. (It appears that the client doesn't react to a 401 by sending authentication data, which may be caused in part by not having all of the headers in httpbasicauthhandler's http_error_auth_reqed(), compared to a run of test-http.t. But in any case, we should probably have a separate set of tests for various authentication scenarios. As it is, without the wildcard, no push access is granted.) There are several deviations from the `lfs-test-server` case: - `hg serve` emits a Server header. I think Gregory indicated that this isn't easily suppressed. - `hg serve` names the "basic" transfer handler in the Batch API response. Not having to specify it was for backwards compatability, so this seems like the right thing to do. (`lfs-test-server` doesn't name it, whether it was explicitly requested by the client or not.) - PUT status for a newly created file is 201, per RFC-2616 [1]. The Basic Transfer API [2] shows an example upload transcript with a 200 response. It doesn't make much sense to re-upload a file (unless it is corrupt) in an example, but I wouldn't be surprised if some other implementations also expect 200 because of this. But the RFC says MUST use 201 for creation. - The Content-Type for the file transfers is "application/octet-stream", like the sample transcript (though I don't see it explicitly called out in the text elsewhere). Using "text/plain" seems clearly wrong. - `lfs-test-server` isn't removing the action property and sending back an error code like the spec calls out when a file is missing or corrupt. Doing so on the `hg serve` side reveals a bug in our client code when handling the response- it indicates the remote file is missing instead of corrupt around line 452. I'll probably glob over the Content-Length differences once this settles down. Prior to the recent hgweb refactoring, the Batch API response was using chunked encodings instead. Back to the RFC, I have no idea if the python framework handles the "MUST NOT ignore any Content-* (e.g. Content-Range) headers that it does not understand or implement and MUST return a 501" for a PUT request. [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6 [2] https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md#uploads

push: use "repo['.']" instead of old form "repo['']" Note that this does not conflict with the commit message of my previous patch: I found this after writing the previous patch (besides, I very easily forget things). Differential Revision: https://phab.mercurial-scm.org/D2965

context: change default changeid from old form '' to '.' I don't think I've seen repo[''] in the codebase at least in the last two years. Differential Revision: https://phab.mercurial-scm.org/D2964

remove: add dry-run functionality

lfs: add support for serving blob files

lfs: add server side support for the Batch API

lfs: add basic routing for the server side wire protocol processing The recent hgweb refactoring yielded a clean point to wrap a function that could handle this, so I moved the routing for this out of the core. While not an hg wire protocol, this seems logically close enough. For now, these handlers do nothing other than check permissions. The protocol requires support for PUT requests, so that has been added to the core, and funnels into the same handler as GET and POST. The permission checking code was assuming that anything not checking 'pull' or None ops should be using POST. But that breaks the upload check if it checks 'push'. So I invented a new 'upload' permission, and used it to avoid the mandate to POST. A function wrap point could be added, but security code should probably stay grouped together. Given that anything not 'pull' or None was requiring POST, the comment on hgweb.common.permhooks is probably wrong- there is no 'read'. The rationale for the URIs is that the spec for the Batch API[1] defines the URL as the LFS server url + '/objects/batch'. The default git URLs are: Git remote: https://git-server.com/foo/bar LFS server: https://git-server.com/foo/bar.git/info/lfs Batch API: https://git-server.com/foo/bar.git/info/lfs/objects/batch '.git/' seems like it's not something a user would normally track. If we adhere to how git defines the URLs, then the hg-git extension should be able to talk to a git based server without any additional work. The URI for the transfer requests starts with '.hg/' to ensure that there are no conflicts with tracked files. Since these are handed out by the Batch API, we can change this at any point in the future. (Specifically, it might be a good idea to use something under the proposed /api/ namespace.) In any case, no files are stored at these locations in the repository directory. I started a new module for this because it seems like a good idea to keep all of the security sensitive server side code together. There's also an issue with `hg verify` in that it will want to download *all* blobs in order to run. Sadly, there's no way in the protocol to ask the server to verify the content of a blob it may have. (The verify action is for storing files on a 3rd party server, and then informing the LFS server when that completes.) So we may end up implementing a custom transfer adapter that simply indicates if the blobs are valid, and fall back to basic transfers for non-hg servers. In other words, this code is likely to get bigger before this is made non-experimental. [1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md

test-lfs: drop trailing ', ' item separators from debug JSON output The trailing space looks weird when conditionalizing the line. The commas shouldn't be necessary because of the indenting. The `lfs-test-server` isn't sending all of the same items (notably, the "transfer" attribute is missing), so having the commas means more lines need to be conditionalized.

lfs: add a blob verification method to the local store A corrupt blob can be signaled through the Batch API response, without actually transferring the file. A true/false indicator is slightly easier than immediately catching an exception.

tests: conditionalize printed environment variable output in test-alias

debugsetparents: avoid using "r1/r2" variable names for nodeids Differential Revision: https://phab.mercurial-scm.org/D2963

tag: avoid using "r" variable name for nodeid Differential Revision: https://phab.mercurial-scm.org/D2962

locate: avoid using "rev" variable name for nodeid Also, drop silly "ctx = repo[ctx.node()]". Differential Revision: https://phab.mercurial-scm.org/D2961

py3: bytes/unicode dance on __doc__ of cmdalias

alias: reject non-ascii characters in user help/doc strings Since command doc/help texts are passed to i18n.gettext(), they must be ASCII. Otherwise, UnicodeError would be raised.

lfs: respect narrowmatcher when testing to add 'lfs' requirement (issue5794) There's a similar test in lfs.wrapper.convertsink(), but I didn't update that because I don't think that the sink repo in a convert can be narrow. It seems reasonable that a narrow clone of an LFS repo may not necessarily be an LFS repo. The only potential issue is that LFS has a hard requirement for changegroup v3, which that extension enables. The use of treemanifest will enable changegroup v3 in narrow clones, because allsupportedversions() in changegroup.py preserves it when it sees a 'treemanifest' requirement. But I don't see where changegroup v3 is enabled for a flat manifest.

localrepo: make filterpats private (API) I'm not sure why this is available on the public API. AFAICT it isn't used outside of the class. .. api:: localrepo.localrepository.filterpats was renamed to localrepo.localrepository._filterpats. Differential Revision: https://phab.mercurial-scm.org/D2927

narrow: use featuresetupfuncs This is the preferred way to register repo requirements that can be opened because it respects the set of loaded extensions at repo open time. Differential Revision: https://phab.mercurial-scm.org/D2926

localrepo: move featuresetupfuncs out of localrepository class (API) I want to establish an interface for local repositories. featuresetupfuncs is a class attribute and is global/shared across all localrepository instances. Let's move it to a module-level attribute to clarify it isn't part of the local repository interface. .. api:: localrepo.localrepository.featuresetupfuncs has been renamed to localrepo.featuresetupfuncs. Differential Revision: https://phab.mercurial-scm.org/D2925

help: supporting both help and doc for aliases This allows an alias to be definted like: [alias] lj = log -Tjson lj:help = [-r REV] lj:doc = Shows the revision log in JSON format. Differential Revision: https://phab.mercurial-scm.org/D2678

grep: fixes erroneous output of grep in forward order (issue3885) If grep is passed a revset in forwards order via -r , say -r 0:tip Then the output is erroneous. This patch fixes that. The output was wrong because we deleted the last revision key in the matches and when we moved to the next revision we didn't had this to compare the diff. So the pstates dict was always empty and in the SequenceMatcher, to convert and empty pstate to the states dictionary you would always insert. This patch keeps the matches dictionary until the end of this window and clears it at once when this window ends. This solves the above mentioned problem and also do not cause any memory leak.

lfs: move the 'supportedoutgoingversions' handling to changegroup.py This handling already exists here for the narrow extension. We still need to either figure out how to enable changegroup v3 without the extension, or figure out how to let the server detect that the client doesn't have it loaded, and emit a user friendly error[1]. I can't tell if D1944 is the appropriate vehicle for the latter. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-January/109550.html

obsolete: refactor function for getting obsolete options The function for returning obsolete option values obtains all options, validates, then returns the option that was requested. Let's create a new function to return all obsolete option values so callers needing multiple values can call that. Differential Revision: https://phab.mercurial-scm.org/D2667

setup: install cbor packages Without this, standalone installs don't have the cbor files. Differential Revision: https://phab.mercurial-scm.org/D2949

wireproto: review fixups Capture various TODOs and return an explicit value. This represents feedback from Yuya and Augie on various commits. Differential Revision: https://phab.mercurial-scm.org/D2944

cbor: add a __init__.py to top level cbor module Differential Revision: https://phab.mercurial-scm.org/D2752

cbor: fix core test-check* This patch fixes couple of test-check* tests by making sure they skip testing the third party library cbor. Differential Revision: https://phab.mercurial-scm.org/D2751

thirdparty: vendor cbor2 python library CBOR stands for Concise Binary Object Representation, which is a data format which is very compact and extensible. This patch moves the python library which can serilaize and deserialize python objects to/from cbor formats. The library is taken from https://github.com/agronholm/cbor2/ from commit 84181540f6eb650437e3f73cd104a65661fe8e67. Unrequired files from the cbor library - docs/, tests/, setup.py, setup.cfg, and tox.ini - have not been vendored. There is another python library for cbor at https://github.com/brianolson/cbor_py/ which is used in evolve extension and was imported in initial version of this series. That library though contains C code and is bit faster, but has known bugs around serializing nested structures, is unmaintained, raises an Exception object instead of a more dedicated Error type. So, it's better to use a bug free and actively maintained library. This library is not yet used and will be used in later commits. # no-check-commit because we are importing a third library module Differential Revision: https://phab.mercurial-scm.org/D2750

merge: deprecate accessing update results by index Now that we have named attributes, let's convert the code base to use them. We also add deprecation warnings so legacy consumers are aware of their transgressions. ``stats.unresolvedcount`` is much easier to read than ``stats[3]``, don't you think? Differential Revision: https://phab.mercurial-scm.org/D2694

procutil: introduce context-manager interface for protect/restorestdio The code looks slightly cleaner since it was pretty easy to pass arguments in wrong order.

procutil: move protectio/restoreio from commandserver Some variants of this will be useful for stdio-based servers such as sshserver.

commandserver: rewrite protectio/restoreio to not depend on ui Prepares for porting to utils.procutil, in which ui shouldn't be known. ui.flush() is replaced with ui.fout.flush() since ui.ferr wasn't involved.

util: deprecate procutil proxy functions (API) Several functions are re-exported by utils.procutil, which require explicit modname parameter. .. api:: Utility functions related to process/executable management have been moved to utils.procutil module.

procutil: bulk-replace function calls to point to new module

procutil: bulk-replace util.std* to point to new module

procutil: move process/executable management functions to new module std* files, pipe helpers, and findexe()s are moved as well since they are likely to be used with sub processes.

util: stop using readfile() in tempfilter() To unblock code move to utils.*. It's merely two lines of very Pythonic code. No helper function should be needed.

util: mark filtertable as private constant Prepares for porting to utils.*.

util: mark platform-specific gethgcmd() as private util.hgcmd() is the public interface for gethgcmd().

test-pathconflicts-merge: stop requiring symlink support The errors from the last time I took a shot at this back in early November have disappeared, so let's just enable this.

commands: use constants for merge things We have nice constants now. Let's use them to make the code easier to reason about. Differential Revision: https://phab.mercurial-scm.org/D2702

merge: use constants for actions We finish up establishing named constants in this file with actions. I remember scratching my head trying to figure out what this code was doing as part of addressing a recent security issue with subrepos. Having the named constants in place definitely makes things easier to read. I'm not convinced the new constants have the best names (I'm not an expert in this code). But they can be changed easily enough. Also, since these constants are internal only, we might want to change their values to something more human readable to facilitate debugging. Or maybe we could employ an enum type some day... Differential Revision: https://phab.mercurial-scm.org/D2701

merge: use constants for merge record state Named constants are easier to read than short string values. Differential Revision: https://phab.mercurial-scm.org/D2700