tests: improve the widening testing in test-narrow-widen* Before this patch, we are testing `hg tracked --addinclude` by adding a command which is not introduced in the changesets till now. If you closely look at the tests, wider/f was introduced on the server after the narrow clone was done and extending the existing clone to include wider/f does not make sense. We should test extending a file which exists. Differential Revision: https://phab.mercurial-scm.org/D4452

narrow: use util.readfile() and improve error message using --narrowspec This patch improves the error message and uses util.readfile() for reading narrowspecs file while cloning. Differential Revision: https://phab.mercurial-scm.org/D4462

merge: use vfs methods for I/O All I/O is supposed to be performed via vfs instances so filesystems can be abstracted. The previous commit ported the old code in purge, which didn't go through the vfs layer. This commit ports the purge code to use the vfs layer. The vfs layer didn't have a method to remove a single directory, so it was added as part of implementing this. Differential Revision: https://phab.mercurial-scm.org/D4478

merge: move purge logic from extension Working directory purging feels like functionality that should be in core rather than in an extension. This commit effectively moves the core purging logic from the purge extension to merge.py. Code was refactored slightly. Rather than deal with printing in this function, the function is a generator of paths and the caller can worry about printing. Differential Revision: https://phab.mercurial-scm.org/D4477

tests: stabilize test-removeemptydirs.t on Windows

help: add internals.wireprotocolv2 to the table, and remove redundant header

match: improve includematcher.visitchildrenset to be much faster and cached This improves the speed of visitchildrenset considerably, especially when there are complicated matchers involved that may have many entries in _dirs or _parents. Unfortunately the benchmark isn't easily upstreamed due to its reliance on https://github.com/vstinner/perf (primarily due to the conflict when importing it if I were to contribute the benchmark as contrib/matcherbenchmarks.py) instead of asv or some other perf measurement system. To describe the benchmark briefly: I generated an includematcher of either 5 or 3500 "rootfilesin:prefix1/prefix2/prefix3/<randomsubdirs, 1-8 levels deep>" items in the 'setup' function, and then called `im.visitchildrenset('prefix1/prefix2')` in the 'stmt' function in perf.timeit. For the set of 5: - before: 15.3 us +- 2.9 us - after: 1.59 us +- 0.02 us For the set of 3500: - before: 3.90 ms +- 0.10 ms - after: 3.15 us +- 0.09 us (note the m->u change) Differential Revision: https://phab.mercurial-scm.org/D4351

py3: add new passing tests spotted by the buildbot Differential Revision: https://phab.mercurial-scm.org/D4495

tests: order the imports in test-fastannotate-hg.t The wrong ordering breaks test-check-module-imports.t on Python 3. I am not sure why that test is so much active on py3. Differential Revision: https://phab.mercurial-scm.org/D4496

lfs: ensure the blob is linked to the remote store on skipped uploads I noticed a "missing" blob when pushing two repositories with common blobs to a fresh server, and then running `hg verify` as a user different from the one running the web server. When pushing the second repo, several of the blobs already existed in the user cache, so the server indicated to the client that it doesn't need to upload the blobs. That's good enough for the web server process to serve up in the future. But a different user has a different cache by default, so verify complains that `lfs.url` needs to be set, because it wants to fetch the missing blobs. Aside from that corner case, it's better to keep all of the blobs in the repo whenever possible. Especially since the largefiles wiki says the user cache can be deleted at any time to reclaim disk space- users switching over may have the same expectations.

cbor: teach the encoder to handle python `long` type for Windows The tests for 2**32 and -7000000000 were blowing up, complaining about not knowing how to encode type 'long'. sys.maxint tops out at 2**31-1 on Windows, but I guess is 2^63-1 on Linux? I *think* we're OK on the decode side, as there is an assertion that the decoded value is equal to the original primitive value. I opted for the pycompat alias instead of swallowing the NameError because the vendored cbor package uses an alias, and I see at least pywatchman and templatefilters open codes their own aliases.

buildrpm: remove TODO that looks done to me Differential Revision: https://phab.mercurial-scm.org/D4494

buildrpm: fix embarassing bug in shell variable expansions `$distance_$node` looks up and concatenates `$distance_` and `$node`. `$distance_` is empty, so we were getting the node without the distance. Using the curly braces makes our intent explicit and produces better-versioned RPMS. Differential Revision: https://phab.mercurial-scm.org/D4493

absorb: clarify the reason for not finding changesets to modify I'm used to pushing to non-publishing repos, so this was the last thing I thought to check.

merge with stable Includes an extra bump of the version number for parsers because the merge produces a new, distinct version of the code.

state: use our CBOR module This was the last consumer of the vendored CBOR package in core. Differential Revision: https://phab.mercurial-scm.org/D4471

wireprotov2peer: use our CBOR decoder Behavior should be the same in order to preserve backwards compatibility. We obviously want to stream values in this code. We'll do that in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D4470

debugcommands: use our CBOR decoder It implements the set of CBOR needed for the wire protocol. Differential Revision: https://phab.mercurial-scm.org/D4469

wireprotov2server: use our CBOR encoder Again, test output changed slightly because of map key ordering differences. This shouldn't matter. I could have called oncommandresponsereadygen() with the raw output from the stream encoder. However, this changed test output further. I left a TODO to follow up on that later. Differential Revision: https://phab.mercurial-scm.org/D4468

wireprotoserver: use our CBOR encoder Moving away from the vendored package. Again, our encoder normalizes keys differently from the vendored package (for now), hence the test changes. Differential Revision: https://phab.mercurial-scm.org/D4467

wireprotoframing: use our CBOR module Tests changed because our CBOR encoder appears to sort map keys differently from the vendored CBOR package. The CBOR specification does define canonical sorting rules for keys based on the byte values. I'm guessing our implementation doesn't follow them. But our encoder doesn't guarantee that it conforms with the canonical specification. Right now, we just care that output is deterministic. And our encoder does guarantee that. Differential Revision: https://phab.mercurial-scm.org/D4466

httppeer: use our CBOR decoder We just implemented our own CBOR decoder. Let's use it in httppeer. Differential Revision: https://phab.mercurial-scm.org/D4465

zsh_completion: complete shelve and unshelve Differential Revision: https://phab.mercurial-scm.org/D4428

zsh_completion: use revsets to exclude this rev from suggestions to hg merge One of the most important aspects of a completion system is its speed, so 1 call to hg is definitely better than 4. Sorting by rev (descending) is to preserve the same order as in `hg heads` output. While at it, declare branches as an array too. Differential Revision: https://phab.mercurial-scm.org/D4426

drawdag: correctly pass repo to super constructor This was caught by someone who was looking for the super(cls, self).__init__(self, ...) pattern in all code in our Google-internal repo. Differential Revision: https://phab.mercurial-scm.org/D4464

help: correct a typo in tweakdefaults

narrow: add '--import-rules' flag to tracked command This patch adds a `--import-rules` flag to tracked command provided by narrow extension. Using the --import-rules flag, you can pass a filename from which narrowspecs should be read and added to main narrowspec. A lot of times, in automation or manually also, when you are working with big repo, specifying each path name on commandline using '--addinclude' and '--addexclude' is tedious and something which can scale. So we needed something where we can pass a file to extend the narrowspecs. Nice thing about this is that the automations which reads some file to change the sparse profile, can now read the same file for changing narrowspecs too. Tests are added for the new feature. Differential Revision: https://phab.mercurial-scm.org/D4125

internals: extract wire protocol version 2 commands to standalone doc wireprotocol.txt is a bit long and unwieldy. I think splitting it up will help aid comprehension. Let's start by extracting wire protocol version 2 commands to a standalone document. As part of the port, I munged with the section titles a bit and expanded the TODO around node namespaces. Differential Revision: https://phab.mercurial-scm.org/D4442

wireprotov2peer: split responsedata handling into separate function So we don't have so much logic inside an if/elif block. Differential Revision: https://phab.mercurial-scm.org/D4439

httppeer: add TODO about delayed handling of ^C When testing wire protocol v2 with large responses, I noticed that ^C doesn't result in immediate termination. I'm not sure what the root cause is. Let's add a TODO so we don't forget about it. Differential Revision: https://phab.mercurial-scm.org/D4438

httppeer: log commands for version 2 peer This will aid testing and debugging. Differential Revision: https://phab.mercurial-scm.org/D4437

wireprotov2peer: add TODO about streaming responses This is a pretty big deficiency in the current implementation. We don't want to forget about this. Differential Revision: https://phab.mercurial-scm.org/D4435

tests: update relnotes test for actual output I'm unclear why I didn't see this before but I consistently see it now. In any event, it doesn't appear to be a regression, so let's just bake it into the test. Differential Revision: https://phab.mercurial-scm.org/D4463

py3: use print as a function in tests/test-hgweb.t Differential Revision: https://phab.mercurial-scm.org/D4461

py3: bytestr a bytes val to make sure we get bytechr while iterating Differential Revision: https://phab.mercurial-scm.org/D4460

py3: make regexes in hgext/keyword.py bytes # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4459

py3: add more missing b'' prefixes in test files # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4458

py3: write the file in str mode in tests/test-rename-merge2.t This is better then converting the value the value of sys.argv to bytes. This makes the test pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4457

py3: add missing b'' prefixes in tests/badserverext.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4456

py3: handle keyword arguments correctly in hgext/acl.py The keys of keyword arguments on python 3 should be str, so when we try to get some key from them, we must make sure we are using str. # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4455

py3: don't return the revid as unicode in hgext/convert/subversion.py I tried digging why u'' was added in first place, and I was unable to found something relevant. This might be because some API's takes unicodes, I am not sure. Differential Revision: https://phab.mercurial-scm.org/D4454

py3: make sure we pass str in os.sysconf in hgext/convert/common.py # skip-blame because just r'' prefix Differential Revision: https://phab.mercurial-scm.org/D4453

manifest: fix leak on error return from lazymanifest_filtercopy() Spotted by ASAN. free(copy->lines) and Py_DECREF(copy->pydata) are replaced by Py_XDECREF(copy), which should call lazymanifest_dealloc(). Freeing half-initialized copy->lines is safe since copy->numlines holds a valid value.

manifest: incref/decref copy->pydata to clarify 'copy' holds a reference

dirstate: use tuple interface to fix leak in pack_dirstate() Spotted by ASAN. Unlike PyTuple_GET_ITEM(), PySequence_ITEM() returns a new reference. This bug could be fixed by inserting Py_CLEAR() and Py_XDECREF() appropriately, but I think requiring a tuple object is simpler and less error-prone. The cext version is jumped to 10 since 6..9 are used in the default branch. We'll need to bump it again at merge.

xdiff: fix leak in hunk_consumer() Spotted by ASAN. Since PyList_Append() does not "steal" a reference, Py_DECREF() is always required. Perhaps, this is the largest leak in this series.

base85: fix leak on error return from b85decode() Spotted by ASAN. We don't need to initialize 'out' to NULL, but I decided to do that for clarity.

merge with stable

Added signature for changeset ede3bf31fe63

Added tag 4.7.1 for changeset ede3bf31fe63

hgweb: load revcount + 1 entries to fill nextentry in log page (issue5972) "revcount + 1" is moved to the call site to make it clearer.

revlog: fix size of Python nodetree object Follows up 9f097214fbf3.

revert: stabilize status message of chunks selected interactively Unfortunately, patch.filterpatch() doesn't preserve the order of the input files. We have to sort them manually.

cborutil: add a buffering decoder The sansiodecoder leaves it up to the callers to feed in data that wasn't fully consumed last time. This commit implements a decoder that performs buffering of leftover chunks from the previous invocation. It otherwise behaves identically to sansiodecoder. Differential Revision: https://phab.mercurial-scm.org/D4434

cborutil: remove readindefinitebytestringtoiter() This was implemented as part of implementing streaming encoding. It was never used outside of tests. Now that we have a full CBOR decoder, it can be used for incremental decoding of indefinite-length byte strings. This also removes the last use of the vendored cbor2 package from this module. Differential Revision: https://phab.mercurial-scm.org/D4433

cborutil: implement sans I/O decoder The vendored CBOR package decodes by calling read(n) on an object. There are a number of disadvantages to this: * Uses blocking I/O. If sufficient data is not available, the decoder will hang until it is. * No support for partial reads. If the read(n) returns less data than requested, the decoder raises an error. * Requires the use of a file like object. If the original data is in say a buffer, we need to "cast" it to e.g. a BytesIO to appease the decoder. In addition, the vendored CBOR decoder doesn't provide flexibility that we desire. Specifically: * It buffers indefinite length bytestrings instead of streaming them. * It doesn't allow limiting the set of types that can be decoded. This property is useful when implementing a "hardened" decoder that is less susceptible to abusive input. * It doesn't provide sufficient "hook points" and introspection to institute checks around behavior. These are useful for implementing a "hardened" decoder. This all adds up to a reasonable set of justifications for writing our own decoder. So, this commit implements our own CBOR decoder. At the heart of the decoder is a function that decodes a single "item" from a buffer. This item can be a complete simple value or a special value, such as "start of array." Using this function, we can build a decoder that effectively iterates over the stream of decoded items and builds up higher-level values, such as arrays, maps, sets, and indefinite length bytestrings. And we can do this without performing I/O in the decoder itself. The core of the sans I/O decoder will probably not be used directly. Instead, it is expected that we'll build utility functions for invoking the decoder given specific input types. This will allow extreme flexibility in how data is delivered to the decoder. I'm pretty happy with the state of the decoder modulo the TODO items to track wanted features to help with a "hardened" decoder. The one thing I could be convinced to change is the handling of semantic tags. Since we only support a single semantic tag (sets), I thought it would be easier to handle them inline in decodeitem(). This is simpler now. But if we add support for other semantic tags, it will likely be easier to move semantic tag handling outside of decodeitem(). But, properly supporting semantic tags opens up a whole can of worms, as many semantic tags imply new types. I'm optimistic we won't need these in Mercurial. But who knows. I'm also pretty happy with the test coverage. Writing comprehensive tests for partial decoding did flush out a handful of bugs. One general improvement to testing would be fuzz testing for partial decoding. I may implement that later. I also anticipate switching the wire protocol code to this new decoder will flush out any lingering bugs. Differential Revision: https://phab.mercurial-scm.org/D4414

tests: remove use of string in CBOR test We don't use the CBOR string major type in the wire protocol. Let's not test for it. Differential Revision: https://phab.mercurial-scm.org/D4413

internals: document CBOR utilization I spoke with some people at Mozilla about CBOR and they advised me that we should be careful about the subset of CBOR we use in order to mitigate security, performance, and compatibility concerns. This commit establishes a document that attempts to formalize our use of CBOR. Its main limitations are on what types are allowed. It explicitly enumerates which types are supported. Notable missing features include: * Indefinite-length arrays and maps * Text strings (bytes all the way) * Floats * Date/time types * Big integers * Use of indefinite-length byte strings for map keys, values in containers. If we have a need for any of these, we can have a discussion about them when the time comes. Differential Revision: https://phab.mercurial-scm.org/D4412

py3: add new passing test to whitelist found by buildbot Differential Revision: https://phab.mercurial-scm.org/D4451

py3: alias xrange to range in tests/test-fastannotate-revmap.py xrange is not available on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4450

py3: add r'' prefix to prevent b'' being prepended The keys of keyword arguments should be str. # skip-blame because we are adding just r'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D4449

revert: fix the inconsistency of status msgs in --interactive mode Before this patch we were priting every action msg before actually performing that action and that was resulting in inconsistencies; like in --interactive session if user decided to not revert any changes in a file foo, still there will be a msg on console saying "reverting foo". To fix this, I have made some changes to print status msg just before the action it is going to perform, no matter if --interactive or not. Changes made in test-revert-interactive.t reflect the changed behavior. There are also some changes in test-revert.t because of change in the order of messages. Differential Revision: https://phab.mercurial-scm.org/D4380

revert: remove unnecessary check as `msg` should never be a callable now Differential Revision: https://phab.mercurial-scm.org/D4448

run-tests: drop support for --with-python3 and $PYTHON3 This backs out cdbc25306696. We can run many tests on python3 without this hack.

test-check-py3-compat: don't run py3 part if not build against python3 This test depends on C extensions built for python3, which aren't available if run-tests.py runs on python2 and no -l/--local is specified.

test-check-py3-compat: don't run py2 part if $PYTHON is python3

resolve: test that we suggest a valid hg command Differential Revision: https://phab.mercurial-scm.org/D4447

bash_completion: complete arguments for --color and --pager

bash_completion: add more global options that receive an argument

tracing: ignore any IOErrors when writing to pipe When the pager forks off the main process, we can end up with the pipe closed prematurely. Rather than break hg entirely when that happens and tracing is active, just let lingering events disappear as needed. Differential Revision: https://phab.mercurial-scm.org/D4445

tests: avoid shellquoting bytes on Python 3 Should fix Python 3 builder. Differential Revision: https://phab.mercurial-scm.org/D4444

resolve: ui.config -> ui.configbool, for better validation Differential Revision: https://phab.mercurial-scm.org/D4446

zsh_completion: handle --rev as well as -r for diff and revert Completion for hg diff and revert should suggest all files in context of a non-current revision. The script used to look only for `-r foo`, and now it also understands `--rev foo`. Differential Revision: https://phab.mercurial-scm.org/D4427

resolve: add config to make hg resolve not re-merge by default Before this, calling 'hg resolve' with neither -m, -u or -l will re-merge. This is highly error prone (it's easy to forget to forget a -m), and pretty bad when it happens (many people have no idea 'hg resolve' can re-merge, and end up redoing the work of resolving all the conflicts, because they have no idea there was a backup of their work). Differential Revision: https://phab.mercurial-scm.org/D4379

resolve: add a flag for the default behavior of re-merging On its own, it's not useful, but the next commit will add an hgrc config option to make it mandatory. There is no short option, as -r almost always means --rev and this option doesn't seem like it would be so common as to mandate a short option. Differential Revision: https://phab.mercurial-scm.org/D4378

tests: conditionalize narrow-widen error output for Windows

largefiles: use a context manager to control the progress bar lifetime

lfs: use a context manager to control the progress bar lifetime

hg: ensure the progress bar is completed when copying the store This is just a block indent under the context manager.

shelve: fix crash on unshelve without .shelve metadata file Follow up for c67c94c0e7ae and 38373da1af02. The inline comment says "we should keep track of the unshelve node in case we need to reuse it." Perhaps such case isn't tested, and this patch does NOT add a test for the reuse of the unbundled revision. Also, I have no idea what should be done if new revision is unbundled because of "node not in repo".

fastannotate: use stringutil.pprint() to dump diffopts to be hashed

test-check-py3-compat: update "python3 check-py3-compat.py" output

test-check-py3-compat: exclude thirdparty and cffi which are known to fail Otherwise the following errors would be spilled out: mercurial/cffi/bdiff.py: error importing: <ImportError> cannot import name '_bdiff' (error at bdiff.py:*) mercurial/cffi/bdiffbuild.py: error importing: <ModuleNotFoundError> No module named 'cffi' (error at bdiffbuild.py:*) mercurial/cffi/mpatch.py: error importing: <ImportError> cannot import name '_mpatch' (error at mpatch.py:*) mercurial/cffi/mpatchbuild.py: error importing: <ModuleNotFoundError> No module named 'cffi' (error at mpatchbuild.py:*) mercurial/cffi/osutilbuild.py: error importing: <ModuleNotFoundError> No module named 'cffi' (error at osutilbuild.py:*) mercurial/thirdparty/concurrent/futures/_base.py: invalid syntax: invalid syntax (<unknown>, line *) mercurial/thirdparty/concurrent/futures/process.py: error importing: <SyntaxError> invalid syntax (_base.py, line 416) (error at __init__.py:*) mercurial/thirdparty/concurrent/futures/thread.py: error importing: <SyntaxError> invalid syntax (_base.py, line 416) (error at __init__.py:*) mercurial/thirdparty/zope/interface/_flatten.py: error importing: <ImportError> cannot import name 'Declaration' (error at _flatten.py:*) Some of them can be suppressed by building cffi modules for example, but I don't think it's worth keeping these modules covered by the compatibility checker.

test-check-py3-compat: remove false output from "python3 check-py3-compat.py" If python3 were python2, these errors would be reported, but we're running check-py3-compat.py on Python 3 here.

test: add a test file that displays ssh behavior in front of various errors Prior to changeset ac0a87160012, these errors were suboptimal since they claim the repository is missing while the actual issue was about permission. This extra test file provides additional documented coverage for the various corner cases.

tests: update test-casefolding.t with recent changes This is caught by the mac-buildbot. This was not found initially because this test requires a case insensitive filesystem. I have edited the test file by hand and I suggest to test this on such a system before pushing. Differential Revision: https://phab.mercurial-scm.org/D4430

py3: don't use dict.iterkeys() in hgext/fastannotate/context.py dict.iterkeys() is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4431

run-tests: replace '/dev/null' with os.devnull for Windows Many of the *.py tests were dying on: File "c:\Users\Matt\projects\hg\hgdemandimport\tracing.py", line 27, in log _pipe = open(os.environ['HGCATAPULTSERVERPIPE'], 'w', 1) IOError: [Errno 2] $ENOENT$: '/dev/null'

stringutil: teach pprint() to indent This will make data structure dumping in various places a bit easier to read and diff. Since I wanted this for `hg debugwireproto` output, I added indentation to it. A more advanced pretty printer implementation would conditionally add newlines if output is too long. But it is vastly simpler to be consistent and always add newlines when indenting. Again, I'm not crazy about the verbosity of the code and there is room to consolidate logic for "print a collection." But this isn't the most complicated code in the world and I'm not convinced it is worth doing. Differential Revision: https://phab.mercurial-scm.org/D4399

zsh_completion: complete merge tools for -t/--tool Differential Revision: https://phab.mercurial-scm.org/D4362

shelve: add an "internal" extra Keeping shelve changeset around increase the risk of collision with normal changesets. To prevent such collision and help with overall clarity, we add an 'internal' key in extra that mark the changeset as created by "shelve". Node changes in tests are expected.

shelve: write metadata file on the fly if they are missing Keeping an explicit reference to the shelve node in order to reuse it directly if the unshelved is repeated (eg: unshelve --keep).

shelve: look for shelved node in the repository before unbundling This prepares the version of shelve that would not strip the shelved node from the repository. If we have the node information, search for it in the repository and only fallback on unbundling if it is missing. To be able to find such nodes, we operate on an unfiltered repository.

shelve: handle shelved node on unfiltered repository To build a version of shelve that use hiding instead of stripping, we need shelve to be able to find these revisions. This it does not hurt the bundle version, so we introduce the necessary code in the relevant place.

shelve: store shelved node in a new data file It is useful for a version of shelve not based on bundle and strip. Having more data does not hurt the bundle based case so we introduce it in all cases before doing more rework. We also keep storing the patch using another method. note: We could have this metadata file list all the file involved in the shelve and use that for cleanup. This would be more future proof than having a hard-coded list of file to purge. However, this is an adventure for another series.

templatekw: add {path} keyword to host documentation It's hidden since I'm not sure if filectx templates can be a thing. The plan is to add {status}, {size}, etc., which are usable within {files % ...} context.