py3: write the file in str mode in tests/test-rename-merge2.t This is better then converting the value the value of sys.argv to bytes. This makes the test pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4457

py3: add missing b'' prefixes in tests/badserverext.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4456

py3: handle keyword arguments correctly in hgext/acl.py The keys of keyword arguments on python 3 should be str, so when we try to get some key from them, we must make sure we are using str. # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4455

py3: don't return the revid as unicode in hgext/convert/subversion.py I tried digging why u'' was added in first place, and I was unable to found something relevant. This might be because some API's takes unicodes, I am not sure. Differential Revision: https://phab.mercurial-scm.org/D4454

py3: make sure we pass str in os.sysconf in hgext/convert/common.py # skip-blame because just r'' prefix Differential Revision: https://phab.mercurial-scm.org/D4453

manifest: fix leak on error return from lazymanifest_filtercopy() Spotted by ASAN. free(copy->lines) and Py_DECREF(copy->pydata) are replaced by Py_XDECREF(copy), which should call lazymanifest_dealloc(). Freeing half-initialized copy->lines is safe since copy->numlines holds a valid value.

manifest: incref/decref copy->pydata to clarify 'copy' holds a reference

dirstate: use tuple interface to fix leak in pack_dirstate() Spotted by ASAN. Unlike PyTuple_GET_ITEM(), PySequence_ITEM() returns a new reference. This bug could be fixed by inserting Py_CLEAR() and Py_XDECREF() appropriately, but I think requiring a tuple object is simpler and less error-prone. The cext version is jumped to 10 since 6..9 are used in the default branch. We'll need to bump it again at merge.

xdiff: fix leak in hunk_consumer() Spotted by ASAN. Since PyList_Append() does not "steal" a reference, Py_DECREF() is always required. Perhaps, this is the largest leak in this series.

base85: fix leak on error return from b85decode() Spotted by ASAN. We don't need to initialize 'out' to NULL, but I decided to do that for clarity.

merge with stable

Added signature for changeset ede3bf31fe63

Added tag 4.7.1 for changeset ede3bf31fe63

hgweb: load revcount + 1 entries to fill nextentry in log page (issue5972) "revcount + 1" is moved to the call site to make it clearer.

revlog: fix size of Python nodetree object Follows up 9f097214fbf3.

revert: stabilize status message of chunks selected interactively Unfortunately, patch.filterpatch() doesn't preserve the order of the input files. We have to sort them manually.

cborutil: add a buffering decoder The sansiodecoder leaves it up to the callers to feed in data that wasn't fully consumed last time. This commit implements a decoder that performs buffering of leftover chunks from the previous invocation. It otherwise behaves identically to sansiodecoder. Differential Revision: https://phab.mercurial-scm.org/D4434

cborutil: remove readindefinitebytestringtoiter() This was implemented as part of implementing streaming encoding. It was never used outside of tests. Now that we have a full CBOR decoder, it can be used for incremental decoding of indefinite-length byte strings. This also removes the last use of the vendored cbor2 package from this module. Differential Revision: https://phab.mercurial-scm.org/D4433

cborutil: implement sans I/O decoder The vendored CBOR package decodes by calling read(n) on an object. There are a number of disadvantages to this: * Uses blocking I/O. If sufficient data is not available, the decoder will hang until it is. * No support for partial reads. If the read(n) returns less data than requested, the decoder raises an error. * Requires the use of a file like object. If the original data is in say a buffer, we need to "cast" it to e.g. a BytesIO to appease the decoder. In addition, the vendored CBOR decoder doesn't provide flexibility that we desire. Specifically: * It buffers indefinite length bytestrings instead of streaming them. * It doesn't allow limiting the set of types that can be decoded. This property is useful when implementing a "hardened" decoder that is less susceptible to abusive input. * It doesn't provide sufficient "hook points" and introspection to institute checks around behavior. These are useful for implementing a "hardened" decoder. This all adds up to a reasonable set of justifications for writing our own decoder. So, this commit implements our own CBOR decoder. At the heart of the decoder is a function that decodes a single "item" from a buffer. This item can be a complete simple value or a special value, such as "start of array." Using this function, we can build a decoder that effectively iterates over the stream of decoded items and builds up higher-level values, such as arrays, maps, sets, and indefinite length bytestrings. And we can do this without performing I/O in the decoder itself. The core of the sans I/O decoder will probably not be used directly. Instead, it is expected that we'll build utility functions for invoking the decoder given specific input types. This will allow extreme flexibility in how data is delivered to the decoder. I'm pretty happy with the state of the decoder modulo the TODO items to track wanted features to help with a "hardened" decoder. The one thing I could be convinced to change is the handling of semantic tags. Since we only support a single semantic tag (sets), I thought it would be easier to handle them inline in decodeitem(). This is simpler now. But if we add support for other semantic tags, it will likely be easier to move semantic tag handling outside of decodeitem(). But, properly supporting semantic tags opens up a whole can of worms, as many semantic tags imply new types. I'm optimistic we won't need these in Mercurial. But who knows. I'm also pretty happy with the test coverage. Writing comprehensive tests for partial decoding did flush out a handful of bugs. One general improvement to testing would be fuzz testing for partial decoding. I may implement that later. I also anticipate switching the wire protocol code to this new decoder will flush out any lingering bugs. Differential Revision: https://phab.mercurial-scm.org/D4414

tests: remove use of string in CBOR test We don't use the CBOR string major type in the wire protocol. Let's not test for it. Differential Revision: https://phab.mercurial-scm.org/D4413

internals: document CBOR utilization I spoke with some people at Mozilla about CBOR and they advised me that we should be careful about the subset of CBOR we use in order to mitigate security, performance, and compatibility concerns. This commit establishes a document that attempts to formalize our use of CBOR. Its main limitations are on what types are allowed. It explicitly enumerates which types are supported. Notable missing features include: * Indefinite-length arrays and maps * Text strings (bytes all the way) * Floats * Date/time types * Big integers * Use of indefinite-length byte strings for map keys, values in containers. If we have a need for any of these, we can have a discussion about them when the time comes. Differential Revision: https://phab.mercurial-scm.org/D4412

py3: add new passing test to whitelist found by buildbot Differential Revision: https://phab.mercurial-scm.org/D4451

py3: alias xrange to range in tests/test-fastannotate-revmap.py xrange is not available on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4450

py3: add r'' prefix to prevent b'' being prepended The keys of keyword arguments should be str. # skip-blame because we are adding just r'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D4449

revert: fix the inconsistency of status msgs in --interactive mode Before this patch we were priting every action msg before actually performing that action and that was resulting in inconsistencies; like in --interactive session if user decided to not revert any changes in a file foo, still there will be a msg on console saying "reverting foo". To fix this, I have made some changes to print status msg just before the action it is going to perform, no matter if --interactive or not. Changes made in test-revert-interactive.t reflect the changed behavior. There are also some changes in test-revert.t because of change in the order of messages. Differential Revision: https://phab.mercurial-scm.org/D4380

revert: remove unnecessary check as `msg` should never be a callable now Differential Revision: https://phab.mercurial-scm.org/D4448

run-tests: drop support for --with-python3 and $PYTHON3 This backs out cdbc25306696. We can run many tests on python3 without this hack.

test-check-py3-compat: don't run py3 part if not build against python3 This test depends on C extensions built for python3, which aren't available if run-tests.py runs on python2 and no -l/--local is specified.

test-check-py3-compat: don't run py2 part if $PYTHON is python3

resolve: test that we suggest a valid hg command Differential Revision: https://phab.mercurial-scm.org/D4447

bash_completion: complete arguments for --color and --pager

bash_completion: add more global options that receive an argument

tracing: ignore any IOErrors when writing to pipe When the pager forks off the main process, we can end up with the pipe closed prematurely. Rather than break hg entirely when that happens and tracing is active, just let lingering events disappear as needed. Differential Revision: https://phab.mercurial-scm.org/D4445

tests: avoid shellquoting bytes on Python 3 Should fix Python 3 builder. Differential Revision: https://phab.mercurial-scm.org/D4444

resolve: ui.config -> ui.configbool, for better validation Differential Revision: https://phab.mercurial-scm.org/D4446

zsh_completion: handle --rev as well as -r for diff and revert Completion for hg diff and revert should suggest all files in context of a non-current revision. The script used to look only for `-r foo`, and now it also understands `--rev foo`. Differential Revision: https://phab.mercurial-scm.org/D4427

resolve: add config to make hg resolve not re-merge by default Before this, calling 'hg resolve' with neither -m, -u or -l will re-merge. This is highly error prone (it's easy to forget to forget a -m), and pretty bad when it happens (many people have no idea 'hg resolve' can re-merge, and end up redoing the work of resolving all the conflicts, because they have no idea there was a backup of their work). Differential Revision: https://phab.mercurial-scm.org/D4379

resolve: add a flag for the default behavior of re-merging On its own, it's not useful, but the next commit will add an hgrc config option to make it mandatory. There is no short option, as -r almost always means --rev and this option doesn't seem like it would be so common as to mandate a short option. Differential Revision: https://phab.mercurial-scm.org/D4378

tests: conditionalize narrow-widen error output for Windows

largefiles: use a context manager to control the progress bar lifetime

lfs: use a context manager to control the progress bar lifetime

hg: ensure the progress bar is completed when copying the store This is just a block indent under the context manager.

shelve: fix crash on unshelve without .shelve metadata file Follow up for c67c94c0e7ae and 38373da1af02. The inline comment says "we should keep track of the unshelve node in case we need to reuse it." Perhaps such case isn't tested, and this patch does NOT add a test for the reuse of the unbundled revision. Also, I have no idea what should be done if new revision is unbundled because of "node not in repo".

fastannotate: use stringutil.pprint() to dump diffopts to be hashed

test-check-py3-compat: update "python3 check-py3-compat.py" output

test-check-py3-compat: exclude thirdparty and cffi which are known to fail Otherwise the following errors would be spilled out: mercurial/cffi/bdiff.py: error importing: <ImportError> cannot import name '_bdiff' (error at bdiff.py:*) mercurial/cffi/bdiffbuild.py: error importing: <ModuleNotFoundError> No module named 'cffi' (error at bdiffbuild.py:*) mercurial/cffi/mpatch.py: error importing: <ImportError> cannot import name '_mpatch' (error at mpatch.py:*) mercurial/cffi/mpatchbuild.py: error importing: <ModuleNotFoundError> No module named 'cffi' (error at mpatchbuild.py:*) mercurial/cffi/osutilbuild.py: error importing: <ModuleNotFoundError> No module named 'cffi' (error at osutilbuild.py:*) mercurial/thirdparty/concurrent/futures/_base.py: invalid syntax: invalid syntax (<unknown>, line *) mercurial/thirdparty/concurrent/futures/process.py: error importing: <SyntaxError> invalid syntax (_base.py, line 416) (error at __init__.py:*) mercurial/thirdparty/concurrent/futures/thread.py: error importing: <SyntaxError> invalid syntax (_base.py, line 416) (error at __init__.py:*) mercurial/thirdparty/zope/interface/_flatten.py: error importing: <ImportError> cannot import name 'Declaration' (error at _flatten.py:*) Some of them can be suppressed by building cffi modules for example, but I don't think it's worth keeping these modules covered by the compatibility checker.

test-check-py3-compat: remove false output from "python3 check-py3-compat.py" If python3 were python2, these errors would be reported, but we're running check-py3-compat.py on Python 3 here.

test: add a test file that displays ssh behavior in front of various errors Prior to changeset ac0a87160012, these errors were suboptimal since they claim the repository is missing while the actual issue was about permission. This extra test file provides additional documented coverage for the various corner cases.

tests: update test-casefolding.t with recent changes This is caught by the mac-buildbot. This was not found initially because this test requires a case insensitive filesystem. I have edited the test file by hand and I suggest to test this on such a system before pushing. Differential Revision: https://phab.mercurial-scm.org/D4430

py3: don't use dict.iterkeys() in hgext/fastannotate/context.py dict.iterkeys() is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4431

run-tests: replace '/dev/null' with os.devnull for Windows Many of the *.py tests were dying on: File "c:\Users\Matt\projects\hg\hgdemandimport\tracing.py", line 27, in log _pipe = open(os.environ['HGCATAPULTSERVERPIPE'], 'w', 1) IOError: [Errno 2] $ENOENT$: '/dev/null'

stringutil: teach pprint() to indent This will make data structure dumping in various places a bit easier to read and diff. Since I wanted this for `hg debugwireproto` output, I added indentation to it. A more advanced pretty printer implementation would conditionally add newlines if output is too long. But it is vastly simpler to be consistent and always add newlines when indenting. Again, I'm not crazy about the verbosity of the code and there is room to consolidate logic for "print a collection." But this isn't the most complicated code in the world and I'm not convinced it is worth doing. Differential Revision: https://phab.mercurial-scm.org/D4399

zsh_completion: complete merge tools for -t/--tool Differential Revision: https://phab.mercurial-scm.org/D4362

shelve: add an "internal" extra Keeping shelve changeset around increase the risk of collision with normal changesets. To prevent such collision and help with overall clarity, we add an 'internal' key in extra that mark the changeset as created by "shelve". Node changes in tests are expected.

shelve: write metadata file on the fly if they are missing Keeping an explicit reference to the shelve node in order to reuse it directly if the unshelved is repeated (eg: unshelve --keep).

shelve: look for shelved node in the repository before unbundling This prepares the version of shelve that would not strip the shelved node from the repository. If we have the node information, search for it in the repository and only fallback on unbundling if it is missing. To be able to find such nodes, we operate on an unfiltered repository.

shelve: handle shelved node on unfiltered repository To build a version of shelve that use hiding instead of stripping, we need shelve to be able to find these revisions. This it does not hurt the bundle version, so we introduce the necessary code in the relevant place.

shelve: store shelved node in a new data file It is useful for a version of shelve not based on bundle and strip. Having more data does not hurt the bundle based case so we introduce it in all cases before doing more rework. We also keep storing the patch using another method. note: We could have this metadata file list all the file involved in the shelve and use that for cleanup. This would be more future proof than having a hard-coded list of file to purge. However, this is an adventure for another series.

templatekw: add {path} keyword to host documentation It's hidden since I'm not sure if filectx templates can be a thing. The plan is to add {status}, {size}, etc., which are usable within {files % ...} context.

status: rename {copy} to {source} for compatibility with {file_copies} (BC) .. bc:: ``{copy}`` in status command template is renamed to ``{source}``.

formatter: rename {abspath}/{file} to {path}, and drop relative {path} (BC) Note that {path} in status is either relative-to-cwd or repository-absolute depending on the command argument and config knob, which can't be reproduced by using the {path|relpath} filter. The default template is updated to always use a relative path. .. bc:: ``{abspath}`` and ``{file}`` in generic templates are renamed to ``{path}``. Any ``{path}`` is a repository-absolute path. Use ``{path|relpath}`` to convert it to a filesystem path.

templatekw: alias {name} of file copies dict to {path} For the same reason as the previous patch. We might want some hack to support {source.path}, {source.rev}, etc., but that's a different issue.

templatekw: alias {file} of files list to {path} This is a part of the name unification. All {path}s will be changed to repository-absolute (i.e. canonical) paths. https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary

copies: improve logic of deciding copytracing on based of config options Few months ago or maybe a year ago, I imported Fb's heuristics based copytracing algorithms. While importing that, I renamed `experimental.disablecopytrace` with `experimental.copytrace` and the behavior of the new config option was like this: * "heuristics" : Fb's heuristic copytracing algorithm * "off" : copytracing is turned off * something else: copytracing is on This is the behavior right now also and this is bad because it hardcodes the string 'off' to turn off the copytracing. On big repositories, copytracing is very slow and people wants to turn copytracing off. However if the user sets it to 'False', 'Off', '0', none of them is going to disbale copytracing while they should. I lacked the understanding of why this can be bad when I coded it. After this patch, the new behavior of the config option will be: * "heuristics": Fb's heuristic copytracing algorithm * '0', 'false', 'off', 'never', 'no', 'NO', all the values which repo.ui.configbool() evaluates to False: copytracing in turned off * something else: copytracing is on Since 'off' still evaluates to copytracing being turned off, this is not BC. Also the config option is experimental. Differential Revision: https://phab.mercurial-scm.org/D4416

relnotes: enable extension when running releasenotes command Last night, when I was working on different machine, the test worked, now on a different machine it does not. Looks like we should make sure releasenotes extension is loaded. Differential Revision: https://phab.mercurial-scm.org/D4425

narrow: rename getbundlechangegrouppart_nonellipsis function The function is renamed to getbundlechangegrouppart_widen as per suggestion in D4383. The motivation is such that we can evolve the function into a handler for new wire-protocol command. Differential Revision: https://phab.mercurial-scm.org/D4424

tests: rename test-narrow-widen-non-ellipsis to match current names The test is renamed to test-narrow-widen-no-ellipsis (note non -> no), to match with test-narrow-clone-no-ellipsis.t. Follows up on D4382. Differential Revision: https://phab.mercurial-scm.org/D4423

contrib: byteify perf.py file This is done using contrib/byteify-strings.py. There is just one extra change done to make sure lines don't get longer than 80 chars. # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D4422

tests: use {rev} instead of {node|short} and (glob) in narrow tests I think we used {node|short} because they were first written for only flat manifests and we didn't think to change the format when we started testing tree manifests. Differential Revision: https://phab.mercurial-scm.org/D4421

shortest: fix an off-by-1 when disambiguating against revnums Revision "5" is an unambiguous hex prefix in a repo with 5 revisions; it's only "4" and lower that are ambiguous. Differential Revision: https://phab.mercurial-scm.org/D4420

filemerge: fix an i18n comment typo

tests: add flat manifest case in test-narrow-widen-non-ellipsis.t We had this test with tree manifest only because the tests were broken and we were fixing them and maintaing two different cases of broken tests was bit hard. Now that things work fine, let's add the flat manifest case. The test shows that things work fine in both cases. Differential Revision: https://phab.mercurial-scm.org/D4385

narrow: update the narrowspecs to new ones after pulling when widening The ellipsis case updates the new narrowspecs at bundle2 handling code, and non-ellipsis widening didn't use to work and never set the new narrowpats. Previous pacth added server side logic for widening without ellipsis. This patch adds the missing repo.setnarrowpats() so that we set the new narrowpats after widening in non-ellipsis cases also. This makes widening with non-ellipsis case work but there are certain things which we can improve. Differential Revision: https://phab.mercurial-scm.org/D4384

narrow: add server logic to send cg while widening without ellipsis Before this patch, if you try to widen a narrow clone without ellipsis enabled, it will be broken and the exchange.pull() done by tracked command to widen the clone will be no-op because no custom logic exists for this and server sees that we have all csets and it says `no changes found`. The widening with ellipsis send KILL for existing changegroups and send new changegroups because of the change in ellipsis hash, but we can prevent that in non-ellipsis cases. This patch adds server side logic to send the changegroups for the changesets which are on the client again with filelogs and manifests for the new includes. This is a very starting implementation and we send changegroups and manifests too while we can prevent them. Following things can definitely be improved in the logic this patch adds: 1) Send just the filelogs and treemanifests 2) Send the filelogs only for the additions in the include I tried 1) here but the code is coupled tightly and the way I was able to do that was hacking into the changegroup generation code in a very dirty way, like adding conditionals and preventing the yield. This patch also adds a 'widen' kwarg to prevent other commands except widening to go through that codepath. The test changes demonstrate that the new implementation is correct and fixes things. Differential Revision: https://phab.mercurial-scm.org/D4383

tests: add test to show widening is broken without ellipsis This patch adds a test to show that widening a narrow clone is broken if ellipsis is disabled. I don't think I can add cases to existing test-narrow-widen.t and check-in a failing version of that. So I created a copy. Once the test is fixed, we can merge this new test file back into the original one using testcases. Also, this is just testing treemanifest case because having two cases and both or are failing with different outputs is a bit hard to manage. This is important because upcoming patches will try to fix the broken part. Differential Revision: https://phab.mercurial-scm.org/D4382

stringutil: emit multiple chunks when pretty printing This avoids concatenating output inside pprintgen() itself. But the real reason for this is it will make it easier to add indentation, as we'll need to account for indentation when emitting each individual object in a collection. The verbosity of this code compared to the original is a bit unfortunate. But I suppose this is the price to pay for having nice things (streaming and indenting). We could probably abstract the "print a collection" bits into a generic function to avoid some duplication. But I'm not overly inclined to do this. Differential Revision: https://phab.mercurial-scm.org/D4398

stringutil: refactor core of pprint so it emits chunks This commit splits the core of pprint() to a new function that is a generator of chunks instead of a function returning a single value. This will make it possible to stream output without waiting for all data to be formatted first. And it will make it easier to implement support for indenting. Differential Revision: https://phab.mercurial-scm.org/D4397

contrib: import the relnotes script from the release-tools repo I figure this makes more sense to keep in the main repo, as it's a guide of sorts on how to use the releasenotes extension in the presence of commits that don't get relnotes annotations. Ported to Python 3, cleaned up some logic in a few places, but for the most part it's what we've been using for years. Differential Revision: https://phab.mercurial-scm.org/D4291

hghave: move from requiring the PYTHON3 env var to looking for `python3` We're starting to write new utility scripts in Python 3, and I think it's reasonable to expect `python3` to be something we should use if it's present. I wrote the regex to specifically ban Python 3 prior to Python 3.5 because we won't ever support that. I moved the only test that was using $PYTHON3 to just using `python3` directly. Differential Revision: https://phab.mercurial-scm.org/D4290

rename: return error status if any rename/copy failed Ever since 447ea621e50e (copy: propagate errors properly, 2007-12-06), we have returned an error status if the source file did not exist. That commit did not return error status for any other errors, and it's unclear if that was on purpose or not. It seems to me like we should return an error in the other cases to, so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D4419

rename: emit hint about using --after consistently Both the code and the message for the case where the source file was missing was inconsistent with the other similar messages. Differential Revision: https://phab.mercurial-scm.org/D4418

rename: quote hg commands in warnings I think we usally use single quotes around hg commands in messages. Differential Revision: https://phab.mercurial-scm.org/D4417

zsh_completion: complete hg files Differential Revision: https://phab.mercurial-scm.org/D4361

zsh_completion: adjust usage of common options addremove, remove and serve accept -S/--subrepos, and purge actually doesn't. qnew accepts -I/--include and -X/--exclude. qrefresh accepts -D/--currentdate, -U/--currentuser, -d/--date and -u/--user. rebase accepts -n/--dry-run. Differential Revision: https://phab.mercurial-scm.org/D4360

resolve: make condition clearer, and able to handle a new flag Differential Revision: https://phab.mercurial-scm.org/D4377

shelve: drop mention of obsshelve file Core shelve extension does not use obsolescence, we clean up this unused file before adding a new info file.

shelve: rename method for data write/read There are no user of this code nor any code related to obsshelve in the extensions right now, so we rename the function before using them. Note: The shelvedfile looks a bit strange. We should probably refactor it, however, this is an adventure for another series.

shelve: move createcmd next to _docreatecmd It is simpler not to have to jump around the file all the time.

minirst: filter blocks by full path to section

minirst: mark getsections() as an internal helper

help: rewrite parsing of help topic to not drop section name with dots A subtopic is no longer lowercased since it should be considered a part of a topic path, not a section filter.

test-help: add tests showing that sections can't be filtered by dotted name

revlogdeltas: move finddeltainfo filtering inside _candidategroups Now all "simple" filtering logic is gathered in one place. The `finddeltainfo` method is more straightforward and the `_candidate_groups` function knows more about filtering revisions before we compute a delta against them.

revlogdeltas: split candidate groups selection from the filtering logic The group iteration has two main components: * walking candidates, the logic that we are about to extend to build intermediate snapshots, * Making sure we test diffs against interesting bases. No duplicated tests, skipping empty revisions, etc. We split `_candidategroups` to separate the two components. This achieves two goals: * It will be simpler to update the walking logic for intermediate snapshots, * We can gather the filtering logic from `finddeltainfo` into `_candidategroups` to centralize it.

revlogdeltas: pass revision number to _candidatesgroups The `_candidates_groups` logic works on revisions, so the nodes are currently just converted to revs at the top of the function. It seems cleaner to instead pass revision numbers to the function. Having this conversion out of the way will help for further cleanups.

revlogdeltas: extract _getcandidaterevs in a function The logic barely uses the object it is attached to. This is an important function that we will clean up in the coming changesets. Moving it at the top level helps us with that cleanup.

revlogdeltas: always return a delta info object in finddeltainfo Previously, the method returned `None` when a full snapshot was needed. The caller had to determine how to produce one itself. In practice, building a `_deltainfo` object for a full snapshot is simple. So we build it at the `finddeltainfo` level and always return a `_deltainfo` object. The caller can now simply process the `_deltainfo` return in all cases.

revlogdeltas: move special cases around raw revisions in finddeltainfo The method already contains logic for no-diff cases. Having everything in the same place is more consistent and unlocks other code improvements.

revlogdeltas: extra fulltext building in its own function The process of building a full text from a delta is complex enough to deserve isolation. The previous function mixed cache management and computation, we now have clearer boundaries and simpler functions/methods.

revlog: split functionality related to deltas computation in a new module The revlog module is getting big and this logic is getting more and more advanced. Moving it to `mercurial.revlogutils.deltas` split a lot off revlog.py and will help this logic to become less interleaved with revlog. The code is simply moved without modification (but for namespace changes). Refactoring and improvement will be made in later changesets.

revlog: split constants into a new `revlogutils.constants` module We want to split some logic out of the main revlog file (the delta computing logic). However, this logic needs access to multiple constants related to the revlog. So we move all revlog related constants into a new module that could be imported from multiple places. We don't copy the file (preserving blame history) because there are only a few moving lines. Also, copying the file would result in annoying merge conflicts with ongoing work from others contributors.

rebase: skip extinct revisions even if it has no successor in rebase set Differential Revision: https://phab.mercurial-scm.org/D4408

tests: add test showing that rebase of extinct commit with successor fails As the test case shows, attempting to rebase a commit that has a successor that is not in the rebase set and not in the destination currently fails because it "will cause divergences". However, it doesn't seem like there's any harm in skipping the extinct commit. I suspect this case missed simply because extinct revisions are usually hidden. Differential Revision: https://phab.mercurial-scm.org/D4407

tests: add log output after stripping for context for next test It's not obvious what the repo state will be after stripping some nodes and obsmarkers, so let's include the log output to make it easier to follow the test case that comes after. Differential Revision: https://phab.mercurial-scm.org/D4406

obsutil: make sure "addedmarkers" is not None in getobsoleted() Similar reasoning as in 52e6171ec822 (obsutil: make sure "addedrevs" is not None in getobsoleted(), 2018-08-25). Differential Revision: https://phab.mercurial-scm.org/D4411

revsetlang: fix position of '-' in spaceless 'a-b' expressions I don't know if there is an observable difference, but it seems like an obvious fix. Differential Revision: https://phab.mercurial-scm.org/D4410

tests: mark manifestfulltextcache as conditional on revlog store This file is currently specific to the revlog store. Other stores may not implement it. Mark it as such in test output. Differential Revision: https://phab.mercurial-scm.org/D4395

manifest: use fulltextcache instead of _fulltextcache fulltextcache is part of the documented interface and is aliased to _fulltextcache. Differential Revision: https://phab.mercurial-scm.org/D4394

manifest: rename manifestlog._revlog to _rootstore Because it may not be a revlog. Differential Revision: https://phab.mercurial-scm.org/D4393

manifest: use public API for obtaining storage object We should not be accessing private attributes on the manifestlog, since its behavior is governed by an interface. Differential Revision: https://phab.mercurial-scm.org/D4392

perf: add function for obtaining manifest revision This will use the modern API if available or fall back to the legacy direct revlog access case. Without this, the perf command won't work on all code bases. Differential Revision: https://phab.mercurial-scm.org/D4391

perf: use storage API for resolving manifest node lookup() isn't part of the storage API. And this code shouldn't be accessing manifestlog._revlog directly for the modern code base. So let's port it to the modern API. Note that the previous code was busted for cases where we needed to call lookup() because lookup() isn't exposed by manifestrevlog any more. This change is strictly BC breaking because we no longer support resolving partial nodes. But it is a perf* command and I don't think we should flag the change as such. Differential Revision: https://phab.mercurial-scm.org/D4390

manifest: change terminology for storage in context classes Storage may not be backed by revlogs. So it is wrong to refer to storage as "revlog" or use abbreviations like "rl." This commit replaces revlog terminology in manifest context classes and starts referring to things as "store" or "storage." Differential Revision: https://phab.mercurial-scm.org/D4389

manifest: don't go through revlog to access node symbols The revlog module re-exports some node.* symbols courtesy of importing them. Let's access the node.* symbols via our local import. Differential Revision: https://phab.mercurial-scm.org/D4388

manifest: make tree a public attribute changegroup generation accesses this attribute. We should make it public and expose it on the interface. Differential Revision: https://phab.mercurial-scm.org/D4387

manifest: proxy to revlog instance instead of inheriting Previously, manifestrevlog inherited revlog.revlog and therefore exposed all its APIs. This inevitably resulted in consumers calling low-level revlog APIs. As part of abstracting storage, we want to formalize the interface for manifest storage. The revlog API is much too large to define as the interface. Like we did for filelog, this commit divorces the manifest class from revlog so that we can standardize on a smaller API surface. The way I went about this commit was I broke the inheritance, ran tests, and added proxies until all tests passed. Like filelog, there are a handful of attributes that don't belong on the interface. And like filelog, we'll tease these out in the future. As part of this, we formalize an interface for manifest storage and add checks that manifestrevlog conforms to the interface. Adding proxies will introduce some overhead due to extra attribute lookups and function calls. On the mozilla-unified repository: $ hg verify before: real 627.220 secs (user 525.870+0.000 sys 18.800+0.000) after: real 628.930 secs (user 532.050+0.000 sys 18.320+0.000) $ hg serve (for a clone) before: user 223.580+0.000 sys 14.270+0.000 after: user 227.720+0.000 sys 13.920+0.000 $ hg clone before: user 506.390+0.000 sys 29.720+0.000 after: user 513.080+0.000 sys 28.280+0.000 There appears to be some overhead here. But it appears to be 1-2%. I think that is an appropriate price to pay for storage abstraction, which will eventually let us have much nicer things. If the overhead is noticed in other operations (whose CPU time isn't likely dwarfed by fulltext resolution) or if we want to cut down on the overhead, we could dynamically build up a type whose methods are effectively aliased to a revlog instance's. I'm inclined to punt on that problem for now. We may have to do it for the changelog. At which point it could be implemented in a generic way and ported to filelog and manifestrevlog easily enough I would think. .. api:: manifest.manifestrevlog no longer inherits from revlog The manifestrevlog class now wraps a revlog instance instead of inheriting from revlog. Various attributes and methods on instances are no longer available. Differential Revision: https://phab.mercurial-scm.org/D4386

pager: do not enable when TERM=dumb Before this, running hg in emacs' M-x shell started `less`, which would complain about the terminal not being fully functional. Differential Revision: https://phab.mercurial-scm.org/D4376

localrepo: do not cache auditor/nofsauditor which would make reference cycle Before, self.auditor and self.nofsauditor held self through self._checknested, and the following code couldn't free a repo by ref-counting. def main(): repo = localrepo.localrepository(uimod.ui(), '../testrepos/hello') main() With this change, the cache of the nofsauditor is limited to a single match session. I think that's okay as the nofsauditor doesn't do any filesystem access. Alternatively, maybe we can remove the callback from nofsauditor since it isn't used unless realfs=True, but I have no idea whether it is a bug or not.

minirst: remove unused function decorateblocks() It was added at ee6988aea74e "minirst: add decorateblocks search helper", but it's been unused since then.

minirst: make format() simply return a formatted text It's a source of bugs to change the type of the return value conditionally.