formatter: use stringutil.pprint() in debug output to drop b''

rust: exposing in parsers module To build with the Rust code, set the HGWITHRUSTEXT environment variable. At this point, it's possible to instantiate and use a rustlazyancestors object from a Python interpreter. The changes in setup.py are obviously a quick hack, just good enough to test/bench without much refactoring. We'd be happy to improve on that with help from the community. Rust bindings crate gets compiled as a static library, which in turn gets linked within 'parsers.so' With respect to the plans at https://www.mercurial-scm.org/wiki/OxidationPlan this would probably qualify as "roll our own FFI". Also, it doesn't quite meet the target of getting rid of C code, since it brings actually more, yet: - the new C code does nothing else than parsing arguments and calling Rust functions. In particular, there's no complex allocation involved. - subsequent changes could rewrite more of revlog.c, this time resulting in an overall decrease of C code and unsafety.

rust: iterator bindings to C code In this changeset, still made of Rust code only, we expose the Rust iterator for instantiation and consumption from C code. The idea is that both the index and index_get_parents() will be passed from the C extension, hence avoiding a hard link dependency to parsers.so, so that the crate can still be built and tested independently. On the other hand, parsers.so will use the symbols defined in this changeset.

rust: pure Rust lazyancestors iterator This is the first of a patch series aiming to provide an alternative implementation in the Rust programming language of the _lazyancestorsiter from the ancestor module. This iterator has been brought to our attention by the people at Octobus, as a potential good candidate for incremental "oxydation" (rewriting in Rust), because it has shown performance issues lately and it merely deals with ints (revision numbers) obtained by calling the index, whih should be directly callable from Rust code, being itself implemented as a C extension. The idea behind this series is to provide a minimal example of Rust code collaborating with existing C and Python code. To open the way to gradually rewriting more of Mercurial's Python code in Rust, without being forced to pay a large initial cost of rewriting the existing fast core into Rust. This patch does not introduce any bindings to other Mercurial code yet. Instead, it introduces the necessary abstractions to address the problem independently, and unit-test it. Since this is the first use of Rust as a Python module within Mercurial, the hg-core crate gets created within this patch. See its Cargo.toml for more details. Someone with a rustc/cargo installation may chdir into rust/hg-core and run the tests by issuing: cargo test --lib The algorithm is a bit simplified (see details in docstrings), and at its simplest becomes rather trivial, showcasing that Rust has batteries included too: BinaryHeap, the Rust analog of Python's heapq does actually all the work. The implementation can be further optimized and probably be made more idiomatic Rust.

run-tests: restore quoting the python executable for running *.py tests This was accidentally dropped in 8cf459d8b111.

tests: replace `cd ..` with an absolute path in a couple ssh tests These tests are broken under py3 on Windows to the point where the `cd ..` was actually escaping into the system wide $TEMP. The subsequent `hg init` created a repo there, and then added a local extension to the hgrc. This breaks every single subsequent test when it tries to `hg init` in its $TESTTMP, and can't load the localwrite.py extension. And since I botched this the first time and replaced the wrong `cd ..`, this just replaces all of them. I've noticed test garbage in $TEMP recently, and maybe this will help. Perhaps `hg init` shouldn't load the config for the local repo, but this is an easy enough workaround for now.

lfs: register the flag processors per repository Previously, enabling the extension for any repo in commandserver or hgweb would enable the flags on all repos. Since localrepo.resolverevlogstorevfsoptions() is called so early, the check to see if the extension is enabled on the repo (which hasn't been instantiated yet) is a bit awkward. But I don't see a better way.

revlog: allow flag processors to be applied via store options This allows flag processors to be registered to specific repos in an extension by wrapping localrepo.resolverevlogstorevfsoptions(). I wanted to add the processors via a function on localrepo, but some of the places where the processors are globally registered don't have a repository available. This makes targeting specific repos in the wrapper awkward, but still manageable.

py3: use str to query registry values on Windows This blew up launching any command if extdiff processed a tool with a regkey config.

py3: convert "usage" literal to bytes Here _() is practically an identity function, but we shouldn't pass in unicode to _().

churn: remove redundant round() To my surprise, the int() is required. Spotted by Mads when he reviewed D5063. Thanks! Differential Revision: https://phab.mercurial-scm.org/D5086

py3: use py3 as the test tag, dropping the k Differential Revision: https://phab.mercurial-scm.org/D5079

tests: fix inline extension in test-fncache.t for Python 3 # skip-blame just some b prefixing Differential Revision: https://phab.mercurial-scm.org/D5083

py3: 3 more passing tests Differential Revision: https://phab.mercurial-scm.org/D5082

simplemerge: port to Python 3 Differential Revision: https://phab.mercurial-scm.org/D5081

contrib: fix up output in check-config.py to use strs to avoid b prefixes Differential Revision: https://phab.mercurial-scm.org/D5059

context: open files in bytes mode I'm stunned this open() call has survived this long without the b in the mode - it seems like it should have been a source of bugs somewhere... Differential Revision: https://phab.mercurial-scm.org/D5080

tests: fix up test-hghave for recent run-tests change to use more CPUs Differential Revision: https://phab.mercurial-scm.org/D5085

py3: fix test-parse-date.t Differential Revision: https://phab.mercurial-scm.org/D5084

obsolete: don't translate internal error message AFAIK, it's caught only by "hg debugobsolete", so it's pretty much a programming error.

py3: get around unicode docstrings in test-encoding-textwrap.t and test-help.t On Python 3, docstrings are converted back to utf-8 bytes, which practically disables the "if type(message) is pycompat.unicode" hack in gettext(). Let's add one more workaround for the Py3 path.

crecord: make enter move cursor down to the next item of the same type Let's replace experimental.spacemovesdown with a separate key: Enter, since it wasn't used for anything in crecord. Not sure if '\n' works on Windows though. nextsametype() strictly only moves to items of the same type as the current item. This, for example, allows to go over individual lines in a diff and skip hunk and file headers (which would toggle multiple lines).

py3: fix infinitepush extension tests Differential Revision: https://phab.mercurial-scm.org/D5078

py3: build help of compression engines in bytes Removes "b''" from help.

py3: do I/O in bytes in test-help.t

tests: accept slightly different zip file in Python 3 I added some `unzip -t` here and I *think* the only change is from Python 3 having more data in the zip file headers or something. Sigh. Differential Revision: https://phab.mercurial-scm.org/D5075

webcommands: use stringutil.pprint() to repr invalid archive types Differential Revision: https://phab.mercurial-scm.org/D5074

archival: don't try and fsdecode non-{bytes,str} objects This function accepts both bytes and file-like objects. Differential Revision: https://phab.mercurial-scm.org/D5073

tests: fix last failure in test-tools.t Differential Revision: https://phab.mercurial-scm.org/D5072

run-tests: run tests with as many processes as cores by default This seems like a useful default behavior so tests run faster by default* * Except in special circumstances where the OS/filesystem doesn't scale well to many CPU cores (like APFS *cough* *cough*). Differential Revision: https://phab.mercurial-scm.org/D5071

run-tests: print number of tests and parallel process count This seems like a useful output message to have. I also sneak in a change to lower the parallel process count if it is larger than the number of tests, as that makes no sense and output saying we're running more tests in parallel than there exists tests would be wonky. Differential Revision: https://phab.mercurial-scm.org/D5070

releasenotes: use stringutil.wrap() instead of handcrafted TextWrapper wrapper It's silly to splitlines() a joined string, but we don't care the performance here.

match: optimize matcher when all patterns are of rootfilesin kind Internally at Google, we use narrowspecs with only rootfilesin-kind patterns. Sometimes there are thousands of such patterns (i.e. thousands of tracked directories). In such cases, it can take quite long to build and evaluate the resulting matcher. This patch optimizes matchers that have only patterns of rootfilesin so it instead of creating a regular expression, it matches the given file's directory against the set of directories. In a repo with ~3600 tracked directories, it takes about 1.35 s to build the matcher and 2.7 s to walk the dirstate before this patch. After, it takes 0.04 s to create the matcher and 0.87 s to walk the dirstate. It may be worthwhile to do similar optimizations for e.g. patterns of type "kind:", but that's not a priority for us right now. Differential Revision: https://phab.mercurial-scm.org/D5058

churn: use integer division consistently This results in slight output changes, but it's at least consistent between Python 2 and 3. Since the output is just bar graphs anyway, I'm content with the changes. Differential Revision: https://phab.mercurial-scm.org/D5063

churn: fix stack traces on Python 3 Differential Revision: https://phab.mercurial-scm.org/D5062

py3: moar passing tests Differential Revision: https://phab.mercurial-scm.org/D5055

py3: one new passing tests Differential Revision: https://phab.mercurial-scm.org/D5006

tests: expand our coverage of errors in Python 3 for bad extensions What a mess. Differential Revision: https://phab.mercurial-scm.org/D5005

tests: add lots of globs and conditional output lines The test now passes on Python 3. I'm going to do one follow-up where I'll adjust some grep calls to improve our checking of things on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5004

releasenotes: fix remaining bytes/unicode issues caught by tests All tests now pass. Differential Revision: https://phab.mercurial-scm.org/D5054

relnotes: port to Python 3 The big annoyance here was having to feed textwrap unicodes instead of bytes, but it all seems to work. Differential Revision: https://phab.mercurial-scm.org/D5053

py3: use __code__ instead of func_code to test if func is cachable Surprisingly, this fixes test-cache-abuse.t, which would generate a different revbranch cache if filtered set had no cache.

py3: pass in system string to vars(branchmap).__contains__()

test-bookmarks-pushpull: use correct tmp dir for no-bm-move.sh test-bookmarks-pushpull.t writes a temporary file in $TESTDIR instead of $TESTTMP. Make it use $TESTTMP instead. Differential Revision: https://phab.mercurial-scm.org/D5057

py3: add some glob to make output compatible with python 3 This makes test-pathconflicts-basic.t and test-origbackup-conflict.t passing on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5042

py3: add a b'' prefix in tests/test-requires.t This makes the test pass on Python 3. # skip-blame because just b'' prefix. Differential Revision: https://phab.mercurial-scm.org/D5041

py3: use '%d' for rev nums instead of '%s' This makes test-close-head.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5040

py3: more globing of things to make output compatible between py2 and py3 This also makes the test-ssh-repoerror.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5038

py3: use stringutil.pprint() instead of '%r' This makes test-pager-legacy.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5037

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D5036

py3: add some py3 specific output to test This makes the test work on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5035

py3: use stringutil.pprint() in color.py instead of '%r' This fixes the test-status-color.t on python 3. Differential Revision: https://phab.mercurial-scm.org/D5034

py3: add b'' prefixes in tests/test-progress.t This makes the test pass on Python 3. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5033

py3: add b'' prefixes to tests/test-keyword.t This test is now pretty close to passing on Python 3. # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D5032

py3: use stringutil.pprint() if we are printing bool values Differential Revision: https://phab.mercurial-scm.org/D5031

py3: glob some difference between py2 and py3 output On py2, the directory names are under quotes and on py3 they are not and I don't know why. Differential Revision: https://phab.mercurial-scm.org/D5030

py3: byteify tests/wireprotosimplecache.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5029

py3: add b'' prefixes in tests/test-help.t This takes us close to make the test passing. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5028

py3: use sys.stdout.buffer for binary output in tests/svnxml.py Just following 6c0b1d964537ab62d76c208f5f04ab414814c94e here. Differential Revision: https://phab.mercurial-scm.org/D5027

py3: use '%s' instead of %r in hgext/convert/subversion.py Using '%r' on bytes in python 3 adds b'' prefixes to output which is bad. I don't see a reason why we want to repr() the string here, so just switched to '%s'. This fixes some output failure in one of the subversion test. Differential Revision: https://phab.mercurial-scm.org/D5026

py3: make test-contrib-perf.t work on python 3 Differential Revision: https://phab.mercurial-scm.org/D5025

py3: add 3 more passing tests to whitelist caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5024

py3: add some r'' prefixes in hgext/narrow/narrowwirepeer.py This fixes test-narrow-widen-no-ellipsis.t on Python 3. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5023

py3: r'' prefix default values for mimetypes.guess_mime() As suggested by @yuja in D4967. Differential Revision: https://phab.mercurial-scm.org/D5020

absorb: update help text Update the help text to reflect the recent changes to absorb in D4999. The `-p` option actually only affects the `-a` option: without `-a`, the changes are printed; with `-a` the changes are only printed if `-p` is specified. Reword the help text for `-p` to reflect this. Differential Revision: https://phab.mercurial-scm.org/D5022

fsmonitor: use vfs instead of opener (issue5938) "opener" of localrepository object was dropped at Mercurial 4.3 (or a7e210167c28). "vfs" should be used instead. wlock is required to write into a file under .hg directory. For efficiency, we should change _cmpsets() from: 1. acquire wlock 2. open log file under .hg directory with write mode 3. compare between result of watchman and Mercurial's dirstate logic 4. write out error info into a file, if error is detected 5. release wlock to: 1. compare between result of watchman and Mercurial's dirstate logic 2. acquire wlock, if error is detected 3. open and write error info into a file 4. release wlock But this is another issue.

tests: configure fsmonitor.mode=paranoid always if fsmonitor is used This forces fsmonitor extension execute "paranoid" code path. Strict speaking, we should make fsmonitor-run-tests.py accept own specific options, but there is no code path, which is disabled in "paranoid" mode, at least now. Therefore, this solution seems reasonable enough.

tests: drop files from exclusion list in test-check-module-imports.t Now, these files has no style issue at checking with import-checker.py. BTW, tests/test-imports-checker.t is still excluded, because almost all code fragments in it has intentional importing style violation. Using NO_CHECK_EOF instead of EOF as heredoc limit mark can make import-checker.py ignore such fragments. But keeping these fragments checkable seems still useful: for example, test-imports-checker.t can be used to test whether import-checker.py can detect erroneous code fragment in test script as expected.

tests: use NO_CHECK_EOF for fragments used to test importing via extension test-extension.t is excluded in test-check-module-imports.t, because import-checker.py reports that some python code fragments in it does not import modules in recommended style. These code fragments are used to test importing modules relatively and absolutely via extension (e.g. issue4029, issue5208, and so on). Test specific package hierarchy becomes complicated for this purpose, and it seems reasonable to avoid checking these code fragments with import-checker.py. But on the other hand, other code fragments in test-extension.t should be checked by import-checker.py. Therefore, this patch uses NO_CHECK_EOF limit mark only for python code fragments, which are used to test importing via extension in test-extension.t. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: use NO_CHECK_EOF for fragments having intentional error Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that code fragments embedded in these test scripts have syntax error. But these syntax error is intentional. This patch uses NO_CHECK_EOF instead of EOF as heredoc limit mark for such fragments, in order to make import-checker.py ignore them. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: fix style issue of importing order in test-lock.py test-lock.py is excluded in test-check-module-imports.t, because import-checker.py reports that some stdlibs are imported after local silenttestrunner module. I can not found out any reason why tests/silenttestrunner.py should be imported before some stdlibs from the point of functionality view.

tests: fix style issue of importing hgweb in embedded code fragments Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that hgweb and/or hgwebdir of mercurial.hgweb are not imported in recommended style. To fix this issues, this patch make python code fragments embedded in these files import hgweb from mercurial package at first, and refer hgweb and hgwebdir via imported hgweb.

py3: encode str to bytes These fields are str on Python 2 and 3. This module doesn't import any Mercurial modules. So I just did the str -> bytes inline. Differential Revision: https://phab.mercurial-scm.org/D5010

py3: sprinkle statprof.py with utf-8 encoding Trying to chase the long tail of unicode badness in this file due to frame / code objects holding str everywhere. Differential Revision: https://phab.mercurial-scm.org/D5016

py3: flush stdout This avoids buffering due to mixed print() and sys.stdout.write() by profiling code. And with this, test-profile.t passes on Python 3 \o/. Honestly, I'd be shocked if there weren't more Python 3 encoding issues lingering in the profiling code. That code isn't super high quality and doesn't seem to have thorough test coverage. Differential Revision: https://phab.mercurial-scm.org/D5018

py3: use raw strings in statprof.py This allows main() and argument parsing to work. Differential Revision: https://phab.mercurial-scm.org/D5017

py3: switch from print(..., file=) to write() Because Python 3's print() doesn't like bytes. Differential Revision: https://phab.mercurial-scm.org/D5015

py3: use %d in a few places Differential Revision: https://phab.mercurial-scm.org/D5014

py3: use raw strings for stack names These are used for simple compares. We don't need to normalize to bytes. As part of this, I expanded a set literal so entries are 1 per line. Differential Revision: https://phab.mercurial-scm.org/D5013

py3: use sysbytes for converting code attributes Differential Revision: https://phab.mercurial-scm.org/D5012

py3: use write() instead of print() Because print() expects str and we want to write bytes. There should be no functional changes as part of this. Differential Revision: https://phab.mercurial-scm.org/D5011

py3: use %d to format ints Differential Revision: https://phab.mercurial-scm.org/D5009

py3: convert sorting field to sysstr This is used as part of an attribute name check and needs to be native str. Differential Revision: https://phab.mercurial-scm.org/D5008

py3: encode json output to bytes and use write() Differential Revision: https://phab.mercurial-scm.org/D5007

absorb: prompt user to accept absorb changes by default Change the default absorb behaviour to print the changes out and then prompt the user if they want to absorb the changes or not. Providing the `-p` option still prints the result and exits. A new `-a` option is provided which applies the changes without printing them, giving the equivalent of the old behaviour. Differential Revision: https://phab.mercurial-scm.org/D4999

templatefuncs: add truncate parameter to pad Add a truncate option to pad that additionally truncates the text to the pad width if it is wider. Since color codes can cause a problem with this, when the text is truncated, the color codes are also stripped. Users of the truncate option should label the text outside the pad. Differential Revision: https://phab.mercurial-scm.org/D5000

absorb: print summary of changesets affected Differential Revision: https://phab.mercurial-scm.org/D4998

absorb: use a formatter to generate output Change absorb to use a formatter to generate its output. This allows the use of templates to customize the output. Differential Revision: https://phab.mercurial-scm.org/D4997

tests: fix up test-bad-extension.t's inline extension Upcoming patches will be simpler, I think. Differential Revision: https://phab.mercurial-scm.org/D5003

polib: update to latest release 1.0.7 (upstream rev d75ce6dbbc2a) # no-check-commit third-party code that doesn't match our style Differential Revision: https://phab.mercurial-scm.org/D5001

tests: glob away some annoying py3 differences Differential Revision: https://phab.mercurial-scm.org/D4994

tests: fix a repr on python3 in test-extension.t Differential Revision: https://phab.mercurial-scm.org/D4993

tests: so many b prefixes in test-extension.t # skip-blame because it's b prefixes Differential Revision: https://phab.mercurial-scm.org/D4992

tests: ensure print() statements in test-extension.t all flush Differential Revision: https://phab.mercurial-scm.org/D4991

revlog: update pure nodecache start lookup offset on insertion test-storage.py is failing in pure builds because the population of the node cache isn't accounting for the new starting offset after a revlog insertion. This commit updates the node cache start offset to account for insertions. I'm not 100% convinced this is the ideal solution. But it works and seems correct. Differential Revision: https://phab.mercurial-scm.org/D4996

exchangev2: use filesdata filesdata is a more efficient mechanism for bulk fetching files data for a range of changesets. Let's use it in exchangev2. With this change, a client performing a full clone of mozilla-unified transmits substantially fewer bytes across the wire: before: 139,124,863 bytes sent after: 20,522,499 bytes sent The bulk of the remaining bytes is likely the transfer of ~1M nodes for changesets and manifests. We can eliminate this by making requests in terms of node ranges instead of explicit node lists... Differential Revision: https://phab.mercurial-scm.org/D4982

wireprotov2: define and implement "filesdata" command Previously, the only way to access file revision data was the "filedata" command. This command is useful to have. But, it only allowed resolving revision data for a single file. This meant that clients needed to send 1 command for each tracked path they were seeking data on. Furthermore, those commands would need to enumerate the exact file nodes they wanted data for. This approach meant that clients were sending a lot of data to remotes in order to request file data. e.g. if there were 1M file revisions, we'd need at least 20,000,000 bytes just to encode file nodes! Many clients on the internet don't have that kind of upload capacity. In order to limit the amount of data that clients must send, we'll need more efficient ways to request repository data. This commit defines and implements a new "filesdata" command. This command allows the retrieval of data for multiple files by specifying changeset revisions and optional file patterns. The command figures out what file revisions are "relevant" and sends them in bulk. The logic around choosing which file revisions to send in the case of haveparents not being set is overly simple and will over-send files. We will need more smarts here eventually. (Specifically, the client will need to tell the server which revisions it knows about.) This work is deferred until a later time. Differential Revision: https://phab.mercurial-scm.org/D4981

wireprotov2: extract file object emission to own function An upcoming commit will introduce another caller. Differential Revision: https://phab.mercurial-scm.org/D4980

wireprotov2: change how revisions are specified to changesetdata Right now, we have a handful of arguments for specifying the revisions whose data should be returned. Defining how all these arguments interact when various combinations are present is difficult. This commit establishes a new, generic mechanism for specifying revisions. Instead of a hodgepodge of arguments defining things, we have a list of dicts that specify revision selectors. The final set of revisions is a union of all these selectors. We implement support for specifying revisions based on: * An explicit list of changeset revisions * An explicit list of changeset revisions plus ancestry depth * A DAG range between changeset roots and heads If you squint hard enough, this problem has already been solved by revsets. But I'm reluctant to expose revsets to the wire protocol because that would require servers to implement a revset parser. Plus there are security and performance implications: the set of revision selectors needs to be narrowly and specifically tailored for what is appropriate to be executing on a server. Perhaps there would be a way for us to express the "parse tree" of a revset query, for example. I'm not sure. We can explore this space another time. For now, the new mechanism should bring sufficient flexibility while remaining relatively simple. The selector "types" are prefixed with "changeset" because I plan to add manifest and file-flavored selectors as well. This will enable us to e.g. select file revisions based on a range of changeset revisions. Differential Revision: https://phab.mercurial-scm.org/D4979

wireprotov2: stop sending phase updates for base revisions This feature is broken and doesn't work properly in all scenarios. e.g. if we have the following DAGs: client server D draft C draft C draft B draft B public A public A public The current code would only send the phase data for C. The client wouldn't see that B moved from draft to public. This feature will be restored in a future commit. For now, it is making refactoring of how revisions are specified in the wire protocol a bit difficult... Differential Revision: https://phab.mercurial-scm.org/D4978

debugcommands: support wrapping long lines If a line within a block is indented more than the line that came before, we automatically concatenate it with the previous line. This allows us to pretty format data. This will make tests easier to read. At some point we may just want to evaluate entire blocks as Python code or something, as even with this change, things aren't perfect, as we can't e.g. have formatting like: foo eval:[ True ] But this is strictly better than before, where we couldn't wrap long lines. Differential Revision: https://phab.mercurial-scm.org/D4977

exchangev2: honor server advertised manifestdata recommended batch size Let's plug the client up to the server-advertised recommended batch size for manifestdata requests. Differential Revision: https://phab.mercurial-scm.org/D4976

wireprotov2: advertise recommended batch size for requests Currently, exchangev2 hardcodes the batch size for how many revisions to fetch per command request. A single value is not appropriate for every repository because some repositories may have a drastically different "shape" from other repositories. e.g. a repo with lots of small files may benefit from larger batch sizes than a repo with lots of large files. And depending on caching used by the server, the server may wish to control the number of commands (to e.g. mitigate overhead of following content redirects). This commit teaches wireprotov2 commands to declare extra metadata which is advertised as part of the command descriptor. The manifestdata command has been taught to advertise a recommended batch size for requests. Differential Revision: https://phab.mercurial-scm.org/D4975

httppeer: expose API descriptor on httpv2peer The API descriptor in wireprotov2 is much more expressive than space-delimited tokens and it will be difficult to define methods to query it in all of the ways we'll want to query it. So let's just declare defeat and expose the API descriptor on the peer instance. As part of this, we define a new interface for version 2 peers, fulfilling a TODO in the process. Differential Revision: https://phab.mercurial-scm.org/D4974

tests: use baseurl instead of advertisedbaseurl The distinction matters for e.g. hosts behind load balancers. But for the test environment, it doesn't matter. For whatever reason, advertisedbaseurl is resolving to http://1.0.0.127.in-addr.arpa:$HGPORT on my MBP. This hostname fails to resolve, causing the test to fail. No clue what's up with that behavior. Differential Revision: https://phab.mercurial-scm.org/D4973

py3: another one started passing Differential Revision: https://phab.mercurial-scm.org/D4990

py3: one more passing test Differential Revision: https://phab.mercurial-scm.org/D4989

scmutil: wrap locker information in bytestr before repr()ing it Differential Revision: https://phab.mercurial-scm.org/D4988

py3: more passing tests Differential Revision: https://phab.mercurial-scm.org/D4987

py3: whitelist another passing test Differential Revision: https://phab.mercurial-scm.org/D4986

obsolete: convert error string to a sysstr This should be okay because we're just %-formatting a hash into a localized string. Differential Revision: https://phab.mercurial-scm.org/D4985

tests: ensure the python 3 passing test list stays sorted Differential Revision: https://phab.mercurial-scm.org/D4984

py3: sort passing test list Differential Revision: https://phab.mercurial-scm.org/D4983

py3: tweak stdout writing in test-hgweb-no-path-info.t We want to write bytes for convenience. This requires sys.stdout.buffer. But using sys.stdout.buffer introducing buffered output. So we sprinkle code with sys.stdout.flush() to force immediate writes. After all that, Python 3 was emitting b'' prefixed output for errors. So we only print errors if there were some. There aren't, so b'' don't come into play and output is identical in Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D4972

py3: use .+ instead of .* in regexp pattern Python 3.7 changed the behavior of re.sub(). See https://bugs.python.org/issue33585. The new code should work on old and new Pythons. Differential Revision: https://phab.mercurial-scm.org/D4971

py3: coerce bytestr to bytes to appease urllib.parse.quote_from_bytes() Differential Revision: https://phab.mercurial-scm.org/D4969

py3: pass str and return bytes from mimetypes.guess_type() This function wants a str (which represents a path) and returns a str. We normalize input to str and output to bytes. Differential Revision: https://phab.mercurial-scm.org/D4967

py3: use bytes literal in test-hgweb-json.t # skip-blame just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4970

py3: convert diff opcode name to bytes Differential Revision: https://phab.mercurial-scm.org/D4968

py3: byteify hgweberror.py # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4966

py3: encode JSON str to bytes json.dumps() will emit UTF-8 str on Python 2 and 3. Use sysbytes to force the .encode('utf-8') on Python 3 and no-op on Python 2. Differential Revision: https://phab.mercurial-scm.org/D4965

py3: use b'' in test-check-interfaces.py # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4964

revlog: optimize ancestors() to not check filtered revisions for each While reviewing the Rust implementation, I noticed iter(ancestors) doesn't need to check filtering state for each parent revision. And doing that appears to have some measurable perf win. $ hg perfancestors -R mercurial (orig) wall 0.038093 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) (this) wall 0.024795 comb 0.020000 user 0.020000 sys 0.000000 (best of 117)

phabricator: drop support for the legacy phabricator.auth.token config (BC) The test for this broke in dc82ad1b7f77 when statistics started being tracked. It wasn't noticed because none of the bots have the vcr module installed. It looks like the custom_patches argument should patch in the custom httpconnection, and I can't figure out what is going on.

absorb: print '{rev}:' as a prefix to the hash I don't see a sane way to print the summary to identify the changed commit, but this at least makes it a bit easier to identify a commit when a group of them changes.

packaging: fix "make centos{5,6,7}" Without this change, make centos{5,6,7} fails with error: cp: cannot stat '<basedir>/rpmbuild/RPMS/*/*': No such file or directory This change is the exact equivalent of bc4bbc42899a (which applied to Fedora).

narrow: don't compress the bundle2 when sending 'error:abort' This is similar to waht getbundle() does and also explicitly specifies that we should get a compressed bundle2 in normal cases when not sending 'error:abort'. Differential Revision: https://phab.mercurial-scm.org/D4934

push: add "remote" to 'repository changed while pushing' messages (issue5971) Differential Revision: https://phab.mercurial-scm.org/D4933

fuzz: try setting PYTHONNOUSERSITE=1 to avoid loading site-packages Right now the fuzzer is crashing trying to look up the home dir for uid 0, which is breaking in the fuzz environment. Differential Revision: https://phab.mercurial-scm.org/D4936

perf: extract result formatting in its own function This will make it easier to reuse in another changesets.

perf: extract the timing of a section in a context manager This makes it easier to reuse it in other (future) part of the code that requires their own time management.

pref: support negative indexing in perfrevlogrevisions This is useful to check for a consistent final slice in multiple different repositories.

perf: accept formatter option for perfmanifest

perf: fix -T json The previous code was mixing formatting and data, breaking `-T json` with unexpected data. We fix the issue and add a test to prevent future regression.

formatter: more details on assertion failure This is useful when the assertion fails.

wireprotov2: raise ProgrammingError on unknown action Suggested by @durin42 in review of D4923. Differential Revision: https://phab.mercurial-scm.org/D4935

wireprotov2: send content encoded frames from server Now that we have support for negotiating encodings and configuring an encoder, we can start sending content encoded frames from the server. This commit teaches the wireprotov2 server code to send content encoded frames. On the mozilla-unified repository with zstd enabled peers, this change reduces the total amount of data transferred from server to client drastically: befor: 7,190,995,812 bytes after: 1,605,508,691 bytes Differential Revision: https://phab.mercurial-scm.org/D4927

wireprotov2: raise exception in objects() if future has been resolved Differential Revision: https://phab.mercurial-scm.org/D4926

wireprotov2: don't emit empty frames Staring at logs revealed the presence of empty frames that should have contained payload. Let's stop that from happening. Differential Revision: https://phab.mercurial-scm.org/D4925

wireprotov2: remove functions for creating response frames from bytes All code in the actual server uses oncommandresponsereadyobjects(). Test code was ported to that method. This resulted in a handful of subtle test changes. Differential Revision: https://phab.mercurial-scm.org/D4924

wireprotov2: handle noop action This action can be returned from the client reactor. We should handle it. Differential Revision: https://phab.mercurial-scm.org/D4923

wireprotov2: send protocol settings frame from client Now that we have client and server reactor support for protocol settings and encoding frames, we can start to send them out over the wire! This commit teaches the client reactor to send out a protocol settings frame when needed. The httpv2 peer has been taught to gather a list of supported content encoders and to advertise them through the client reactor. Because the client is now sending new frame types by default, this constitutes a compatibility break in the framing protocol. The media type version has been bumped accordingly. This will ensure existing clients won't attempt to send the new frames to old servers not supporting this explicit media type. I'm not bothering with the BC annotation because everything wireprotov2 is highly experimental and nobody should be running a server yet. Differential Revision: https://phab.mercurial-scm.org/D4922

wireprotov2: define and use stream encoders Now that we have basic support for defining stream encoding, it is time to start doing something with it. We define various classes implementing stream encoders/decoders for the defined encoding profiles. This is relatively straightforward. We teach the inputstream and outputstream classes how to encode, decode, and flush data. We then teach the clientreactor how to filter received data through the inputstream decoder. One of the features of the framing format is that streams can span requests. This is a differentiating feature from say HTTP/2, which associates streams with requests. By allowing streams to span requests, we can reuse compression context data across requests/responses. But in order to do this, we need a mechanism to "flush" the encoder at logical boundaries so that receivers receive all data where it is expected. And a "flush" event is distinct from a "finish" event from the perspective of certain compressors because a "flush" will retain compression context state whereas a "finish" operation will not. This is why encoders have both a flush() and a finish() and each uses specific flushing semantics on the underlying compressor. The added tests verify various behavior of decoders via clientreactor. These tests do test some compression behavior via use of outputstream. But for all intents and purposes, server reactor support for encoding is not yet implemented. Differential Revision: https://phab.mercurial-scm.org/D4921

lfs: avoid a potential variable reference before assignment error in cmdserver A coworker hit this once yesterday when pulling in thg (a retry worked), and then I hit it with strip after a pull. I had a difficult time recreating a test for this (at least one of the tricks was to not use '-R', which seems to cause reposetup() to be called for each command), so I'm not sure how large of a window there actually is for this. Calling reposetup() *after* the requirement is added will skip the hook entirely. The other issue I had was adding a couple `ui.status()` lines around the check that installs the hook. On Windows, the cmdserver process ballooned to 1.6GB and hung. Changing that to `ui.warn()` avoided the hang. It also hung on macOS, but without the large memory usage.

wireprotov2: establish dedicated classes for input and output streams Streams are unidirectional. As part of implementing encoding/decoding support, it became clear that it didn't make sense for a generic "stream" class to hold functionality related to both encoding and decoding. So we create new classes to represent the flavor of stream. Differential Revision: https://phab.mercurial-scm.org/D4920

wireprotov2: pass ui into clientreactor and serverreactor This will allow us to use config options to influence compression settings. Differential Revision: https://phab.mercurial-scm.org/D4919

wireprotov2: handle stream encoding settings frames Like what we just did for the server reactor, we teach the client reactor to handle stream encoding settings frames. The code is very similar. We define a method on the stream class to handle processing the data within the decoded frames. However, it doesn't yet do anything useful. Differential Revision: https://phab.mercurial-scm.org/D4918

wireprotov2: document client reactor actions We should document these so consumers have an easier life. Differential Revision: https://phab.mercurial-scm.org/D4917

wireprotov2: handle sender protocol settings frames We teach the server reactor to handle the optional sender protocol settings frames, which can only be sent at the beginning of frame exchange. Right now, we simply decode the data and record the sender protocol settings on the server reactor instance: we don't yet do anything meaningful with the data. Differential Revision: https://phab.mercurial-scm.org/D4916

wireprotov2: update stream encoding specification The encoding of data within streams in the frame-based protocol is not yet defined or implemented. This means that all data in wire protocol version 2 is currently being sent out raw, without compression. That's obviously not ideal. This commit formalizes the beginnings of stream encoding support in the protocol. I suspect we'll change behavior substantially in the future. My goal is to get something landed so we can use compression. We can build out more robust support later. Because the frame type ID changed, this is strictly BC. But existing code wasn't using the frame. I'll bump the framing protocol version later once code is introduced to use the new frame. Differential Revision: https://phab.mercurial-scm.org/D4915

cborutil: cast bytearray to bytes This code didn't like passing in bytearray instances. Let's cast bytearray to bytes so it works. Differential Revision: https://phab.mercurial-scm.org/D4914

tests: disable zstd in test This makes the test pass in pure installs. Differential Revision: https://phab.mercurial-scm.org/D4913

wireprotov2: remove "compression" from capabilities response This is not used. And future commits will change how this mechanism works. Let's remove it. As a bonus, this fixes some test failures on pure installs (due to zstd references). Differential Revision: https://phab.mercurial-scm.org/D4912

zstandard: vendor python-zstandard 0.10.1 This was just released. The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. setup.py was updated to pass a new argument to python-zstandard's function for returning an Extension instance. Upstream had to change to use relative paths because Python 3.7's packaging doesn't seem to like absolute paths when defining sources, includes, etc. The default relative path calculation is relative to setup_zstd.py which is different from the directory of Mercurial's setup.py. The project contains a vendored copy of zstandard 1.3.6. The old version was 1.3.4. The API should be backwards compatible and nothing in core should need adjusted. However, there is a new "chunker" API that we may find useful in places where we want to emit compressed chunks of a fixed size. There are a pair of bug fixes in 0.10.0 with regards to compressobj() and decompressobj() when block flushing is used. I actually found these bugs when introducing these APIs in Mercurial! But existing Mercurial code is not affected because we don't perform block flushing. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D4911

rust-chg: install signal handlers to forward signals to server I use sync::Once as a synchronization primitive because it's quite easy to use, and is good enough to prevent data race in these C functions.

rust-chg: remove SIGCHLD handler which won't work in oxidized chg Since pager is managed by the Rust part, the C code doesn't know the pager pid. I could make the Rust part teach the pid to C, but still installing SIGCHLD handler seems horrible idea since we no longer use handcrafted low-level process management functions. Instead, I'm thinking of adding async handler to send SIGPIPE at the exit of the pager.

rust-chg: extract signal handlers from chg/procutil.c abortmsgerrno() and debugmsg() are removed, and the public interface instead returns success/error status. Since signal handlers can't propagate errors, the result of kill() is just ignored.

help: document about "version" template keywords

help: document about "tags" template keywords

help: document about "status" template keywords

help: document about "resolve" template keywords

help: document about "paths" template keywords

help: document about "identify" template keywords

help: document about "grep" template keywords

chgserver: catch Abort while parsing early args to shut down cleanly _loadnewui() calls dispatcher functions, which may raise Abort if unparsable arguments are passed in. The server should catch such errors and translate them to the "exit 255" instruction so the client can finish the IPC session cleanly. Spotted while porting the chg client to Rust.

chg: upgrade client to use "setumask2" command No compatibility code is added to the client side, since it's unlikely for new client to communicate with the old server.

chgserver: add "setumask2" command which uses correct message frame The first 4 bytes should be a length field, not a value. Spotted while porting chg functions to the Rust one.

packaging: "make deb" no longer fails Release 4.7 rationalized the layout of the build scripts. Unfortunately, while "make docker-ubuntu-*" and "make docker-debian-*" worked as expected, "make deb" was broken. Before this change "make deb" was failing with the following error: You are not inside a Mercurial repository! Or, after the latest changes: You are inside <fullpath>, which is not the root of a Mercurial repository Moreover, when "make deb" failed, the cleanup routine deleted the wrong directory (contrib/packaging/debian instead of <reporoot>/debian) resulting in a corrupted working copy that needed to be hg revert-ed. After this change the docker targets continue to work, and the deb one is able to finish.

packaging: cleanup() did not read the value of $CLEANUP When the original author put CLEANUP in a conditional statement he was probably willing to use it to control the "if". This change tries to restore that behaviour: the "rm" clause is triggered if and only if CLEANUP is defined and not empty.

packaging: builddeb's cleanup needs to expand PWD, safely Single quotes would not expand the variable.

packaging: blindly factor out trap's cleanup function in builddeb This commit blindly extracts builddeb's trap routine in a dedicated function. While doing so, I think two bugs are exposed, which will be addressed in the next commits: - single quoting around '$CLEANUP' will always evaluate to the literal '$CLEANUP' regardless of the variable's value. The "if" will always be true. - the removal operation will not expand $PWD (and a variable expansion would need double quotes, anyways.

packaging: print full path to the packages when builddeb finishes successfully

packaging: print more specific error messages when builddeb fails

cmdutil: sort unresolved paths I noticed that `hg status` was printing unresolved paths in a non-deterministic order. This patch fixes that. I'm not sure if the sorting should be done in merge.mergestate.unresolved() instead. Either way fixes the presentation issue. Differential Revision: https://phab.mercurial-scm.org/D4929

fuzz: report error if Python code raised exception I think that's what we wanted to do, given the most of the code block is surrounded by try-except. 'lazymanifest(mdata)' is moved to the try block as it can fail.

revlog: explicitly initialize static variables I know .bss section is zero-filled, but explicit initialization should be better as we rely on that.

tests: do not change sys.path, it can break loading cext.parsers When running this tests with run-tests, the prefix would resolve mercurial.cext to the source tree and the attempt to load mercurial.cext.parsers would therefore fail since it doesn't exist in it. With the regular search path from run-tests, it is picked up from the temporary prefix correctly. Differential Revision: https://phab.mercurial-scm.org/D4910

tests: deal with differences in tic from ncurses and NetBSD Differential Revision: https://phab.mercurial-scm.org/D4909

closehead: fix close-head -r listification Differential Revision: https://phab.mercurial-scm.org/D4908

import-checker: use testparseutil.embedded() to centralize detection logic This patch fixes issues of embedded() in import-checker.py below, too. - overlook (or mis-detect) the end of inline script in doctest style - overlook inline script in doctest style at the end of file (and ignore invalid un-closed heredoc at the end of file, too) - overlook code fragment in styles below - "python <<EOF" (heredoc should be "cat > file <<EOF" style) - "cat > foobar.py << ANYLIMIT" (limit mark should be "EOF") - "cat << EOF > foobar.py" (filename should be placed before limit mark) - "cat >> foobar.py << EOF" (appending is ignored)

tests: use NO_CHECK_EOF as heredoc limit mark to omit checking code fragments This patch uses NO_CHECK_EOF as heredoc limit mark instead of EOF, in order to avoid checking all python code fragments in test-contrib-check-code.t, because almost all of them has un-recommended implementations intentionally.

contrib: add an utility module to parse test scripts This patch centralizes the logic to pick up code fragments embedded in *.t script, in order to: - apply checking with patterns in check-code.py on such embedded code fragments Now, check-code.py completely ignores embedded code fragments. I'll post another patch series to check them. - replace similar code path in contrib/import-checker.py Current import-checker.py has problems below. Fixing each of them is a little difficult, because parsing logic and pattern strings are tightly coupled. - overlook (or mis-detect) the end of inline script in doctest style 8a8dd6e4a97a fixed a part of this issue, but not enough. - it overlooks inline script in doctest style at the end of file (and ignores invalid un-closed heredoc at the end of file, too) - it overlooks code fragment in styles below - "python <<EOF" (heredoc should be "cat > file <<EOF" style) - "cat > foobar.py << ANYLIMIT" (limit mark should be "EOF") - "cat << EOF > foobar.py" (filename should be placed before limit mark) - "cat >> foobar.py << EOF" (appending is ignored) - it is not extensible for other than python code fragments (e.g. shell script, hgrc file, and so on) This new module can detect python code fragments in styles below: - inline script in doctest style (starting by " >>> " line) - python invocation with heredoc script ("python <<EOF") - python script in heredoc style (redirected into ".py" file) As an example of extensibility of new module, this patch also contains implementation to pick up code fragment below. This will be useful to add additional restriction for them, for example. - shell script in heredoc style (redirected into ".sh" file) - hgrc configuration in heredoc style (redirected into hgrc or $HGRCPATH)

tests: use environment variable indirectly Using environment variable directly in heredoc python code will cause syntax error at checking module importation by import-checker.py strictly, because "$varname" is invalid in Python syntax. "$varname" becomes valid after environment variable substitution by shell at writing text into file. Current import-checker.py overlooks code fragment changed in this patch, because of a restriction below for a line starting code fragment. - filename must be specified before limit mark NG: cat <<EOF > FILE.py OK: cat > FILE.py <<EOF import-checker.py itself is fixed in subsequent patch.

tests: import multiple modules separately Current import-checker.py overlooks code fragment changed in this patch, because of restrictions below for a line starting code fragment. - filename must be specified before limit mark NG: cat <<EOF > FILE.py OK: cat > FILE.py <<EOF - limit mark must not be quoted NG: cat > FILE.py <<'EOF' OK: cat > FILE.py <<EOF import-checker.py itself is fixed in subsequent patch.

fuzz: allow manifest fuzzer to detect leaks Huzzah! Differential Revision: https://phab.mercurial-scm.org/D4907

fuzzers: init Python in LLVMFuzzerInitialize and intentionally leak it This sidesteps leaks (or "leaks", I'm not sure) in CPython, and lets our fuzzer work. Differential Revision: https://phab.mercurial-scm.org/D4906

revlog: if the module is initialized more than once, don't leak nullentry Caught (annoyingly) by the manifest fuzzer. Differential Revision: https://phab.mercurial-scm.org/D4905

narrow: move remaining narrow-limited dirstate walks to core In most places we now filter at a higher level (the context object), but there are few places that relied on the dirstate walk to be filtered by the narrowspec. The important cases are those used by `hg add` and `hg addremove`. This patch updates them to pass in a matcher instead of relying on the dirstate to do the filtering. The dirstate filtering is also dropped in narrowdirstate.py. Not always filtering in the dirstate should be useful for a future `hg status --include-outside-narrow` option. These places now end up doing an unrestricted dirstate walk after this patch: * debugfileset * perfwalk * sparse (but restricted to sparse config) * largefiles I'll let anyone who cares about these cases adapt them to work with narrow if necessary. Differential Revision: https://phab.mercurial-scm.org/D4901

narrow: allow repo.narrowmatch(match) to include exact matches from "match" Differential Revision: https://phab.mercurial-scm.org/D4900

narrow: filter files by narrowspec in ctx.matches() This has no effect yet because 1) for committed changes, ctx.matches() just calls ctx.walk(), which we updated in the previous patch, and 2) for the working copy, the filtering is also done in the overridden dirstate.walk() in narrowdirstate. Differential Revision: https://phab.mercurial-scm.org/D4899

narrow: only walk files within narrowspec also for committed revisions Narrow has been walking only paths matching the narrowspec when walking the working copy. We have not done the same filtering when walking committed revisions (e.g. "hg files -r "), which seems a little odd. Let's make it consistent. Differential Revision: https://phab.mercurial-scm.org/D4898

status: intersect matcher with narrow matcher instead of filtering afterwards I seem to have done a very naive move of the code from the narrow extension into core in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). It seems obvious that a better way is to intersect the matchers. Note that this means that when requesting status for the working directory in a narrow repo, we now pass the narrow matcher (possibly intersected with a user-provided matcher) into _buildstatus() and then into dirstate.status() and dirstate.walk(), which will the intersect it again with the narrow matcher. That's functionally fine, but wasteful. I hope to later remove the dirstate wrapping that adds the second layer of matcher intersection. Differential Revision: https://phab.mercurial-scm.org/D4897

localrepo: allow narrowmatch() to accept matcher to intersect with It's pretty common that we need to intersect a matcher we already have (usually from the user) with the narrow matcher. Let's make repo.narrowmatch() take an optional matcher to intersect with. Differential Revision: https://phab.mercurial-scm.org/D4896

obsolete: fix ValueError when stored note contains ':' char (issue5783) The newer version of `amend -n 'Some some'` accepts containing ':' char. The information contained in this note 'Testing::Obstore' gives ValueError, because we are trying to store more than 2 values in key and value. Differential Revision: https://phab.mercurial-scm.org/D4883 Differential Revision: https://phab.mercurial-scm.org/D4882

narrow: update TODO.rst now that we share format with sparse The narrowspec format was unified with the sparse format in f64ebe7d2259 (narrowspec: use sparse.parseconfig() to parse narrowspec file (BC), 2018-08-03). Differential Revision: https://phab.mercurial-scm.org/D4904

narrow: update TODO.rst now that we filter status in ctx The comment referred to was addressed in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). I also think 84092edd5c88 (narrow: drop unnecessary overrides of patch, 2018-09-28) suggests that it was the right thing to do. Differential Revision: https://phab.mercurial-scm.org/D4903

narrow: update TODO.rst now that the narrowspec is in .hg/store We no longer have the unfortunate wrappostshare() and unsharenarrowspec() since 576eef1ab43d (narrow: move .hg/narrowspec to .hg/store/narrowspec (BC), 2018-08-02). Differential Revision: https://phab.mercurial-scm.org/D4902

py3: add 8 new passing tests to whitelist found by buildbot We are getting close! Differential Revision: https://phab.mercurial-scm.org/D4893

py3: use '%f' for floats instead of '%s' I remember Yuya saying we need to use bytestr() or '%r' because '%s' was clever. Not sure it applies to this or not. Differential Revision: https://phab.mercurial-scm.org/D4894

narrow: move adding of narrow server capabilities to core We use the experimental.narrow config option introduced in one of the previous patch and move the functionality of adding narrow server capabilities to core. Differential Revision: https://phab.mercurial-scm.org/D4891

wireprotoserver: move narrow capabilities to wireprototypes.py This is done because wireprotoserver import wireprotov1server, so you cannot import wireprotoserver in wireprotov1server to use the capabilities constants. Differential Revision: https://phab.mercurial-scm.org/D4890

narrow: introduce a config option to check if narrow is enabled or not This patch introduces a new config option experimental.narrow which is set to False by default and set to True by the narrow extension. While moving narrow related logic into core, we need to know at places whether narrow extension is enabled or not. Checking the list of extension enabled is one solution but once narrow is inbuilt, we will definitely want a config option to check whether narrow is turned on or not. So this patch introduces a config option, which will evolve to the main point to turn narrow capability on and off once all the narrow is in core. Differential Revision: https://phab.mercurial-scm.org/D4889

narrow: move the code to generate a widening bundle2 to core This is a part of moving more narrow related bits to core. Differential Revision: https://phab.mercurial-scm.org/D4888

narrow: start returning bundle2 from widen_bundle() Differential Revision: https://phab.mercurial-scm.org/D4838

narrow: the first version of narrow_widen wireprotocol command This patch introduces a wireprotocol command narrow_widen() which will be used to widen a narrow copy using `hg tracked` command provided by narrow extension. The wireprotocol command takes the old and new includes and excludes, common heads, changegroup version, known revs, and a boolean ellipses and generates a bundle2 of the required data and send it. The clients receives the bundle2 and applies that. A bundle2 instead of changegroup because in future we might want to add more things to send while widening. Thanks for martinvonz for the suggestion. I am not sure whether we need changegroup version as an argument to the command as I *think* narrow needs changegroup3 already. The tests shows that we don't exchange phase data now while widening which is nice. Also we don't check for pushkeys, rbc-cache, bookmarks etc. This does not support ellipses cases for now but will be supported in future patches. Since we send bundle2, it won't be hard to plug the ellipses logic in here. The existing code for widening a non-ellipses case is also dropped in this patch. Differential Revision: https://phab.mercurial-scm.org/D4813

remotenames: abort if literal revset pattern matches nothing This is the convention of the other namespace revsets such as tag(). Let's make the remote variants do the same.

remotenames: remove unneeded sorted() from revset implementation The order is constrained by the subset.

remotenames: don't call a set of nodes as "revs"

remotenames: use util.always instead of handcrafted lambda

remotenames: inline _parseargs() into _revsetutil() The _parseargs() function gets quite simple, and the 0/1 loop can be rewritten as "if".

repo: create changectx in a single place in localrepo.__getitem__ Differential Revision: https://phab.mercurial-scm.org/D4885

repo: remove the last few "pass" statements in localrepo.__getitem__ In case of IndexError or LookupError, we used "pass" statements and fell through to the end of localrepo.__getitem__. I find the pass statements easy to miss. Consistently raising and catching exceptions seems easier to follow. Oh -- and I didn't plan this before I wrote the above -- that probably also lets us reuse the "return context.changectx(self, rev, node)" in a later patch. Differential Revision: https://phab.mercurial-scm.org/D4884

filectx: correct docstring about "changeid" The changeid argument must be a revnum (basefile.rev() is defined as "return self._changeid"), so fix the lie in the docstring. It seems to have been incorrect for at least 10 years (I didn't check further back). Differential Revision: https://phab.mercurial-scm.org/D4881

context: drop incorrect and superfluous docstring It's been incorrect at least since 8b86acc7aa64 (context: drop support for looking up context by ambiguous changeid (API), 2018-04-28). Differential Revision: https://phab.mercurial-scm.org/D4880

remotenames: follow-up on D3639 to make revset funcs take only one arg Per the review discussion on D3639, we want this to just take one argument. That ended up simplifying the code, so I'm sharing this as a follow-up to that revision rather than editing in-flight.

remotenames: add names argument to remotenames revset This patch adds names argument to the revsets provided by the remotenames extension. The revsets are remotenames(), remotebranches() and remotebookmarks(). names can be a single names, list of names or can be empty too which means it's an optional argument. If names is/are passed, changesets which have those remotenames will be returned. If names are not passed, changesets from all the remotenames are shown. Passing an invalid remotename does not throw error. The name argument also supports pattern matching. Tests are added for the argument in tests/test-logexchange.t Differential Revision: https://phab.mercurial-scm.org/D3639

copies: add time information to the debug information

copies: add a devel debug mode to trace what copy tracing does Mercurial can spend a lot of time finding renames between two commits. Having more information about that process help to understand what makes it slow in an individual instance. (eg: many files vs 1 file, etc...)

revlog: rewrite censoring logic I was able to corrupt a revlog relatively easily with the existing censoring code. The underlying problem is that the existing code doesn't fully take delta chains into account. When copying revisions that occur after the censored revision, the delta base can refer to a censored revision. Then at read time, things blow up due to the revision data not being a compressed delta. This commit rewrites the revlog censoring code to take a higher-level approach. We now create a new revlog instance pointing at temp files. We iterate through each revision in the source revlog and insert those revisions into the new revlog, replacing the censored revision's data along the way. The new implementation isn't as efficient as the old one. This is because it will fully engage delta computation on insertion. But I don't think it matters. The new implementation is a bit hacky because it attempts to reload the revlog instance with a new revlog index/data file. This is fragile. But this is needed because the index (which could be backed by C) would have a cached copy of the old, possibly changed data and that could lead to problems accessing index or revision data later. One benefit of the new approach is that we integrate with the transaction. The old revlog is backed up and if the transaction is rolled back, the original revlog is restored. As part of this, we had to teach the transaction about the store vfs. I'm not super keen about this. But this was the easiest way to hook things up to the transaction. We /could/ just ignore the transaction like we were doing before. But any file mutation should be governed by transaction semantics, including undo during rollback. Differential Revision: https://phab.mercurial-scm.org/D4869

revlog: move loading of index data into own method This will allow us to "reload" a revlog instance from a rewritten index file, which will be used in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D4868

revlog: clear revision cache on hash verification failure The revision cache is populated after raw revision fulltext is retrieved but before hash verification. If hash verification fails, the revision cache will be populated and subsequent operations to retrieve the invalid fulltext may return the cached fulltext instead of raising. This commit changes hash verification so it will invalidate the revision cache if the cached node fails hash verification. The side-effect is that subsequent operations to request the revision text - even the raw revision text - will always fail. The new behavior is consistent and is definitely less wrong. There is an open question of whether revision(raw=True) should validate hashes. But I'm going to punt on this problem. We can always change behavior later. And to be honest, I'm not sure we should expose raw=True on the storage interface at all. Another day... Differential Revision: https://phab.mercurial-scm.org/D4867

fuzz: new fuzzer for cext/manifest.c This is a bit messy, because lazymanifest is tightly coupled to the cpython API for performance reasons. As a result, we have to build a whole Python without pymalloc (so ASAN can help us out) and link against that. Then we have to use an embedded Python interpreter. We could manually drive the lazymanifest in C from that point, but experimentally just using PyEval_EvalCode isn't really any slower so we may as well do that and write the innermost guts of the fuzzer in Python. Leak detection is currently disabled for this fuzzer because there are a few global-lifetime things in our extensions that we more or less intentionally leak and I didn't want to take the detour to work around that for now. This should not be pushed to our repo until https://github.com/google/oss-fuzz/pull/1853 is merged, as this depends on having the Python tarball around. Differential Revision: https://phab.mercurial-scm.org/D4879

revlog: rename _cache to _revisioncache "cache" is generic and revlog instances have multiple caches. Let's be descriptive about what this is a cache for. Differential Revision: https://phab.mercurial-scm.org/D4866

testing: add file storage integration for bad hashes and censoring In order to implement these tests, we need a backdoor to write data into storage backends while bypassing normal checks. We invent a callable to do that. As part of writing the tests, I found a bug with censorrevision() pretty quickly! After calling censorrevision(), attempting to access revision data for an affected node raises a cryptic error related to malformed compression. This appears to be due to the revlog not adjusting delta chains as part of censoring. I also found a bug with regards to hash verification and revision fulltext caching. Essentially, we cache the fulltext before hash verification. If we look up the fulltext after a failed hash verification, we don't get a hash verification exception. Furthermore, the behavior of revision(raw=True) can be inconsistent depending on the order of operations. I'll be fixing both these bugs in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D4865

testing: add file storage tests for getstrippoint() and strip() Differential Revision: https://phab.mercurial-scm.org/D4864

wireprotov2: always advertise raw repo requirements I'm pretty sure my original thinking behind making it conditional on stream clone support was that the behavior mirrored wire protocol version 1. I don't see a compelling reason for us to not advertise the server's storage requirements. The proper way to advertise stream clone support in wireprotov2 would be to not advertise the command(s) required to perform stream clone or to advertise a separate capability denoting stream clone support. Stream clone isn't yet implemented on wireprotov2, so we can cross this bridge later. Differential Revision: https://phab.mercurial-scm.org/D4863

tests: don't be as verbose in wireprotov2 tests I don't think that printing low-level I/O and frames is beneficial to testing command-level functionality. Protocol-level testing, yes. But command-level functionality shouldn't care about low-level details in most cases. This output makes tests more verbose and harder to read. It also makes them harder to maintain, as you need to glob over various dynamic width fields. Let's remove these low-level details from many of the wireprotov2 tests. Differential Revision: https://phab.mercurial-scm.org/D4861

repository: define and use revision flag constants Revlogs have a per-revision 2 byte field holding integer flags that define how revision data should be interpreted. For historical reasons, these integer values are sent verbatim on the wire protocol as part of changegroup data. From a semantic standpoint, the flags that go out over the wire are different from the flags stored internally by revlogs. Failure to establish this semantic distinction creates unwanted strong coupling between revlog's internals and the wire protocol. This commit establishes new constants on the repository module that define the revision flags used by the wire protocol (and by some internal storage APIs, sadly). The changegroups internals documentation has been updated to document them explicitly. Various references throughout the repo now use the repository constants instead of the revlog constants. This is done to make it clear that we're operating on generic revision data and this isn't tied to revlogs. Differential Revision: https://phab.mercurial-scm.org/D4860

context: reverse conditional branch order in introrev Positive logic will be simpler to follow. It will help to clarify coming refactoring.

context: drop a redundant fast path in introrev Now that _adjustlinkrev fast path this case itself, we no longer need an extra conditional. A nice side effect is that we are no longer calling `self.rev()`. In case where `_descendantrev` is set, calling `self.rev` will trigger a potentially expensive `_adjustlinkrev` call. So blindly calling `self.rev()` to avoid another `_adjustlinkrev` call can be counterproductive. Note that `_descendantrev` is currently never taken into account in `introrev` so far which is wrong. We'll fix that in changeset later in this series.

context: fast path linkrev adjustement in trivial case If the search starts from the linkrev, there is nothing to adjust.

url: allow to configure timeout on http connection By default, httplib.HTTPConnection opens connection with no timeout. If the server is hanging, Mercurial will wait indefinitely. This may be an issue for automated scripts. Differential Revision: https://phab.mercurial-scm.org/D4878

obsolete: explicitly track folds inside the markers We now record information to be able to recognize "fold" event from obsolescence markers. To do so, we track the following pieces of information: a) a fold ID. Unique to that fold (per successor), b) the number of predecessors, c) the index of the predecessor in that fold. We will now be able to create an algorithm able to find "predecessorssets". We now store this data in the generic "metadata" field of the markers. Updating the format to have a more compact storage for this would be useful. This way of tracking a fold through multiple markers could be applied to split too. This would have two advantages: 1) We get a simpler format, since number of successors is limited to [0-1]. 2) We can better deal with situations where only some of the split successors are pushed to a remote repository. We should look into the relevance of such a change before updating the on-disk format. note: unlike splits, folds do not have to deal with cases where only some of the markers have been synchronized. As they all share the same successor changesets, they are all relevant to the same nodes.

cleanupnodes: update comment to drop mention of filtering Since changeset 1857f50a9643 drop the filtering, we should not longer mention it in code comment.

treemanifests: remove _loadalllazy when doing copies 'before' here is https://phab.mercurial-scm.org/D4845 (not the committed/rebased version) diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.329 s +- 0.011 s | 1.320 s +- 0.010 s | 99.3% m-u | | x | 1.316 s +- 0.005 s | 1.334 s +- 0.018 s | 101.4% m-u | x | | 1.330 s +- 0.021 s | 1.322 s +- 0.005 s | 99.4% m-u | x | x | 87.2 ms +- 0.7 ms | 86.9 ms +- 1.5 ms | 99.7% l-d-r | | | 203.3 ms +- 7.8 ms | 199.4 ms +- 1.8 ms | 98.1% l-d-r | | x | 204.6 ms +- 2.8 ms | 201.7 ms +- 2.1 ms | 98.6% l-d-r | x | | 90.5 ms +- 11.0 ms | 86.2 ms +- 1.0 ms | 95.2% l-d-r | x | x | 66.3 ms +- 2.0 ms | 66.4 ms +- 0.9 ms | 100.2% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 239.4 ms +- 2.0 ms | 241.7 ms +- 4.6 ms | 101.0% m-u | | x | 128.9 ms +- 1.9 ms | 130.9 ms +- 7.7 ms | 101.6% m-u | x | | 241.1 ms +- 1.6 ms | 240.1 ms +- 1.4 ms | 99.6% m-u | x | x | 133.4 ms +- 1.5 ms | 133.4 ms +- 1.2 ms | 100.0% l-d-r | | | 84.3 ms +- 1.5 ms | 83.5 ms +- 1.0 ms | 99.1% l-d-r | | x | 200.9 ms +- 6.3 ms | 203.0 ms +- 4.4 ms | 101.0% l-d-r | x | | 108.1 ms +- 1.4 ms | 108.7 ms +- 2.1 ms | 100.6% l-d-r | x | x | 190.2 ms +- 4.8 ms | 191.6 ms +- 2.0 ms | 100.7% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.655 s +- 0.029 s | 5.640 s +- 0.036 s | 99.7% m-u | | x | 5.813 s +- 0.038 s | 5.773 s +- 0.028 s | 99.3% m-u | x | | 5.593 s +- 0.043 s | 5.589 s +- 0.028 s | 99.9% m-u | x | x | 648.2 ms +- 19.2 ms | 637.3 ms +- 27.7 ms | 98.3% l-d-r | | | 673.3 ms +- 8.0 ms | 673.2 ms +- 6.8 ms | 100.0% l-d-r | | x | 6.583 s +- 0.030 s | 5.721 s +- 0.028 s | 86.9% <-- l-d-r | x | | 277.8 ms +- 6.7 ms | 276.0 ms +- 2.7 ms | 99.4% l-d-r | x | x | 1.692 s +- 0.013 s | 720.9 ms +- 13.3 ms | 42.6% <-- status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 220.9 ms +- 1.6 ms | 219.9 ms +- 2.2 ms | 99.5% m-u | | x | 109.2 ms +- 1.0 ms | 109.4 ms +- 0.8 ms | 100.2% m-u | x | | 222.6 ms +- 1.7 ms | 221.4 ms +- 2.1 ms | 99.5% m-u | x | x | 113.4 ms +- 0.5 ms | 113.1 ms +- 1.1 ms | 99.7% l-d-r | | | 82.1 ms +- 1.7 ms | 82.1 ms +- 1.2 ms | 100.0% l-d-r | | x | 199.8 ms +- 4.0 ms | 200.7 ms +- 3.6 ms | 100.5% l-d-r | x | | 85.4 ms +- 1.5 ms | 85.2 ms +- 0.3 ms | 99.8% l-d-r | x | x | 202.6 ms +- 4.4 ms | 208.0 ms +- 4.0 ms | 102.7% status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.941 s +- 0.014 s | 1.930 s +- 0.009 s | 99.4% m-u | | x | 1.924 s +- 0.007 s | 1.950 s +- 0.010 s | 101.4% m-u | x | | 1.959 s +- 0.085 s | 1.926 s +- 0.009 s | 98.3% m-u | x | x | 96.2 ms +- 1.0 ms | 96.4 ms +- 0.7 ms | 100.2% l-d-r | | | 604.4 ms +- 10.6 ms | 602.6 ms +- 7.1 ms | 99.7% l-d-r | | x | 605.7 ms +- 4.1 ms | 607.4 ms +- 6.1 ms | 100.3% l-d-r | x | | 182.4 ms +- 1.2 ms | 183.4 ms +- 1.2 ms | 100.5% l-d-r | x | x | 150.8 ms +- 2.0 ms | 150.6 ms +- 1.0 ms | 99.9% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.185 s +- 0.027 s | 3.181 s +- 0.017 s | 99.9% m-u | | x | 3.028 s +- 0.021 s | 2.954 s +- 0.010 s | 97.6% m-u | x | | 3.168 s +- 0.010 s | 3.175 s +- 0.023 s | 100.2% m-u | x | x | 317.5 ms +- 3.5 ms | 313.2 ms +- 2.9 ms | 98.6% l-d-r | | | 456.2 ms +- 10.6 ms | 454.4 ms +- 5.8 ms | 99.6% l-d-r | | x | 9.236 s +- 0.063 s | 757.9 ms +- 9.2 ms | 8.2% <-- l-d-r | x | | 257.6 ms +- 2.3 ms | 261.2 ms +- 1.7 ms | 101.4% l-d-r | x | x | 1.614 s +- 0.013 s | 478.0 ms +- 14.3 ms | 29.6% <-- Differential Revision: https://phab.mercurial-scm.org/D4875

treemanifests: store whether a lazydirs entry needs copied after materializing Due to the way that things like manifestlog.get caches its values, without making a copy (if necessary) after calling readsubtree(), we might end up adjusting the state of the same object on different contexts, breaking things like dirty state tracking (and probably other things). Differential Revision: https://phab.mercurial-scm.org/D4874

treemanifests: extract _loaddifflazy from _diff, use in _filesnotin Differential Revision: https://phab.mercurial-scm.org/D4873

identify: show remote bookmarks in `hg id url -Tjson -B` I didn't display bookmarks when `default and not ui.quiet`: it seems strange for templates to depend on --id or -q, and it would take more code for `hg id url -T {node}` to not request remote bookmarks. An alternative I thought of was providing lazy data to the formatter, `fm.data(bookmarks=lambda: fm.formatlist(getbms(), name='bookmark'))`. The plainformatter would naturally not compute it, the templateformatter would compute only what it needs, and the other ones would compute everything, but that's not supported (or I don't see how), so I abandoned this idea. Differential Revision: https://phab.mercurial-scm.org/D4872

showstack: also handle SIGALRM This is looking *very* handy when debugging mysterious hangs in a test: you can wrap a hanging invocation in `perl -e 'alarm shift @ARGV; exec @ARGV' 1` for example, a hanging `hg pull` becomes `perl -e 'alarm shift @ARGV; exec @ARGV' 1 hg pull` where the `1` is the timeout in seconds before the process will be hit with SIGALRM. After making that edit to the test file, you can then use --extra-config-opt on run-tests.py to globaly enable showstack during the test run, so you'll get full stack traces as you force your hg to exit. I wonder (but only a little, not enough to take action just yet) if we should wire up some scaffolding in run-tests itself to automatically wrap all commands in alarm(3) somehow to avoid hangs in the future? Differential Revision: https://phab.mercurial-scm.org/D4870

exchangev2: add progress bar around manifest scanning This can take a long time on large repositories. Let's add a progress bar so we don't have long periods where it isn't obvious what is going on. Differential Revision: https://phab.mercurial-scm.org/D4859