rust: core implementation for lazyancestors Once exposed through appropriate bindings, this should be able to replace ancestor.lazyancestors entirely. Differential Revision: https://phab.mercurial-scm.org/D5440

rust-cpython: binding for AncestorsIterator It's now reachable from Python as rustext.ancestor.AncestorsIterator Tests are provided in the previously introduced Python testcase: this is much more convenient that writing lengthy Rust code to call into Python. Differential Revision: https://phab.mercurial-scm.org/D5439

rust-cpython: implement Graph using C parents function We introduce the `Index` struct that wraps the C index. It is not intrinsically protected by the GIL (see the lengthy discussion in its docstring). Improving on this seems prematurate at this point. A pointer to the parents function is stored on the parsers C extension module as a capsule object. This is the recommended way to export a C API for consumption from other extensions. See also: https://docs.python.org/2.7/c-api/capsule.html In our case, we use it in cindex.rs, retrieving function pointer from the capsule and storing it within the CIndex struct, alongside with a pointer to the index. From there, the implementation is very close to the one from hg-direct-ffi. The naming convention for the capsule is inspired from the one in datetime: >>> import datetime >>> datetime.datetime_CAPI <capsule object "datetime.datetime_CAPI" at 0x7fb51201ecf0> although in datetime's case, the capsule points to a struct holding several type objects and methods. Differential Revision: https://phab.mercurial-scm.org/D5438

pull: fix inconsistent view of bookmarks during pull (issue4700) I had a share where a pull apparently pulled a bookmark but not the revision pointed to by the bookmark, which I suspect is due to this (and if not, we might as well remove known issues in this area). I do this by combining doing all the queries that could read the bookmarks in one round trip. I had to change the handling of the case where the server doesn't support the lookup query, because if it fails, it would otherwise make fremotebookmark.result() block forever. This is due to wireprotov1peer.peerexecutor.sendcommands's behavior (it fills a single future if any query fails synchronously and leaves all other futures unchanged), but I don't know if the fix is to cancel all other futures, or to keep going with the other queries. Differential Revision: https://phab.mercurial-scm.org/D5449

merge: modify the logical statement Differential Revision: https://phab.mercurial-scm.org/D5476

exthelper: correct a documentation typo

lfs: convert to using exthelper to wrap functions I'm not 100% sure that upgraderequirements() can be double annotated safely, but it seems OK based on printing the address of the function being wrapped. One thing I've noticed is that @eh.reposetup doesn't do the usual check to ensure that it's a local repo. Should that be baked into @eh.reposetup() somehow, possibly with a non-default option to skip the check? It seems like a gaping hole if every function that gets registered needs to add this check.

tests: convert a test extension to use exthelper This provides test coverage to uipopulate().

exthelper: drop fileset/revset/template support for now Yuya raised concerns about duplicating registrar functionality. There are a couple of ideas to work around this, which would allow bringing them back, and then backporting to evolve. For now, I just want to get the subsequent changes landed before the bulk b'' rewrite makes rebasing too hard.

exthelper: simplify configitem registration

extensions: import the exthelper class from evolve 980565468003 (API) This should help make extensions that wrap a lot of stuff more comprehendible. It was copied unmodified, except: - fix up the imports - rename final_xxxsetup() -> finalxxxsetup() to appease checkcode - avoid a [] default arg to wrapcommand() .. api:: Add `exthelper` class to simplify extension writing by allowing functions, commands, and configitems to be registered via annotations. The previous APIs are still available for use.

narrow: detect if narrowspec was changed in a different share With this commit, `hg share` should be usable with narrow repos. Design explained on https://www.mercurial-scm.org/wiki/NarrowSharePlan I was running into cache invalidation problems when updating the narrowspec. After spending a day trying to figure out a good solution, I resorted to just assigning repo.narrowpats and repo._narrowmatch after invalidating them. Differential Revision: https://phab.mercurial-scm.org/D5278

tests: add test for narrow+share For how narrow+share is supposed to work, see https://www.mercurial-scm.org/wiki/NarrowSharePlan. Differential Revision: https://phab.mercurial-scm.org/D5276

narrow: keep narrowspec backup in store As suggested by Yuya in review of D4099. Differential Revision: https://phab.mercurial-scm.org/D5470

tests: update narrowspec when narrowspec, not dirstate, is accessed test-narrow-expanddirstate.t mimics a Google-internal extension that updates the narrowspec whenever the dirstate is accessed. Since 1d09ba0d2ed3 (narrow: move remaining narrow-limited dirstate walks to core, 2018-10-01) and a few commits before it, we no longer restrict repo.dirstate.walk() to the narrowspec. It is instead done at a higher level (e.g. context.status()). We were running into problems with the Google-internal extension when importing those commits. The issue was that the narrowspec was read before the first dirstate access. I believe the right fix is to instead update the narrowspec when trying to read it (not when reading the dirstate), so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D5275

merge: extract helper for creating empty "actions" dict Replicating the set of actions in multiple places is bad. Differential Revision: https://phab.mercurial-scm.org/D5472

manifest: accept narrowmatch into constructor instead of getting from repo The manifest should ideally not know at all about the repo, so this is just a little step towards cleaning that up. Differential Revision: https://phab.mercurial-scm.org/D5469

py3: byteify one more sys.argv in gendoc.py

test: introduce a new flag to display env variable line per line It's easier to conditionalize some of the environment variables per Mercurial version once there is only one value per line. Differential Revision: https://phab.mercurial-scm.org/D5453

revlog: add an explicit test for `issnapshot` We test the method on a real revlog containing "real" data.

revlog: add some direct testing of the slicing logic This test check slicing backed by an actual revlog. It will test the C version of slicing (if the test are run with the C extensions).

revlog: limit base to rev size ratio to 500 instead of 50 While a value of 50 provided large speedup in some case (400s → 7s) it also creates a slow down for a whole class of revision we are seeing in a private repository (0.1s → 3s). A value of 500 makes them disappear improving the total runtime (the slower revision still improve significantly (400s → 21s)). We need to run a wider array of tests on various repositories to see the effect on speed and size of different values for this acceptable constant. However, in the meantime, it seems safer to move back to a less impactful value.

contrib: remove unused version of os Caught by test-check-pyflakes.t Differential Revision: https://phab.mercurial-scm.org/D5471

py3: convert `'{}'.format(foo)` to `'%s' % foo` in the bookflow extension Byte strings don't have the former. Converting these to byte strings is waiting on the mass rewrite.

py3: byteify sys.argv in gendoc.py

repository: update interface signature of narrowmatch() This should have been part of 4fd0fac48922 (localrepo: allow narrowmatch() to accept matcher to intersect with, 2018-09-28) and 41fcdfe3bfeb (narrow: allow repo.narrowmatch(match) to include exact matches from "match", 2018-10-01). Differential Revision: https://phab.mercurial-scm.org/D5466

narrow: when narrowing, write new narrowspec before removing revlogs If revlogs were removed and then the process crashed before the narrowspec was written, the repo would be corrupt. Differential Revision: https://phab.mercurial-scm.org/D5467

narrow: replace "ui.warn(); return 1" by "raise error.Abort()" This is the usual way of doing it and I don't see a reason to do it differently here. Differential Revision: https://phab.mercurial-scm.org/D5468

test-bookmarks-pushpull: add failing test of issue4700 Differential Revision: https://phab.mercurial-scm.org/D5447

tests: update printenv.py argument parsing We are about to introduce a new flag for printing the HG environment variables one per line and it's easier to do when using the argparse module for argument parsing. Differential Revision: https://phab.mercurial-scm.org/D5452

fuzz: new fuzzer for parsers.fm1readmarkers Differential Revision: https://phab.mercurial-scm.org/D5465

server: always close http socket if responding with an error (issue6033) It's possible for hgweb to respond _very_ early with an error if we're catching certain types of errors. When we do, we need to tell the client the socket is toast when there's a POST involved because otherwise there can be lingering POST data on the socket that will confuse any future requests on the socket. This manifested as a flaky failure on Linux in an lfs extension test and a reliable failure on FreeBSD. With this patch applied, test-lfs-serve-access.t now passes for me on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5498

match: fix assertion for fileset with no context (issue6046) A falsy changectx should be allowed.

templatekw: fix documentation typos

update: do not pass in user revspec as default destination (issue6044) When the revsingle() was introduced at 61c0df2b089a, it couldn't handle revspec=0 (not '0') properly. That's probably why the default was set to rev. This is technically BC since "hg update ''" was identical to "hg update '.'" whereas "hg update -r ''" is "hg update", but I believe that's a bug given no test fails with this change.

revlog: cache delta base value under -1 Such base are invalid so we better report them early.

revlog: catch revlog corruption in index_baserev A revision cannot use a base above itself, it can only happens one corrupted repository. Ignoring such corrupted could lead to infinite loop.

phabricator: properly encode boolean types in the request body I tripped over this playing with `hg debugcallconduit` to query for valid reviewers. If the JSON on stdin is written as 'True' or 'False', python complains it isn't valid JSON. If it's written as 'true' or 'false', it made it to the server, but got kicked back with this: abort: Conduit Error (ERR-CONDUIT-CORE): Error while reading "isBot": Expected boolean (true or false), got something else. The test isn't really relevant here (the code can be reverted, and it will pass), but this gives us coverage for the debug command.

parsers: better bounds checking in fm1readmarkers Our Python already calls this with reasonable values consistently, but my upcoming fuzzer is extremely quick to discover the lack of sanity checking here. Differential Revision: https://phab.mercurial-scm.org/D5464

fuzz: new fuzzer for dirstate parser Differential Revision: https://phab.mercurial-scm.org/D5463

fuzz: new fuzzer for revlog's parse_index2 method Differential Revision: https://phab.mercurial-scm.org/D5462

fuzz: extract Python initialization to utility package Avoids code duplication between fuzzers of parsers.so. Differential Revision: https://phab.mercurial-scm.org/D5461

fuzz: remove probably-wrong -fsanitize from fuzzutil.o rule Differential Revision: https://phab.mercurial-scm.org/D5460

parsers: remove long-dead parse_manifest method We haven't used this in years, I think it's fine to ditch it now. We had previously kept it around to ease bisecting with built extensions, but these days we've got a better versioning scheme anyway. Noticed this method kicking around while looking in parsers.so for likely fuzzing targets. Differential Revision: https://phab.mercurial-scm.org/D5459

help: hide default value for default-off flags If we no longer show the "[no-]" for default-off flags, it also seems unnecessary to show the "default: off" for them, since that's quite clearly the default. It's extra confusing for action flags like `hg bookmarks --delete`. Differential Revision: https://phab.mercurial-scm.org/D5455

help: show "[no-]" only for default-on Flags As Anton (av6) pointed out, the "[no-]" is confusing for action flags like `hg bookmark --delete`. We could come up with a way of indicating which flags are action flags (e.g. use None for the default value instead of False). However, it's probably also unlikely that users will want to negate even non-action flags like --hidden. One of the more common flags where the "[no-]" prefix would be useful is `hg evolve --update`. The reason it's helpful there is that it defaults to on. So I think we can simply include "[no-]" only for flags that are on by default (and thus require the user to add the "[no-]" for the option to have any effect). Note that there are use cases for negating flags that already off by default. For example, you may have an alias for `hg log -G --hidden -T foo` and now want to pass "--no-hidden" to that alias. However, I think that users who want that are likely to be advanced enough that they've already learnt about the "no-" prefix by seeing it somewhere else. Differential Revision: https://phab.mercurial-scm.org/D5454

shelve: drop unnecessary backup of narrowspec I mechanically added the backup code everywhere in ad24b581e4d9 (narrow: call narrowspec.{save,restore,clear}backup directly, 2018-08-03), but I can't think of a reason it would be needed in the shelve code, so let's drop it. Differential Revision: https://phab.mercurial-scm.org/D5457

shelve: pass transaction around to clarify where it's used Differential Revision: https://phab.mercurial-scm.org/D5456

py3: byteify docchecker The exception is printed as str because I'm too lazy to convert it and the pieces.

py3: replace hard coded `python` with $PYTHON in check-gendoc

py3: byteify gendoc.py This is mostly b'' prefixing, with some cargoculting of help.py to get around `textwrap.dedent()` and __doc__ string requirements.

py3: use bytes stdout in hghave.py This fixes a failure in test-run-tests.t around notarealhghavefeature. It seems crazy to me that all of this needs to be adjusted in all of these tests, but the line as run-tests.py sees it in _processoutput() before doing anything is already mangled with a trailing '\r'. Switching to normalizenewlines=True for TTest works, but I'm sure that breaks other stuff.

perfrevlogwrite: fix a typo in the option name The submitted patches was apparently damaged, and a 's' was missing.

fuzz: improve makefile per feedback from some folks at work This is a *ton* better. Once this lands, I can remove a nasty kludge from the oss-fuzz build.sh. Bonus: this fixes the coverage build. Differential Revision: https://phab.mercurial-scm.org/D5458

py3: use bytes stdout in test-check-help.t Setting stdout to binary seemed to have no effect on Windows, as it was appending a literal '\r' to each topic keyword. This also stops prepending 'b' to the topic on all platforms as well.

commandserver: preload repository in master server and reuse its file cache This greatly speeds up repository operation with lots of obsolete markers: $ ls -lh .hg/store/obsstore -rw-r--r-- 1 yuya yuya 21M Dec 2 17:55 .hg/store/obsstore $ time hg log -G -l10 --pager no (hg) 1.79s user 0.13s system 99% cpu 1.919 total (chg uncached) 0.00s user 0.01s system 0% cpu 1.328 total (chg cached) 0.00s user 0.00s system 3% cpu 0.180 total As you can see, the implementation of the preloader function is highly experimental. It works, but I'm yet to be sure how things can be organized. So I don't want to formalize the API at this point.

commandserver: add IPC channel to teach repository path on command finished The idea is to load recently-used repositories first in the master process, and fork(). The forked worker can reuse a warm repository if it's preloaded. There are a couple of ways of in-memory repository caching. They have pros and cons: a. "preload by master" pros: can use a single cache dict, maximizing cache hit rate cons: need to reload a repo in master process (because worker process dies per command) b. "prefork" pros: can cache a repo without reloading (as worker processes persist) cons: lower cache hit rate since each worker has to maintain its own cache c. "shared memory" (or separate key-value store server) pros: no need to reload a repo in master process, ideally cons: need to serialize objects to sharable form Since my primary goal is to get rid of the cost of loading obsstore without massive rewrites, (c) doesn't work. (b) isn't ideal since it would require much more SDRAMs than (a). So I take (a). The idea credits to Jun Wu.

upgrade: correct implementation of improvement.__ne__ The "not" operator binds more closely than "==": >>> not False == False False

extensions: use ui.log() interface to provide detailed loading information The output format changes and the messages will be sent to stderr instead of stdout, but I don't think that matters.

mq: implement log() on dummyui Otherwise ui.log() in extensions.py would explode.

ui: install logger that sends debug.extensions messages to stderr This will replace the custom log function introduced at d58958676b3c "extensions: add detailed loading information."

blackbox: resurrect recursion guard If I added ui.log() to hg.repository() function, test-merge-subrepos.t exploded. The problem is that the blackbox may create new repository instance while logging is active, and the created repository owns its new ui derived from the baseui, not from the ui which is processing the active logging. I tried to work around the issue in ui.log(), but that turned out to be not easy. We shouldn't globally lock the ui.log() since there may be more than one active repo/ui instances in threaded environment. We could store the logging state in thread-local storage, but that seems unnecessarily complex. So this patch reintroduces the _inlog flag to per-repository logger instances.

tests: filter out uninteresting log events This helps adding more log()s without updating the tests.

context: error out if basefilectx.cmp() is called without self._filenode The base implementation can't handle such cases because the filelog has no knowledge about the working directory. Loading self._filenode should have no extra cost since self.size() would load it anyway.

context: collapse complex condition to see if filelog have to be compared It's hard to read. I'd rather make the return statement duplicated.

merge with stable

worker: do not swallow exception occurred in main process Before, SystemExit(255) would be most likely raised since the worker processes were terminated by the main process and the status would be set to 255 in response. We should instead re-raise the exception occurred first. It's pretty hard to debug problems like the issue 6035 with no traceback.

run-tests: fix permission to clean up unreadable directories I found many hgtests.* directories left in $TMPDIR, which couldn't be deleted because test-ssh-repoerror.t created some directories with a-rx mode.

context: reimplement memfilectx.cmp() If I added a sanity check to basefilectx, test-context.py exploded. This patch copies the naive implementation from overlayworkingfilectx.

rust-cpython: build and support for Python3 Defined Cargo features for Python3, making them overall simpler to use, hooked them in build and made mercurial.rustext importable. This is tested with Python 3.6.7. Differential Revision: https://phab.mercurial-scm.org/D5446

py3: convert popen() command arguments in hgclient to str on Windows This fixes test-commandserver.t and test-keyword.t, which was previously complaining TypeError("a bytes-like object is required, not 'str'")

py3: byteify tests/pullext.py # skip-blame for b'' prepending

py3: byteify the fakeversion extension in test-http-bad-server.t # skip-blame for b'' prefixing

py3: spawn all python instances with legacy stdio enabled on Windows This fixes 6 more tests. See ef7119cd4965.

setup: avoid attempting to invoke the system-wide hg.exe on Windows On Windows, the executable in the current directory gets priority over anything in $PATH (both for cmd.exe and MSYS). That means, the former code was launching the local hg.exe instead of the system-wide one, if it was previously built. If that failed, it then fell back to the local hg code, but run through python.exe. I'm not sure what it is about ef7119cd4965, but that started throwing up a messagebox that python37.dll couldn't be loaded. (And indeed, python37 is not in $PATH by default.) Invoking the local hg via the current python avoids that.

delta: ignore base whose chains already don't match expectations If we know the existing chain does not match our criteria, there is no point to build a delta to append. This is especially useful when dealing with a full text much smaller than its parent. In that case, the parent chain is probably already too large. example affected manifest write before: 1.421005s after: 0.815520s (-42%)

delta: exclude base candidate much smaller than the target If a revision's full text is that much bigger than a base candidate full text, we no longer consider that candidate. This solves a pathological case we encountered on a very specify repository. It contains a long series of changesets with a very small manifest (one file) co-existing with others changesets using a very large manifest. Without this filtering, we ended up considering a large number of tiny full snapshots as a potential base. It resulted in very large delta (the size of the full text) and mercurial spending 99% of its time compressing these deltas. The timing of a commit moved from about 400s to about 10s (still slow, but not ridiculously slow).

perfrevflogwrite: clear revlog cache between each write We want to measure write time from a cold cache (similar to commit). So we need to clear the cache to prevent computation from rev N-1 to interfere with rev N.

py3: enable legacy stdio mode in exewrapper This drops the test failure count from 166 to 117. The failures were typically in the form of `hg serve -d` spawning but crashing immediately, leaving clients with "bad http status" lines, connection refusals, and so forth. The underlying message on the server side was either "OSError: [WinError 6] The handle is invalid" or "OSError: [WinError 1] Incorrect function". Additionally, no output was rendered if the pager was activated. Thanks to Yuya for diagnosing the problem. The failure count drops to 107 when PYTHONLEGACYWINDOWSSTDIO=1 is defined in the environment. These failures seem to revolve around the dummyssh server process, and dumbhttp.py. So I'll probably add that to the test runner. One small regression here (only in py3) is that if hg.exe is already built, a messagebox appears when building it again saying that python37.dll can't be loaded. Python3 isn't in PATH by default, and setup.py tries running bare `hg` first. But MSYS prepends '.' to PATH, so it runs the local hg, but can't find the library. It falls back to the python used to invoke setup.py, so ultimately it works. I'm not sure if it's better to strip '.' from PATH or just skip right to `sys.executable hg` on Windows. Also, something seems to be wrong with run-tests._usecorrectpython(). I accidentially left off the 'PYTHON="py -3"' when building (thus making py2 stuff), and yet managed to invoke run-tests.py with "py -3". (And that only had 67 failures.)

run-tests: alias hg to hg.exe on Windows To enable legacy stdio mode on Windows, hg.exe needs to be updated. But before that, we actually have to use it when running the tests. I *think* what was happening before was when MSYS invoked `hg`, it looked at the shbang line and ran python.exe found there. The test harness must be updating $PATH to include the python used to launch it, and therefore it ran py3. As a side note, this also fixed `py -3 run-tests.py` (without --local), which complained about the space in the shbang line before this. This should also help in WSL, because the explicit '.exe' is needed to invoke a Windows app instead of invoking the Linux app. I have no idea why this change capitalizes Lib in the tests, as it was previously lowercase for both py2 and py3.

py3: conditionalize the python version in test-install.t

tests: followup on network related errors on Debian 9 First, update test-clonebundles.t which was failing since the introduction of the `$EADDRNOTAVAIL$` common pattern. Also update two tests from 8695fbe17f7c with a more compact condition line. Differential Revision: https://phab.mercurial-scm.org/D5445

py3: unimplement RevlogError.__str__() On Python 2, str(exc) would crash if __str__() returned a unicode containing non-ASCII characters.

py3: use forcebytestr() to stringify hook exception This fixes test-narrow-exchange.t.

color: fix a documentation typo

py3: quote $PYTHON in test-patchbomb.t for Windows I couldn't get the quoting right in the environment variable, so now it's a function.

rust-cpython: testing the bindings from Python This is easier and more convincing than doing the same tests from a Rust tests module. Differential Revision: https://phab.mercurial-scm.org/D5437

rust-cpython: build via HGWITHRUSTEXT=cpython The existing behaviour, building the direct ffi bindings if HGIWTHRUSTEXT is just set is unchanged, but if HGWITHRUSTEXT is cpython, then the cpython bindings (aka mercurial/rustext.so) are built. Differential Revision: https://phab.mercurial-scm.org/D5436

rust: better treatment of cargo/rustc errors Differential Revision: https://phab.mercurial-scm.org/D5435

rust-cpython: start cpython crate bindings This changeset introduces the hg-cpython crate, that compiles as a shared library holding a whole Python package (mercurial.rustext), with only the empty 'ancestor' submodule for now. Such bindings will be easier and safer to develop and maintain that those of `hg-direct-ffi`. They don't involve C code, only unsafe Rust that's mostly isolated within the cpython crate. The long-term goal would be to import the provided modules, such as rustext.ancestor with mercurial.policy.importmod, same as we already do with cext modules. Differential Revision: https://phab.mercurial-scm.org/D5434

rust-cpython: exclude hgcli from workspace hgcli uses a specific rust-cpython commit by indygreg, of which a PR has been derived which is not merged nor released yet. But we can't use several versions of the sys-python2.7 crate in a single workspace: it makes for a build error. Since hgcli does not at the time being need anything from hg-core, whereas the upcoming hg-cpython will. So for now we're moving hgcli aside, hoping we could base all of them on the same version of rust-cpython again in the future. Differential Revision: https://phab.mercurial-scm.org/D5433

sparse-revlog: protect C code against delta chain including nullrev For unclear reasons, some repositories include nullrev (-1). Re-computing delta for such repo remove nullrev from all chain, so some older versions have been creating them. This currently raise an IndexError with the new C code doing chain slicing as it expect all item to be positive. Both python and C code for reading delta chain preserve nullrev, and the Python code for chain slicing handle the case fine. So we take the safe route and make the new C code works fine in that case.

sparse-revlog: handle nullrev in index_get_length The more generic index_get method handle nullrev fine, we apply the same logic here.

sparse-revlog: handle nullrev in index_get_start The more generic index_get method handle nullrev fine, we apply the same logic here.

revlog: introduce a constant for nullrev in `revlog.c` The value is important enough to be explicitly tracked.

rust: translation of missingancestors This is as direct as possible a translation of the ancestor.missingancestors Python class in pure Rust. The goal for this changeset is to make it easy to compare with the Python version. We also add to Python tests the cases that helped us develop and debug this implementation. Some possible optimizations are marked along the way as TODO comments Differential Revision: https://phab.mercurial-scm.org/D5416