Boris Feld <boris.feld@octobus.net> [Wed, 19 Dec 2018 15:42:20 +0100] rev 41028
tests: update printenv.py argument parsing
We are about to introduce a new flag for printing the HG environment variables
one per line and it's easier to do when using the argparse module for argument
parsing.
Differential Revision: https://phab.mercurial-scm.org/D5452
Augie Fackler <augie@google.com> [Thu, 20 Dec 2018 01:22:58 -0500] rev 41027
fuzz: new fuzzer for parsers.fm1readmarkers
Differential Revision: https://phab.mercurial-scm.org/D5465
Augie Fackler <augie@google.com> [Thu, 20 Dec 2018 01:26:39 -0500] rev 41026
parsers: better bounds checking in fm1readmarkers
Our Python already calls this with reasonable values consistently, but
my upcoming fuzzer is extremely quick to discover the lack of sanity
checking here.
Differential Revision: https://phab.mercurial-scm.org/D5464
Augie Fackler <augie@google.com> [Wed, 19 Dec 2018 23:48:35 -0500] rev 41025
fuzz: new fuzzer for dirstate parser
Differential Revision: https://phab.mercurial-scm.org/D5463
Augie Fackler <augie@google.com> [Wed, 19 Dec 2018 20:26:53 -0500] rev 41024
fuzz: new fuzzer for revlog's parse_index2 method
Differential Revision: https://phab.mercurial-scm.org/D5462
Augie Fackler <augie@google.com> [Wed, 19 Dec 2018 21:57:23 -0500] rev 41023
fuzz: extract Python initialization to utility package
Avoids code duplication between fuzzers of parsers.so.
Differential Revision: https://phab.mercurial-scm.org/D5461
Augie Fackler <augie@google.com> [Wed, 19 Dec 2018 23:40:37 -0500] rev 41022
fuzz: remove probably-wrong -fsanitize from fuzzutil.o rule
Differential Revision: https://phab.mercurial-scm.org/D5460
Augie Fackler <augie@google.com> [Wed, 19 Dec 2018 23:51:02 -0500] rev 41021
parsers: remove long-dead parse_manifest method
We haven't used this in years, I think it's fine to ditch it now. We
had previously kept it around to ease bisecting with built extensions,
but these days we've got a better versioning scheme anyway. Noticed
this method kicking around while looking in parsers.so for likely
fuzzing targets.
Differential Revision: https://phab.mercurial-scm.org/D5459
Martin von Zweigbergk <martinvonz@google.com> [Wed, 19 Dec 2018 09:33:42 -0800] rev 41020
help: hide default value for default-off flags
If we no longer show the "[no-]" for default-off flags, it also seems
unnecessary to show the "default: off" for them, since that's quite
clearly the default. It's extra confusing for action flags like `hg
bookmarks --delete`.
Differential Revision: https://phab.mercurial-scm.org/D5455
Martin von Zweigbergk <martinvonz@google.com> [Wed, 19 Dec 2018 09:20:32 -0800] rev 41019
help: show "[no-]" only for default-on Flags
As Anton (av6) pointed out, the "[no-]" is confusing for action flags
like `hg bookmark --delete`. We could come up with a way of indicating
which flags are action flags (e.g. use None for the default value
instead of False). However, it's probably also unlikely that users
will want to negate even non-action flags like --hidden.
One of the more common flags where the "[no-]" prefix would be useful
is `hg evolve --update`. The reason it's helpful there is that it
defaults to on. So I think we can simply include "[no-]" only for
flags that are on by default (and thus require the user to add the
"[no-]" for the option to have any effect).
Note that there are use cases for negating flags that already off by
default. For example, you may have an alias for `hg log -G --hidden -T
foo` and now want to pass "--no-hidden" to that alias. However, I
think that users who want that are likely to be advanced enough that
they've already learnt about the "no-" prefix by seeing it somewhere
else.
Differential Revision: https://phab.mercurial-scm.org/D5454
Martin von Zweigbergk <martinvonz@google.com> [Wed, 05 Dec 2018 15:37:03 -0800] rev 41018
shelve: drop unnecessary backup of narrowspec
I mechanically added the backup code everywhere in ad24b581e4d9
(narrow: call narrowspec.{save,restore,clear}backup directly,
2018-08-03), but I can't think of a reason it would be needed in the
shelve code, so let's drop it.
Differential Revision: https://phab.mercurial-scm.org/D5457
Martin von Zweigbergk <martinvonz@google.com> [Mon, 07 May 2018 17:08:17 -0700] rev 41017
shelve: pass transaction around to clarify where it's used
Differential Revision: https://phab.mercurial-scm.org/D5456
Matt Harbison <matt_harbison@yahoo.com> [Wed, 19 Dec 2018 13:35:11 -0500] rev 41016
py3: byteify docchecker
The exception is printed as str because I'm too lazy to convert it and the
pieces.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 19 Dec 2018 13:32:42 -0500] rev 41015
py3: replace hard coded `python` with $PYTHON in check-gendoc
Matt Harbison <matt_harbison@yahoo.com> [Tue, 18 Dec 2018 21:17:27 -0500] rev 41014
py3: byteify gendoc.py
This is mostly b'' prefixing, with some cargoculting of help.py to get around
`textwrap.dedent()` and __doc__ string requirements.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 19 Dec 2018 14:51:21 -0500] rev 41013
py3: use bytes stdout in hghave.py
This fixes a failure in test-run-tests.t around notarealhghavefeature. It seems
crazy to me that all of this needs to be adjusted in all of these tests, but the
line as run-tests.py sees it in _processoutput() before doing anything is
already mangled with a trailing '\r'. Switching to normalizenewlines=True for
TTest works, but I'm sure that breaks other stuff.
Boris Feld <boris.feld@octobus.net> [Wed, 19 Dec 2018 02:57:48 +0100] rev 41012
perfrevlogwrite: fix a typo in the option name
The submitted patches was apparently damaged, and a 's' was missing.
Augie Fackler <augie@google.com> [Wed, 19 Dec 2018 19:20:21 -0500] rev 41011
fuzz: improve makefile per feedback from some folks at work
This is a *ton* better. Once this lands, I can remove a nasty kludge
from the oss-fuzz build.sh. Bonus: this fixes the coverage build.
Differential Revision: https://phab.mercurial-scm.org/D5458
Matt Harbison <matt_harbison@yahoo.com> [Tue, 18 Dec 2018 14:01:06 -0500] rev 41010
py3: use bytes stdout in test-check-help.t
Setting stdout to binary seemed to have no effect on Windows, as it was
appending a literal '\r' to each topic keyword. This also stops prepending 'b'
to the topic on all platforms as well.
Yuya Nishihara <yuya@tcha.org> [Wed, 31 Oct 2018 22:43:08 +0900] rev 41009
commandserver: preload repository in master server and reuse its file cache
This greatly speeds up repository operation with lots of obsolete markers:
$ ls -lh .hg/store/obsstore
-rw-r--r-- 1 yuya yuya 21M Dec 2 17:55 .hg/store/obsstore
$ time hg log -G -l10 --pager no
(hg) 1.79s user 0.13s system 99% cpu 1.919 total
(chg uncached) 0.00s user 0.01s system 0% cpu 1.328 total
(chg cached) 0.00s user 0.00s system 3% cpu 0.180 total
As you can see, the implementation of the preloader function is highly
experimental. It works, but I'm yet to be sure how things can be organized.
So I don't want to formalize the API at this point.
Yuya Nishihara <yuya@tcha.org> [Wed, 31 Oct 2018 22:19:03 +0900] rev 41008
commandserver: add IPC channel to teach repository path on command finished
The idea is to load recently-used repositories first in the master process,
and fork(). The forked worker can reuse a warm repository if it's preloaded.
There are a couple of ways of in-memory repository caching. They have pros
and cons:
a. "preload by master"
pros: can use a single cache dict, maximizing cache hit rate
cons: need to reload a repo in master process (because worker process
dies per command)
b. "prefork"
pros: can cache a repo without reloading (as worker processes persist)
cons: lower cache hit rate since each worker has to maintain its own cache
c. "shared memory" (or separate key-value store server)
pros: no need to reload a repo in master process, ideally
cons: need to serialize objects to sharable form
Since my primary goal is to get rid of the cost of loading obsstore without
massive rewrites, (c) doesn't work. (b) isn't ideal since it would require
much more SDRAMs than (a). So I take (a).
The idea credits to Jun Wu.
Benjamin Peterson <benjamin@python.org> [Thu, 13 Dec 2018 23:20:28 -0800] rev 41007
upgrade: correct implementation of improvement.__ne__
The "not" operator binds more closely than "==":
>>> not False == False
False
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Dec 2018 16:28:29 +0900] rev 41006
extensions: use ui.log() interface to provide detailed loading information
The output format changes and the messages will be sent to stderr instead of
stdout, but I don't think that matters.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Dec 2018 16:52:04 +0900] rev 41005
mq: implement log() on dummyui
Otherwise ui.log() in extensions.py would explode.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Dec 2018 16:24:08 +0900] rev 41004
ui: install logger that sends debug.extensions messages to stderr
This will replace the custom log function introduced at d58958676b3c
"extensions: add detailed loading information."
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Dec 2018 19:05:42 +0900] rev 41003
blackbox: resurrect recursion guard
If I added ui.log() to hg.repository() function, test-merge-subrepos.t
exploded. The problem is that the blackbox may create new repository instance
while logging is active, and the created repository owns its new ui derived
from the baseui, not from the ui which is processing the active logging.
I tried to work around the issue in ui.log(), but that turned out to be not
easy. We shouldn't globally lock the ui.log() since there may be more than
one active repo/ui instances in threaded environment. We could store the
logging state in thread-local storage, but that seems unnecessarily complex.
So this patch reintroduces the _inlog flag to per-repository logger instances.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Dec 2018 17:52:14 +0900] rev 41002
tests: filter out uninteresting log events
This helps adding more log()s without updating the tests.
Yuya Nishihara <yuya@tcha.org> [Sun, 16 Dec 2018 16:31:31 +0900] rev 41001
context: error out if basefilectx.cmp() is called without self._filenode
The base implementation can't handle such cases because the filelog has no
knowledge about the working directory.
Loading self._filenode should have no extra cost since self.size() would
load it anyway.
Yuya Nishihara <yuya@tcha.org> [Sun, 16 Dec 2018 16:24:45 +0900] rev 41000
context: collapse complex condition to see if filelog have to be compared
It's hard to read. I'd rather make the return statement duplicated.
Augie Fackler <augie@google.com> [Tue, 18 Dec 2018 10:21:25 -0500] rev 40999
merge with stable
Georges Racinet <gracinet@anybox.fr> [Mon, 17 Dec 2018 15:05:52 +0100] rev 40998
rust-cpython: build and support for Python3
Defined Cargo features for Python3, making them overall simpler to
use, hooked them in build and made mercurial.rustext importable.
This is tested with Python 3.6.7.
Differential Revision: https://phab.mercurial-scm.org/D5446
Matt Harbison <matt_harbison@yahoo.com> [Tue, 18 Dec 2018 00:54:32 -0500] rev 40997
py3: convert popen() command arguments in hgclient to str on Windows
This fixes test-commandserver.t and test-keyword.t, which was previously
complaining
TypeError("a bytes-like object is required, not 'str'")
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Dec 2018 23:08:26 -0500] rev 40996
py3: byteify tests/pullext.py
# skip-blame for b'' prepending
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Dec 2018 21:27:37 -0500] rev 40995
py3: byteify the fakeversion extension in test-http-bad-server.t
# skip-blame for b'' prefixing
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Dec 2018 18:25:18 -0500] rev 40994
py3: spawn all python instances with legacy stdio enabled on Windows
This fixes 6 more tests. See ef7119cd4965.
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Dec 2018 17:44:45 -0500] rev 40993
setup: avoid attempting to invoke the system-wide hg.exe on Windows
On Windows, the executable in the current directory gets priority over anything
in $PATH (both for cmd.exe and MSYS). That means, the former code was launching
the local hg.exe instead of the system-wide one, if it was previously built. If
that failed, it then fell back to the local hg code, but run through python.exe.
I'm not sure what it is about ef7119cd4965, but that started throwing up a
messagebox that python37.dll couldn't be loaded. (And indeed, python37 is not
in $PATH by default.) Invoking the local hg via the current python avoids that.
Boris Feld <boris.feld@octobus.net> [Mon, 17 Dec 2018 10:46:37 +0100] rev 40992
delta: ignore base whose chains already don't match expectations
If we know the existing chain does not match our criteria, there is no point
to build a delta to append. This is especially useful when dealing with a full
text much smaller than its parent. In that case, the parent chain is probably
already too large.
example affected manifest write
before: 1.421005s
after: 0.815520s (-42%)
Boris Feld <boris.feld@octobus.net> [Mon, 17 Dec 2018 10:42:19 +0100] rev 40991
delta: exclude base candidate much smaller than the target
If a revision's full text is that much bigger than a base candidate full text,
we no longer consider that candidate.
This solves a pathological case we encountered on a very specify repository.
It contains a long series of changesets with a very small manifest (one file)
co-existing with others changesets using a very large manifest.
Without this filtering, we ended up considering a large number of tiny full
snapshots as a potential base. It resulted in very large delta (the size of
the full text) and mercurial spending 99% of its time compressing these
deltas.
The timing of a commit moved from about 400s to about 10s (still slow, but not
ridiculously slow).
Boris Feld <boris.feld@octobus.net> [Mon, 17 Dec 2018 10:37:22 +0100] rev 40990
perfrevflogwrite: clear revlog cache between each write
We want to measure write time from a cold cache (similar to commit). So we
need to clear the cache to prevent computation from rev N-1 to interfere with
rev N.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 16 Dec 2018 17:42:45 -0500] rev 40989
py3: enable legacy stdio mode in exewrapper
This drops the test failure count from 166 to 117. The failures were typically
in the form of `hg serve -d` spawning but crashing immediately, leaving clients
with "bad http status" lines, connection refusals, and so forth. The underlying
message on the server side was either "OSError: [WinError 6] The handle is
invalid" or "OSError: [WinError 1] Incorrect function". Additionally, no output
was rendered if the pager was activated. Thanks to Yuya for diagnosing the
problem.
The failure count drops to 107 when PYTHONLEGACYWINDOWSSTDIO=1 is defined in the
environment. These failures seem to revolve around the dummyssh server process,
and dumbhttp.py. So I'll probably add that to the test runner.
One small regression here (only in py3) is that if hg.exe is already built, a
messagebox appears when building it again saying that python37.dll can't be
loaded. Python3 isn't in PATH by default, and setup.py tries running bare `hg`
first. But MSYS prepends '.' to PATH, so it runs the local hg, but can't find
the library. It falls back to the python used to invoke setup.py, so ultimately
it works. I'm not sure if it's better to strip '.' from PATH or just skip right
to `sys.executable hg` on Windows.
Also, something seems to be wrong with run-tests._usecorrectpython(). I
accidentially left off the 'PYTHON="py -3"' when building (thus making py2
stuff), and yet managed to invoke run-tests.py with "py -3". (And that only had
67 failures.)
Matt Harbison <matt_harbison@yahoo.com> [Sun, 16 Dec 2018 17:36:51 -0500] rev 40988
run-tests: alias hg to hg.exe on Windows
To enable legacy stdio mode on Windows, hg.exe needs to be updated. But before
that, we actually have to use it when running the tests. I *think* what was
happening before was when MSYS invoked `hg`, it looked at the shbang line and
ran python.exe found there. The test harness must be updating $PATH to include
the python used to launch it, and therefore it ran py3. As a side note, this
also fixed `py -3 run-tests.py` (without --local), which complained about the
space in the shbang line before this.
This should also help in WSL, because the explicit '.exe' is needed to invoke a
Windows app instead of invoking the Linux app.
I have no idea why this change capitalizes Lib in the tests, as it was
previously lowercase for both py2 and py3.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 16 Dec 2018 15:47:08 -0500] rev 40987
py3: conditionalize the python version in test-install.t
Boris Feld <boris.feld@octobus.net> [Sun, 16 Dec 2018 18:51:48 +0000] rev 40986
tests: followup on network related errors on Debian 9
First, update test-clonebundles.t which was failing since the introduction of
the `$EADDRNOTAVAIL$` common pattern.
Also update two tests from 8695fbe17f7c with a more compact condition line.
Differential Revision: https://phab.mercurial-scm.org/D5445
Yuya Nishihara <yuya@tcha.org> [Sun, 16 Dec 2018 17:55:08 +0900] rev 40985
py3: unimplement RevlogError.__str__()
On Python 2, str(exc) would crash if __str__() returned a unicode containing
non-ASCII characters.
Yuya Nishihara <yuya@tcha.org> [Sun, 16 Dec 2018 17:53:17 +0900] rev 40984
py3: use forcebytestr() to stringify hook exception
This fixes test-narrow-exchange.t.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Dec 2018 23:14:03 -0500] rev 40983
color: fix a documentation typo
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Dec 2018 22:31:54 -0500] rev 40982
py3: quote $PYTHON in test-patchbomb.t for Windows
I couldn't get the quoting right in the environment variable, so now it's a
function.
Georges Racinet <gracinet@anybox.fr> [Tue, 16 Oct 2018 19:58:27 +0200] rev 40981
rust-cpython: testing the bindings from Python
This is easier and more convincing than doing the same tests
from a Rust tests module.
Differential Revision: https://phab.mercurial-scm.org/D5437
Georges Racinet <gracinet@anybox.fr> [Thu, 06 Dec 2018 16:34:22 +0100] rev 40980
rust-cpython: build via HGWITHRUSTEXT=cpython
The existing behaviour, building the direct ffi bindings if
HGIWTHRUSTEXT is just set is unchanged, but if HGWITHRUSTEXT is
cpython, then the cpython bindings (aka mercurial/rustext.so) are
built.
Differential Revision: https://phab.mercurial-scm.org/D5436
Georges Racinet <gracinet@anybox.fr> [Thu, 06 Dec 2018 16:23:20 +0100] rev 40979
rust: better treatment of cargo/rustc errors
Differential Revision: https://phab.mercurial-scm.org/D5435
Georges Racinet <gracinet@anybox.fr> [Mon, 03 Dec 2018 06:52:17 +0100] rev 40978
rust-cpython: start cpython crate bindings
This changeset introduces the hg-cpython crate,
that compiles as a shared library holding a whole
Python package (mercurial.rustext), with only the empty
'ancestor' submodule for now.
Such bindings will be easier and safer to develop and maintain
that those of `hg-direct-ffi`.
They don't involve C code, only unsafe Rust that's mostly isolated
within the cpython crate.
The long-term goal would be to import the provided modules, such
as rustext.ancestor with mercurial.policy.importmod, same as
we already do with cext modules.
Differential Revision: https://phab.mercurial-scm.org/D5434
Georges Racinet <gracinet@anybox.fr> [Mon, 03 Dec 2018 06:54:19 +0100] rev 40977
rust-cpython: exclude hgcli from workspace
hgcli uses a specific rust-cpython commit by indygreg, of which
a PR has been derived which is not merged nor released yet.
But we can't use several versions of the sys-python2.7 crate in
a single workspace: it makes for a build error. Since hgcli does not at
the time being need anything from hg-core, whereas the upcoming hg-cpython
will. So for now we're moving hgcli aside, hoping we could base all of them
on the same version of rust-cpython again in the future.
Differential Revision: https://phab.mercurial-scm.org/D5433
Boris Feld <boris.feld@octobus.net> [Fri, 14 Dec 2018 17:25:41 +0100] rev 40976
sparse-revlog: protect C code against delta chain including nullrev
For unclear reasons, some repositories include nullrev (-1). Re-computing
delta for such repo remove nullrev from all chain, so some older versions have
been creating them.
This currently raise an IndexError with the new C code doing chain slicing as
it expect all item to be positive.
Both python and C code for reading delta chain preserve nullrev, and the Python
code for chain slicing handle the case fine. So we take the safe route and make
the new C code works fine in that case.
Boris Feld <boris.feld@octobus.net> [Fri, 14 Dec 2018 17:24:44 +0100] rev 40975
sparse-revlog: handle nullrev in index_get_length
The more generic index_get method handle nullrev fine, we apply the same logic
here.
Boris Feld <boris.feld@octobus.net> [Fri, 14 Dec 2018 17:23:41 +0100] rev 40974
sparse-revlog: handle nullrev in index_get_start
The more generic index_get method handle nullrev fine, we apply the same logic
here.
Boris Feld <boris.feld@octobus.net> [Fri, 14 Dec 2018 17:22:42 +0100] rev 40973
revlog: introduce a constant for nullrev in `revlog.c`
The value is important enough to be explicitly tracked.
Georges Racinet <gracinet@anybox.fr> [Fri, 30 Nov 2018 00:46:55 +0100] rev 40972
rust: translation of missingancestors
This is as direct as possible a translation of the ancestor.missingancestors
Python class in pure Rust. The goal for this changeset is to make it easy
to compare with the Python version.
We also add to Python tests the cases that helped us develop and debug
this implementation.
Some possible optimizations are marked along the way as TODO comments
Differential Revision: https://phab.mercurial-scm.org/D5416
Paul Morelle <paul.morelle@octobus.net> [Fri, 14 Dec 2018 18:15:19 +0100] rev 40971
contrib: provide a small script that draw performance plot
We have been using this script to look into the result of various runs of the
`hg perfrevlogwrite` command. It seems useful enough to be shared more widely.
Boris Feld <boris.feld@octobus.net> [Thu, 06 Dec 2018 10:39:05 +0100] rev 40970
delta: filter nullrev out first
When picking a potential candidate, we filter them on various criteria. The
"different from nullrev" criteria is very fast to compute and we should
process it first.
Boris Feld <boris.feld@octobus.net> [Thu, 06 Dec 2018 10:38:30 +0100] rev 40969
perf: report more of the higher range in perfrevlogwrite
Since the delta chain length is limited to 1000 revisions, we get a new
snapshot about every 1000 revisions. If we assume that the snapshot will be
most of the slowest revision, the current display (99% and max) are not very
precise in their area. We now include more information about this space in the
default report.