Gregory Szorc <gregory.szorc@gmail.com> [Thu, 10 Nov 2016 22:15:58 -0800] rev 30444
zstd: vendor python-zstandard 0.5.0
As the commit message for the previous changeset says, we wish
for zstd to be a 1st class citizen in Mercurial. To make that
happen, we need to enable Python to talk to the zstd C API. And
that requires bindings.
This commit vendors a copy of existing Python bindings. Why do we
need to vendor? As the commit message of the previous commit says,
relying on systems in the wild to have the bindings or zstd present
is a losing proposition. By distributing the zstd and bindings with
Mercurial, we significantly increase our chances that zstd will
work. Since zstd will deliver a better end-user experience by
achieving better performance, this benefits our users. Another
reason is that the Python bindings still aren't stable and the
API is somewhat fluid. While Mercurial could be coded to target
multiple versions of the Python bindings, it is safer to bundle
an explicit, known working version.
The added Python bindings are mostly a fully-featured interface
to the zstd C API. They allow one-shot operations, streaming,
reading and writing from objects implements the file object
protocol, dictionary compression, control over low-level compression
parameters, and more. The Python bindings work on Python 2.6,
2.7, and 3.3+ and have been tested on Linux and Windows. There are
CFFI bindings, but they are lacking compared to the C extension.
Upstream work will be needed before we can support zstd with PyPy.
But it will be possible.
The files added in this commit come from Git commit
e637c1b214d5f869cf8116c550dcae23ec13b677 from
https://github.com/indygreg/python-zstandard and are added without
modifications. Some files from the upstream repository have been
omitted, namely files related to continuous integration.
In the spirit of full disclosure, I'm the maintainer of the
"python-zstandard" project and have authored 100% of the code
added in this commit. Unfortunately, the Python bindings have
not been formally code reviewed by anyone. While I've tested
much of the code thoroughly (I even have tests that fuzz APIs),
there's a good chance there are bugs, memory leaks, not well
thought out APIs, etc. If someone wants to review the code and
send feedback to the GitHub project, it would be greatly
appreciated.
Despite my involvement with both projects, my opinions of code
style differ from Mercurial's. The code in this commit introduces
numerous code style violations in Mercurial's linters. So, the code
is excluded from most lints. However, some violations I agree with.
These have been added to the known violations ignore list for now.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 10 Nov 2016 21:45:29 -0800] rev 30443
zstd: vendor zstd 1.1.1
zstd is a new compression format and it is awesome, yielding
higher compression ratios and significantly faster compression
and decompression operations compared to zlib (our current
compression engine of choice) across the board.
We want zstd to be a 1st class citizen in Mercurial and to eventually
be the preferred compression format for various operations.
This patch starts the formal process of supporting zstd by vendoring
a copy of zstd. Why do we need to vendor zstd? Good question.
First, zstd is relatively new and not widely available yet. If we
didn't vendor zstd or distribute it with Mercurial, most users likely
wouldn't have zstd installed or even available to install. What good
is a feature if you can't use it? Vendoring and distributing the zstd
sources gives us the highest liklihood that zstd will be available to
Mercurial installs.
Second, the Python bindings to zstd (which will be vendored in a
separate changeset) make use of zstd APIs that are only available
via static linking. One reason they are only available via static
linking is that they are unstable and could change at any time.
While it might be possible for the Python bindings to attempt to
talk to different versions of the zstd C library, the safest thing to
do is link against a specific, known-working version of zstd. This
is why the Python zstd bindings themselves vendor zstd and why we
must as well. This also explains why the added files are in a
"python-zstandard" directory.
The added files are from the 1.1.1 release of zstd (Git commit
4c0b44f8ced84c4c8edfa07b564d31e4fa3e8885 from
https://github.com/facebook/zstd) and are added without modifications.
Not all files from the zstd "distribution" have been added. Notably
missing are files to support interacting with "legacy," pre-1.0
versions of zstd. The decision of which files to include is made by
the upstream python-zstandard project (which I'm the author of). The
files in this commit are a snapshot of the files from the 0.5.0
release of that project, Git commit
e637c1b214d5f869cf8116c550dcae23ec13b677 from
https://github.com/indygreg/python-zstandard.
Mads Kiilerich <madski@unity3d.com> [Tue, 15 Nov 2016 21:56:49 +0100] rev 30442
bdiff: give slight preference to removing trailing lines
[This change could be folded into the previous changeset to minimize the repo
churn ...]
Similar to the previous change, introduce an exception to the general
preference for matches in the middle of bdiff ranges: If the best match on the
B side starts at the beginning of the bdiff range, don't aim for the
middle-most A side match but for the earliest.
New (later) matches on the A side will only be considered better if the
corresponding match on the B side *not* is at the beginning of the range.
Thus, if the best (middle-most) match on the B side turns out to be at the
beginning of the range, the earliest match on the A side will be used.
The bundle size for 4.0 (hg bundle --base null -r 4.0 x.hg) happens to go from
22807275 to 22808120 bytes - a 0.004% increase.
Mads Kiilerich <madski@unity3d.com> [Tue, 15 Nov 2016 21:56:49 +0100] rev 30441
bdiff: give slight preference to appending lines
[This change could be folded into the previous changeset to minimize the repo
churn ...]
The general preference to matches in the middle of bdiff ranges helps getting
balanced recursion and efficient computation. But, as previous changes have
shown, it might also give diffs that seems "obviously wrong".
To mitigate that: If the best match on the A side starts at the beginning of
the bdiff range, don't aim for the middle-most B side match but for the
earliest.
This will make the matches balanced (by both sides being "early") even though
the bisection will be less balanced. Still, this case only apply if the *best*
and middle-most match was fully unbalanced on the A side. Each recursion will
thus even in this worst case reduce the problem significantly and we are not
re-introducing the problem that was fixed in f1ca249696ed.
The bundle size for 4.0 (hg bundle --base null -r 4.0 x.hg) happens to go from
22806817 to 22807275 bytes - a 0.002% increase.
This make the recent test-bdiff.py changes give a more pretty output ... but
they no longer show that the recursion is around middle matches (because it in
these cases isn't).
Mads Kiilerich <madski@unity3d.com> [Tue, 08 Nov 2016 18:37:33 +0100] rev 30440
bdiff: give slight preference to longest matches in the middle of the B side
We already have a slight preference for matches close to the middle on the A
side. Now, do the same on the B side.
j is iterating the b range backwards and we thus accept a new j if the previous
match was in the upper half.
This makes the test-bhalf diff "correct". It obviously also gives more
preference to balanced recursion than to appending to sequences. That is kind
of correct, but will also unfortunately make some bundles bigger. No doubt, we
can also create examples where it will make them smaller ...
The bundle size for 4.0 (hg bundle --base null -r 4.0 x.hg) happens to go from
22803824 to 22806817 bytes - an 0.01% increase.
Mads Kiilerich <madski@unity3d.com> [Tue, 08 Nov 2016 18:37:33 +0100] rev 30439
bdiff: rearrange the "better longest match" code
This is primarily to make the code more managable and prepare for later
changes.
More specific assignments might also be slightly faster, even thought it also
might generate a bit more code.
Mads Kiilerich <madski@unity3d.com> [Tue, 08 Nov 2016 18:37:33 +0100] rev 30438
bdiff: adjust criteria for getting optimal longest match in the A side middle
We prefer matches closer to the middle to balance recursion, as introduced in
f1ca249696ed.
For ranges with uneven length, matches starting exactly in the middle should
have preference. That will be optimal for matches of length 1. We will thus
accept equality in the half check.
For ranges with even length, half was ceil'ed when calculated but we got the
preference for low matches from the 'less than half' check. To get the same
result as before when we also accept equality, floor it. Without that,
test-annotate.t would show some different (still correct but less optimal)
results.
This will change the heuristics. Tests shows a slightly different output - and
sometimes slightly smaller bundles.
The bundle size for 4.0 (hg bundle --base null -r 4.0 x.hg) happens to go from
22804885 to 22803824 bytes - an 0.005% reduction.
Mads Kiilerich <madski@unity3d.com> [Tue, 08 Nov 2016 18:37:33 +0100] rev 30437
tests: explore some bdiff cases
Mads Kiilerich <madski@unity3d.com> [Tue, 15 Nov 2016 21:56:49 +0100] rev 30436
tests: make test-bdiff.py easier to maintain
Add more stdout logging to help navigate the .out file.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 17 Nov 2016 08:52:52 -0800] rev 30435
perf: unbust perfbdiff --alldata
This broke in f84fc6a92817 due to a refactored manifest API.
The fix is a bit hacky - perfbdiff doesn't yet support tree manifests
for example. But it gets the job done.
A test has been added for --alldata so this doesn't happen again.
Yuya Nishihara <yuya@tcha.org> [Thu, 17 Nov 2016 20:57:09 +0900] rev 30434
worker: discard waited pid by anyone who noticed it first
This makes sure all waited pids are removed before calling killworkers()
even if waitpid()-pids.discard() sequence is interrupted by another SIGCHLD.
Yuya Nishihara <yuya@tcha.org> [Thu, 17 Nov 2016 21:08:58 +0900] rev 30433
worker: kill workers after all zombie processes are reaped
Since we now wait child processes in non-blocking way (changed by 7bc25549e084
and e8fb03cfbbde), we don't have to kill them in the middle of the waitpid()
loop. This change will help solving a possible race of waitpid()-pids.discard()
sequence and another SIGCHLD.
waitforworkers() is called by cleanup(), in which case we do killworkers()
beforehand so we can remove killworkers() from waitforworkers().
Yuya Nishihara <yuya@tcha.org> [Thu, 17 Nov 2016 20:44:05 +0900] rev 30432
worker: make sure killworkers() never be interrupted by another SIGCHLD
killworkers() iterates over pids, which can be updated by SIGCHLD handler.
So we should either copy pids or prevent killworkers() from being interrupted
by SIGCHLD. I chose the latter as it is simpler and can make pids handling
more consistent.
This fixes a possible "set changed size during iteration" error at
killworkers() before cleanup().
Yuya Nishihara <yuya@tcha.org> [Thu, 17 Nov 2016 21:43:01 +0900] rev 30431
worker: fix missed break on successful waitpid()
Follow-up for 5069a8a40b1b.
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:49:42 -0500] rev 30430
filterpyflakes: dramatically simplify the entire thing by blacklisting
We've only got one kind of pyflakes failure left in our codebase, so
it's time to switch over to a blacklist-based checking scheme. I've
left in the filtering of two undefined names for now out of paranoia,
but those can probably go too.
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:07:24 -0500] rev 30429
run-tests: forward Python USER_BASE from site (issue5425)
We do this so that any linters installed via pip install --user don't
break. See https://docs.python.org/2/library/site.html#site.USER_BASE
for a description of what this nonsense is all about.
An alternative would be to not set HOME, but that'll cause other
problems (see issue2707), or to forward every single path entry from
sys.path in PYTHONPATH (which seems sketchy in its own way).
Jun Wu <quark@fb.com> [Tue, 15 Nov 2016 20:25:51 +0000] rev 30428
util: improve iterfile so it chooses code path wisely
We have performance concerns on "iterfile" as it is 4X slower on normal
files. While modern systems have the nice property that reading a "fast"
(on-disk) file cannot be interrupted and should be made use of.
This patch dumps the related knowledge in comments. And "iterfile" chooses
code paths wisely:
1. If it's CPython 3, or PyPY, use the fast path.
2. If fp is a normal file, use the fast path.
3. If fp is not a normal file and CPython version >= 2.7.4, use the same
workaround (4x slower) as before.
4. If fp is not a normal file and CPython version < 2.7.4, use another
workaround (2x slower but may block longer then necessary) which
basically re-invents the buffer + readline logic in Python.
This will give us good confidence on both correctness and performance
dealing with EINTR in iterfile(fp) for all known supported Python versions.
Augie Fackler <augie@google.com> [Wed, 16 Nov 2016 23:29:28 -0500] rev 30427
merge with stable
Jun Wu <quark@fb.com> [Sat, 12 Nov 2016 03:06:07 +0000] rev 30426
worker: stop using a separate thread waiting for children
Now that we have a SIGCHLD hander, and it could get executed when waiting
for I/O. It's no longer necessary to have a separated waitpid thread. So
just remove it.
Jun Wu <quark@fb.com> [Sat, 12 Nov 2016 03:07:22 +0000] rev 30425
worker: add a SIGCHLD handler to collect worker immediately
As planned by previous patches, add a SIGCHLD handler to get notifications
about worker exits, and deals with worker failure immediately.
Note that the SIGCHLD handler gets unregistered before killworkers(), so
SIGCHLD won't interrupt "killworkers" - making it harder to send kill
signals to waited processes.
Jun Wu <quark@fb.com> [Tue, 15 Nov 2016 02:12:16 +0000] rev 30424
worker: make waitforworkers reentrant
We are going to use it in the SIGCHLD handler. The handler will be executed
in the main thread with the non-blocking version of waitpid, while the
waitforworkers thread runs the blocking version. It's possible that one of
them collects a worker and makes the other error out (no child to wait).
This patch handles these errors: ECHILD is ignored. EINTR needs a retry.
The "pids" set is designed to be only modifiable by "waitforworkers". And we
only remove items after a successful waitpid. Since a child process can only
be "waitpid"-ed once. It's guaranteed that "pids.remove(p)" won't be called
with duplicated "p"s. And once a "p" is removed from "pids", that "p" does
not need to be killed or waited any more.
Jun Wu <quark@fb.com> [Tue, 15 Nov 2016 02:10:40 +0000] rev 30423
worker: change "pids" to a set
There is no need to keep any order of the "pids" array. A set is more
efficient for the "remove" operation. And the following patch will use that.
Jun Wu <quark@fb.com> [Thu, 28 Jul 2016 20:57:07 +0100] rev 30422
worker: allow waitforworkers to be non-blocking
This patch adds a boolean flag to waitforworkers and makes it non-blocking
if set to True.
This is to make it possible that we can reap our workers while keep other
unrelated children untouched, after receiving SIGCHLD.
Jun Wu <quark@fb.com> [Thu, 28 Jul 2016 20:51:20 +0100] rev 30421
worker: wait worker pid explicitly
Before this patch, waitforworkers uses os.wait() to collect child workers, and
only wait len(pids) processes. This can have serious issues if other code
spawns new processes and does not reap them: 1. worker.py may get wrong exit
code and kill innocent workers. 2. worker.py may continue without waiting for
all workers to complete.
This patch fixes the issue by using waitpid to wait worker pid explicitly.
However, this patch introduces a new issue: worker failure may not be handled
immediately. The issue will be addressed in next patches.
Jun Wu <quark@fb.com> [Thu, 28 Jul 2016 20:49:57 +0100] rev 30420
worker: move killworkers and waitforworkers up
We need to use them in the SIGCHLD handler and SIGCHLD handler should be
installed before fork.
Jun Wu <quark@fb.com> [Fri, 11 Nov 2016 21:11:17 +0000] rev 30419
osutil: implement setprocname to set process title for some platforms
This patch adds a simple setprocname method to osutil. The operation is not
defined by any standard and is platform-specific, the current implementation
tries to cover some major platforms (ex. Linux, OS X, FreeBSD) that is
relatively easy to support. Other platforms (Windows [4], other BSDs, ...)
can be added in the future.
The current implementation supports two methods to change process title:
a. setproctitle if available (works in FreeBSD).
b. rewrite argv in place (works in Linux [1] and Mac OS X). [2] [3]
[1]: Linux has "prctl(PR_SET_NAME, ...)" but 1) it has 16-byte limit, which
is too small; 2) it is not quite equivalent to what we want - it changes
"/proc/self/comm", not "/proc/self/cmdline" - "comm" change won't show up
in "ps" output unless "-o comm" is used.
[2]: The implementation does not rewrite the **environ buffer like some
other implementations do, just to make the code simpler and safer. However,
this also means the buffer size we can rewrite is significantly shorter. If
we are really greedy and want the "environ" space, we can change the
implementation later.
[3]: It requires a CPython private API: Py_GetArgcArgv to get the original
argv. Unfortunately Python 3 makes a copy of argv and returns the wchar_t
version, so it is not supported for now. (if we really want to, we could
count backwards from "char **environ", given known argc and argv, not sure
if that's a good idea - probably not)
[4]: The feature is aimed to make it easier for forked command server
processes to show what they are doing. Since Windows does not support
fork(), despite it's a major platform, its support is not added in this
patch.
Jun Wu <quark@fb.com> [Fri, 11 Nov 2016 20:45:40 +0000] rev 30418
setup: test setproctitle before building osutil
We are going to use setproctitle (provided by FreeBSD) if it's available in
the next patch. Therefore provide a macro to give some clues to the C
pre-processor so it could choose code path wisely.
Henning Schild <henning@hennsch.de> [Sat, 12 Nov 2016 13:36:17 +0100] rev 30417
patch: remove unused git parameter from patch.diffstat()
Since 628a4a9e411d the parameter is not used anymore.
Philippe Pepiot <philippe.pepiot@logilab.fr> [Thu, 29 Sep 2016 10:16:34 +0200] rev 30416
perf: add asv benchmarks
Airspeed velocity (ASV) is a python framework for benchmarking Python packages
over their lifetime. The results are displayed in an interactive web frontend.
Add ASV benchmarks for mercurial that use contrib/perf.py extension that could
be run against multiple reference repositories.
The benchmark suite now includes revsets from contrib/base-revsets.txt with
variants, perftags, perfstatus, perfmanifest and perfheads.
Installation requires asv>=0.2, python-hglib and virtualenv
This is part of PerformanceTrackingSuitePlan
https://www.mercurial-scm.org/wiki/PerformanceTrackingSuitePlan
Philippe Pepiot <philippe.pepiot@logilab.fr> [Tue, 15 Nov 2016 16:10:57 +0100] rev 30415
perf: omit copying ui and redirect to ferr if buffer API is in use
This allow to get the output of contrib/perf.py commands using the
ui.pushbuffer() API.
Durham Goode <durham@fb.com> [Mon, 14 Nov 2016 15:24:07 -0800] rev 30414
manifest: change treemanifestctx to construct subtrees from the manifestlog
Previously, treemanifestctx would directly construct its subtrees. By making it
get the subtrees through manifestlog.get() we consolidate all treemanifestctx
creation into manifestlog.get() and therefore extensions that need to wrap
manifestctx creation (like narrow-hg) can intercept manifestctxs at that single
place.
This also means fetching subtrees will take advantage of the manifestlog ctx
cache now, which it did not before.
Durham Goode <durham@fb.com> [Mon, 14 Nov 2016 15:17:27 -0800] rev 30413
manifest: make revlog verification optional
This patches adds an parameter to manifestlog.get() to disable hash checking.
This will be used in an upcoming patch to support treemanifestctx reading
sub-trees without loading them from the revlog. (This is already supported but
does not go through the manifestlog.get() code path)
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 10 Nov 2016 09:45:42 -0800] rev 30412
debugcommands: move debugbuilddag
And we drop some now unused imports from commands.py.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 17 Aug 2016 21:07:38 -0700] rev 30411
debugcommands: introduce standalone module for debug commands
commands.py is our largest .py file by nearly 2x. Debug commands live
in a world of their own. So let's extract them to their own module.
We start with "debugancestor."
We currently reuse the commands table with commands.py and have a hack
in dispatch.py for loading debugcommands.py. In the future, we could
potentially use a separate commands table and avoid the import of
debugcommands.py.
Jun Wu <quark@fb.com> [Mon, 14 Nov 2016 23:17:15 +0000] rev 30410
convert: migrate to util.iterfile
Jun Wu <quark@fb.com> [Mon, 14 Nov 2016 23:16:05 +0000] rev 30409
match: migrate to util.iterfile
Jun Wu <quark@fb.com> [Mon, 14 Nov 2016 23:15:01 +0000] rev 30408
store: migrate to util.iterfile
Jun Wu <quark@fb.com> [Mon, 14 Nov 2016 23:14:06 +0000] rev 30407
patch: migrate to util.iterfile
Jun Wu <quark@fb.com> [Mon, 14 Nov 2016 23:12:11 +0000] rev 30406
worker: migrate to util.iterfile
Jun Wu <quark@fb.com> [Mon, 14 Nov 2016 23:32:54 +0000] rev 30405
util: add iterfile to workaround a fileobj.__iter__ issue with EINTR
The fileobj.__iter__ implementation in Python 2.7.12 (hg changeset
45d4cea97b04) is buggy: it cannot handle EINTR correctly.
In Objects/fileobject.c:
size_t Py_UniversalNewlineFread(....) {
....
if (!f->f_univ_newline)
return fread(buf, 1, n, stream);
....
}
According to the "fread" man page:
If an error occurs, or the end of the file is reached, the return value
is a short item count (or zero).
Therefore it's possible for "fread" (and "Py_UniversalNewlineFread") to
return a positive value while errno is set to EINTR and ferror(stream)
changes from zero to non-zero.
There are multiple "Py_UniversalNewlineFread": "file_read", "file_readinto",
"file_readlines", "readahead". While the first 3 have code to handle the
EINTR case, the last one "readahead" doesn't:
static int readahead(PyFileObject *f, Py_ssize_t bufsize) {
....
chunksize = Py_UniversalNewlineFread(
f->f_buf, bufsize, f->f_fp, (PyObject *)f);
....
if (chunksize == 0) {
if (ferror(f->f_fp)) {
PyErr_SetFromErrno(PyExc_IOError);
....
}
}
....
}
It means "readahead" could ignore EINTR, if "Py_UniversalNewlineFread"
returns a non-zero value. And at the next time "readahead" got executed, if
"Py_UniversalNewlineFread" returns 0, "readahead" would raise a Python error
without a incorrect errno - could be 0 - thus "IOError: [Errno 0] Error".
The only user of "readahead" is "readahead_get_line_skip".
The only user of "readahead_get_line_skip" is "file_iternext", aka.
"fileobj.__iter__", which should be avoided.
There are multiple places where the pattern "for x in fp" is used. This
patch adds a "iterfile" method in "util.py" so we can migrate our code from
"for x in fp" to "fox x in util.iterfile(fp)".
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:37:18 -0500] rev 30404
filterpyflakes: whitelist listcomp aliasing checking
The test change is because of how filterpyflakes is organized - a line
number changed.
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:35:54 -0500] rev 30403
verify: avoid shadowing two variables with a list comprehension
The variable names are clearly worse now, but since we're really just
transposing key and value I'm not too worried about the clarity loss.
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:35:10 -0500] rev 30402
revset: avoid shadowing a variable with a list comprehension
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:34:43 -0500] rev 30401
revlog: avoid shadowing several variables using list comprehensions
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:33:41 -0500] rev 30400
minirst: avoid shadowing a variable in a list comprehension
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:33:23 -0500] rev 30399
hbisect: avoid shadowing a variable in a list comprehension
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:33:07 -0500] rev 30398
filemerge: avoid shadowing a variable in a list comprehension
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:32:51 -0500] rev 30397
color: avoid shadowing a variable inside a list comprehension
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 16:32:38 -0500] rev 30396
memory: avoid shadowing variables inside a list comprehension
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:15:41 -0800] rev 30395
shelve: move shelve-finishing logic to a separate function
With future obs-based shelve, finishing shelve will be different
from just aborting a transaction and I would like to keep both
variants of this functionality in a separate function.
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:20:28 -0800] rev 30394
shelve: move unknown files handling to a separate function
This change has nothing to do with future obsshelve introduction,
it is done just for readability purposes.
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:07:20 -0800] rev 30393
shelve: move actual created commit shelving to a separate function
Currently, this code does not have any branching, it just bundles
a commit and saves a patch file. Later, obsolescence-based shelve
will be added, so this code will also create some obsmarkers and
will be one of the few places where obsshelve will be different
from traditional shelve.
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:33:01 -0800] rev 30392
shelve: move 'nothing changed' messaging to a separate function
This has nothing to do with the future obsshelve implementation, I just
thought that moving this messaging to a separate function will improve
shelve code readability.
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:26:31 -0800] rev 30391
shelve: move commitfunc creation to a separate function
Special commitfuncs are created as closures at least twice in shelve's
code and one time special commitfunc is used within another closure.
They all serve very specific purposes like temporarily tweak some
configuration or enable editor, etc. This is not immediately important
to someone reading shelve code, so I think moving this logic to a separate
function is a good idea.
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:24:07 -0800] rev 30390
shelve: move mutableancestors to not be a closure
There's no value in it being a closure and everyone who tries to read
the outer function code will be distracted by it. IMO moving it out
significantly improves readability, especially given how clear it is
what mutableancestors function does from its name.
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:22:55 -0800] rev 30389
shelve: move shelve name generation to a separate function
This has nothing to do with future obsshelve introduction, done just
for readability purposes.
Kostia Balytskyi <ikostia@fb.com> [Thu, 10 Nov 2016 03:07:20 -0800] rev 30388
shelve: move possible shelve file extensions to a single place
This and a couple of following patches are a preparation to
implementing obsolescense-enabled shelve which was discussed
on a Sprint. If this refactoring is not done, shelve is going
to look even more hackish than now.
This particular commit introduces a slight behavior change. Previously,
if only .hg/shelve/name.patch file exists, but .hg/name.hg does not,
'hg shelve -d name' would fail saying "shelve not found". Now deletion
will only fail if .patch file does not exist (since .patch is used
as an indicator of an existing shelve). Other shelve files being absent
are skipped silently to accommodate for future introduction of obs-based
shelve, which will mean that for some shelves .hg and .patch files exist,
while for others .hg and .oshelve.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30387
manifest: delete manifest.manifest class
Now that nothing uses the primary manifest class, we can delete it.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30386
localrepo: delete localrepo.manifest
Now that nothing uses normal manifests, we can delete localrepo.manifest.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30385
manifest: remove last uses of repo.manifest
Now that all the functionality has been moved to manifestlog/manifestrevlog/etc,
we can finally change all the uses of repo.manifest to use the new versions. A
future diff will then delete repo.manifest.
One additional change in this commit is to change repo.manifestlog to be a
@storecache property instead of @property. This is required by some uses of
repo.manifest require that it be settable (contrib/perf.py and the static http
server). We can't do this in a prior change because we can't use @storecache on
this until repo.manifest is no longer used anywhere.
Durham Goode <durham@fb.com> [Fri, 11 Nov 2016 01:20:13 -0800] rev 30384
manifest: add unionmanifestlog support
As part of deprecating manifest, we need to make the union repo support
manifestlog.
Durham Goode <durham@fb.com> [Fri, 11 Nov 2016 01:15:59 -0800] rev 30383
manifest: add bundlemanifestlog support
As part of deprecating manifest.manifest we need to make bundlerepo support
manifestlog.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30382
manifest: make manifestlog use it's own cache
As we start to make manifestlog the primary manifest source, the dependency on
manifest.manifest will cause circular dependency problems. Let's break this
dependency by making manifestlog use it's own cache. In a near future patch we
will remove the previous manifest cache so we're not duplicating it.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30381
manifest: delete unused dirlog and _newmanifest functions
As part of migrating all manifest functionality out of manifest.manifest, let's
migrate a couple spots off of manifest.dirlog() to use the revlog specific
accessor. Then we can delete manifest.dirlog() and other unused functions.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30380
manifest: move clearcaches to manifestlog
This is part of removing all functionality from manifest.manifest so we can
delete the class entirely.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30379
manifest: remove usages of manifest.read
Now that the two manifestctx implementations have working read() functions,
let's remove the existing uses of manifest.read and drop the function.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:13:19 -0800] rev 30378
manifest: remove dependency on manifestrevlog being able to create trees
A future patch will be removing the read() function from the manifest class.
Since manifestrevlog currently depends on the read function that manifest
implements (as a derived class), we need to break the dependency from
manifestrevlog to read(). We do this by adding an argument to
manifestrevlog.write() which provides it with the ability to read a manifest.
This is good in general because it further separates revlog as the storage
format from the actual inmemory data structure implementation.
Xidorn Quan <me@upsuper.org> [Fri, 11 Nov 2016 13:06:05 +1100] rev 30377
color: show mode warning based on ui.formatted
ui.interactive is only for input and ui.formatted is for output.
Augie Fackler <augie@google.com> [Thu, 10 Nov 2016 15:14:05 -0500] rev 30376
protocol: drop unused import of zlib
Something weird is happening that breaks pyflakes installed via 'pip
install --user'. I haven't had a chance to finish debugging this, but
this at least fixes the build.
Yuya Nishihara <yuya@tcha.org> [Tue, 08 Nov 2016 22:41:45 +0900] rev 30375
hook: lower inflated use of sys.__stdout__ and __stderr__
They were introduced at 9f76df0edb7d, where sys.stdout could be replaced by
sys.stderr. After that, we've changed the way of stdout redirection by
afccc64eea73, so we no longer need to reference the original __stdout__ and
__stderr__ objects.
Let's move away from using __std*__ objects so we can simply wrap sys.std*
objects for Python 3 porting.
Yuya Nishihara <yuya@tcha.org> [Tue, 08 Nov 2016 22:22:22 +0900] rev 30374
hook: flush stdout before restoring stderr redirection
There was a similar issue to 8b011ededfb2. If an in-process hook writes
to stdout, the data may be buffered. In which case, stdout must be flushed
before restoring its file descriptor. Otherwise, remaining data would be sent
over the ssh wire and corrupts the protocol.
Note that this is a different redirection from the one I've just removed.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 22:39:59 +0900] rev 30373
hook: do not redirect stdout/err/in to ui while running in-process hooks (BC)
It was introduced by a59058fd074a to address command-server issues. After
that, I've made a complete fix by 69f86b937035, so we don't need to replace
sys.stdio objects to protect the IPC channels.
This change means we no longer see data written to sys.stdout/err by an
in-process hook on command server. I think that's okay because the canonical
way is to use ui functions and in-process hooks should respect the Mercurial
API.
This will help Python 3 porting, where sys.stdout is TextIO but ui.fout is
BytesIO.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:21:15 -0800] rev 30372
merge: change modified indicator to be 20 bytes
Previously we indicated that the .hgsubstate file was dirty by adding a '+' to
the end of its hash in the wctx manifest. This made is complicated to have new
manifest implementations that rely on the node length being fixed.
In previous patches we added added and modified node placeholders, so let's use
those to indicate dirty here as well. It doesn't look like anything ever
depended on this '+' (aside from it being different to the parent), so nothing
else needed to change here.
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:19:16 -0800] rev 30371
dirstate: change added/modified placeholder hash length to 20 bytes
Previously the added/modified placeholder hash for manifests generated from the
dirstate was a 21byte long string consisting of the p1 file hash plus a single
character to indicate an add or a modify. Normal hashes are only 20 bytes long.
This makes it complicated to implement more efficient manifest implementations
which rely on the hashes being fixed length.
Let's change this hash to just be 20 bytes long, and rely on the astronomical
improbability of an actual hash being these 20 bytes (just like we rely on no
hash every being the nullid).
This changes the possible behavior slightly in that the hash for all
added/modified entries in the dirstate manifest will now be the same (so simple
node comparisons would say they are equal), but we should never be doing simple
node comparisons on these nodes even with the old hashes, because they did not
accurately represent the content (i.e. two files based off the same p1 file
node, with different working copy contents would have the same hash (even with
the appended character) in the old scheme too, so we couldn't depend on the
hashes period).
Durham Goode <durham@fb.com> [Thu, 10 Nov 2016 02:17:22 -0800] rev 30370
dirstate: change placeholder hash length to 20 bytes
Previously the new-node placeholder hash for manifests generated from the
dirstate was a 21byte long string of "!" characters. Normal hashes are only 20
bytes long. This makes it complicated to implement more efficient manifest
implementations which rely on the hashes being fixed length.
Let's change this hash to just be 20 bytes long, and rely on the astronomical
improbability of an actual hash being 20 "!" bytes in a row (just like we rely
on no hash ever being the nullid).
A future diff will do this for added and modified dirstate markers as well, so
we're putting the new newnodeid in node.py so there's a common place for these
placeholders.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:57:54 -0800] rev 30369
util: remove compressorobj API from compression engines
All callers have been replaced with "compressstream." It is quite
low-level and redundant with "compressstream." So eliminate it.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:54:35 -0800] rev 30368
hgweb: use compression engine API for zlib compression
More low-level compression code elimination because we now have nice
APIs.
This patch also demonstrates why we needed and implemented the
"level" option on the "compressstream" API.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:46:37 -0800] rev 30367
bundle2: use compressstream compression engine API
Compression engines now have an API for compressing a stream of
chunks. Switch to it and make low-level compression code disappear.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:57:07 -0800] rev 30366
util: add a stream compression API to compression engines
It is a common pattern throughout the code to perform compression
on an iterator of chunks, yielding an iterator of compressed chunks.
Let's formalize that as part of the compression engine API.
The zlib and bzip2 implementations allow an optional "level" option
to control the compression level. The default values are the same as
what the Python modules use. This option will be used in subsequent
patches.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:39:08 -0800] rev 30365
util: remove decompressors dict (API)
All in-tree consumers are now using the compengines registrar.
Extensions should switch to it as well.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:38:13 -0800] rev 30364
changegroup: use compression engines API
The new API doesn't have the equivalence for None and 'UN' so we
introduce code to use 'UN' explicitly.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:36:48 -0800] rev 30363
bundle2: use compression engines API to obtain decompressor
Like the recent change for the compressor side, this too is
relatively straightforward. We now store a compression engine
on the instance instead of a low-level decompressor. Again, this
will allow us to easily transition to different compression engine
APIs when they are implemented.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:34:51 -0800] rev 30362
util: remove compressors dict (API)
We no longer have any in-tree consumers of this object. Use
util.compengines instead.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:35:43 -0800] rev 30361
bundle2: use new compression engine API for compression
Now that we have a new API to define compression engines, let's put it
to use!
The new code stores a reference to the compression engine instead of
a low-level compressor object. This will allow us to more easily
transition to different APIs on the compression engine interface
once we implement them.
As part of this, we change the registration in bundletypes to use 'UN'
instead of None. Previously, util.compressors had the no-op compressor
registered under both the 'UN' and None keys. Since we're switching to
a new API, I don't see the point in carrying this dual registration
forward.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 07 Nov 2016 18:31:39 -0800] rev 30360
util: create new abstraction for compression engines
Currently, util.py has "compressors" and "decompressors" dicts
mapping compression algorithms to callables returning objects that
perform well-defined operations. In addition, revlog.py has code
for calling into a compressor or decompressor explicitly. And, there
is code in the wire protocol for performing zlib compression.
The 3rd party lz4revlog extension has demonstrated the utility of
supporting alternative compression formats for revlog storage. But
it stops short of supporting lz4 for bundles and the wire protocol.
There are also plans to support zstd as a general compression
replacement.
So, there appears to be a market for a unified API for registering
compression engines. This commit starts the process of establishing
one.
This commit establishes a base class/interface for defining
compression engines and how they will be used. A collection class
to hold references to registered compression engines has also been
introduced.
The built-in zlib, bz2, truncated bz2, and no-op compression engines
are registered with a singleton instance of the collection class.
The compression engine API will change once consumers are ported
to the new API and some common patterns can be simplified at the
engine API level. So don't get too attached to the API...
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 09:25:39 -0400] rev 30359
config: mark parser regexes as bytes explicitly
r-strings are not transformed into bytes by our source transformer magic.
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 09:17:49 -0400] rev 30358
ui: explicitly open config files in binary mode
This has been working mostly accidentally, but now it works explicitly.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 09 Nov 2016 16:04:44 -0800] rev 30357
help: fix double word usage
"most" was used twice.
(I fixed a grammar error before timeless spotted it!)
Jun Wu <quark@fb.com> [Wed, 09 Nov 2016 22:08:30 +0000] rev 30356
setup: move cffi stuff to mercurial/cffi
This patch moves all setup*cffi stuff to mercurial/cffi to make the root
directory cleaner. The idea was from mpm [1]:
> It seems like we could have a fair amount of cffi definitions, and
> cluttering the root directory (or mercurial/) with them is probably not
> a great long-term solution. We could probably add a cffi/ directory
> under mercurial/ to parallel pure/.
[1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-July/086442.html
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30355
manifest: remove manifest.add and add memmfctx.write
This removes one more dependency on the manifest class by moving the write
functionality onto the memmanifestctx classes and changing the one consumer to
use the new API.
By moving the write path to a manifestctx, we now give the individual manifests
control over how they're read and serialized. This will be useful in developing
new manifest formats and storage systems.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30354
context: add manifestctx property on changectx
This allows us to access the manifestctx for a given commit. This will be used
in a later patch to be able to copy the manifestctx when we want to make a new
commit.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30353
manifest: add copy to mfctx classes
This adds copy functionality to the manifestctx classes. This will be used in an
upcoming diff to copy a manifestctx during commit so we can modify the manifest
before committing.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30352
manifest: introduce memmanifestctx and memtreemanifestctx
This introduces two new classes to represent in-memory manifest instances.
Similar to memchangectx, this lets us prepare a manifest in memory, then in a
future patch we will add the apis that can commit this in memory structure.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30351
manifestctx: add _revlog() function
The `self._repo.manifestlog._revlog` code is getting copy and pasted a lot in
manifestctx. Let's make it a function so it can be reused. This will make future
patches cleaner too.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30350
manifest: remove manifest.find
As part of removing dependencies on manifest, this drops the find function and
fixes up the two existing callers to use the equivalent apis on manifestctx.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30349
changegroup: remove remaining uses of repo.manifest
The remaining uses of repo.manifest in the changegroup module are treating the
manifest exclusively as a revlog, so let's replace them with instances of the
revlog directly.
This is part of dropping all dependencies on repo.manifest in favor of
repo.manifestlog.