Anton Shestakov <av6@dwimlabs.net> [Sat, 04 Feb 2017 20:29:13 +0800] rev 30864
debian: update mailing list address
Kevin Bullock <kbullock@ringworld.org> [Wed, 01 Feb 2017 10:19:49 -0600] rev 30863
Added signature for changeset e1526da1e6d8
Kevin Bullock <kbullock@ringworld.org> [Wed, 01 Feb 2017 10:18:59 -0600] rev 30862
Added tag 4.1 for changeset e1526da1e6d8
Kevin Bullock <kbullock+mercurial@ringworld.org> [Wed, 01 Feb 2017 10:15:10 -0600] rev 30861
merge with i18n
Wagner Bruna <wbruna@softwareexpress.com.br> [Wed, 01 Feb 2017 08:47:11 -0200] rev 30860
i18n-pt_BR: synchronized with dfc6663f97ca
Mads Kiilerich <mads@kiilerich.com> [Wed, 01 Feb 2017 02:10:30 +0100] rev 30859
merge: more safe detection of criss cross merge conflict between dm and r
41f6af50c0d8 introduced handling of a crash in this case. A review comment
suggested that it was not entirely obvious that a 'dm' always would have a 'r'
for the source file.
To mitigate that risk, make the code more conservative and make less
assumptions.
Augie Fackler <augie@google.com> [Mon, 30 Jan 2017 18:03:17 -0500] rev 30858
tests: correct (I think) command in test-largefiles-update
When this test was introduced, it used the short-form of all the flags
on this update invocation. I suspect, based on the "start with clean
dirstates" comment and the fact that the no-exec branch of the #if
guard leaves dirstate clean, that this should have been 'update -qCr'
instead of 'update -qcr', but that a bug in largefiles --check
handling left this problem unnoticed.
I'll leave a breadcrumb further up about the current failure mode in
the hopes that we can fix this some day.
This was previously discussed in [0] but the trail in that thread goes
cold after a few replies. Given that this is still a flaky test, that
appears to only be passing by bad fortune, I think it's worth
correcting the code of the test to make a correct assertion, and to
keep track of the suspected bug with some other mechanism than an
invalid test (if we had support for "expected failure" blocks this
might be a worthwhile use of them?).
0: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089501.html
Augie Fackler <augie@google.com> [Mon, 30 Jan 2017 17:57:21 -0500] rev 30857
tests: expand flags to long form in test-largefiles-update.t
I spent some time confused by this test. I'm pretty sure that this
line intends to be cleaning the dirstate, not checking that it's clean
before updating: the preceding #if block leaves the dirstate clean in
the noexec case, and dirty in the exec case, so we can't expect
consistent behavior across that platform variation. A subsequent patch
will modify this command to use --clean instead of --check.
I'll elaborate in that patch about the hypothetical bug here.
Mads Kiilerich <mads@kiilerich.com> [Tue, 31 Jan 2017 03:25:59 +0100] rev 30856
merge: fix crash on criss cross merge with dir move and delete (issue5020)
Work around that 'dm' in the data model only can have one operation for the
target file, but still can have multiple and conflicting operations on the
source file where the other operation is a 'rm'. The move would thus fail with
'abort: No such file or directory'.
In this case it is "obvious" that the file should be removed, either before or
after moving it. We thus keep the 'rm' of the source file but drop the 'dm'.
This is not a pretty fix but quite "obviously" safe (famous last words...) as
it only touches a rare code path that used to crash. It is possible that it
would be better to swap the files for 'dm' as suggested on
https://bz.mercurial-scm.org/show_bug.cgi?id=5020#c13 but it is not entirely
obvious that it not just would create conflicts on the other file. That can be
revisited later.
Mads Kiilerich <mads@kiilerich.com> [Tue, 31 Jan 2017 03:20:07 +0100] rev 30855
tests: use 'f' in test-merge-criss-cross.t to prepare for recursive dumping
Prepare for adding a test case with files in a directory.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 30 Jan 2017 22:58:56 -0800] rev 30854
util: make sortdict.keys() return a copy
dict.keys() is documented to return a copy, so it's surprising that
sortdict.keys() did not. I noticed this because we have an extension
that calls readlocaltags(). That method tries to remove any tags that
point to non-existent revisions (most likely stripped). However, since
it's unintentionally working on the instance it's modifying, it
sometimes fails to remove tags when there are multiple bad tags in a
row. This was not caught because localrepo.tags() does an additional
layer of filtering.
sortdict is also used in other places, but I have not checked whether
its keys() and/or __delitem__() methods are used there.
Yuya Nishihara <yuya@tcha.org> [Mon, 30 Jan 2017 22:50:20 +0900] rev 30853
test-highlight: add normalization rule for Pygments 2.2
The test failed on Debian sid because of new class="vm".
Danek Duvall <danek.duvall@oracle.com> [Sun, 29 Jan 2017 12:40:56 -0800] rev 30852
tests: account for different newline behavior between Solaris and GNU grep
GNU grep, when emitting a matching line that doesn't have a terminating
newline, will add an extra newline. Solaris grep passes the original line
through without the newline. This causes differences in test output when
looking at the last line of the output of get-with-headers.py, which
doesn't usually emit (and certainly doesn't guarantee) a terminating
newline.
Both grep implementations succeed in matching the requested pattern,
though, so rely on specifying the full pattern on grep's commandline
instead of expecting it in the output, and send the output to /dev/null.
Augie Fackler <augie@google.com> [Fri, 20 Jan 2017 10:17:34 -0500] rev 30851
tests: also allow "Protocol not supported" in test-http-proxy error
I've seen this in a (misconfigured) FreeBSD jail which has ::1 as an
entry for localhost, but IPv6 support is disabled in the jail. It took
me months to figure out what was going on (and I only figured it out
when tinyproxy.py got confused by similar IPv4-level misconfiguration
of the localhost domain in /etc/hosts.)
I don't feel strongly about this patch: on the one hand, it's papering
over a host-level misconfiguration, but on the other it avoids some
weird and hard to diagnose problems that can occur in weirdly
restricted environments.
Yuya Nishihara <yuya@tcha.org> [Fri, 20 Jan 2017 21:33:18 +0900] rev 30850
revset: prevent using outgoing() and remote() in hgweb session (BC)
outgoing() and remote() may stall for long due to network I/O, which seems
unsafe per definition, "whether a predicate is safe for DoS attack." But I'm
not 100% sure about this. If our concern isn't elapsed time but CPU resource,
these predicates are considered safe. Perhaps that would be up to the
web/application server configuration?
Anyway, outgoing() and remote() wouldn't be useful in hgweb, so I think
it's okay to ban them.
Augie Fackler <augie@google.com> [Thu, 19 Jan 2017 16:23:49 -0500] rev 30849
tests: use an absolute path to get around '..' being invalid on a dead CWD
Only FreeBSD seems to be this picky. Note that this explicit
absolute-path `cd` exposes a defect in the test, in that we end up
still inside the cwd-vanish repository, but that's not a regression in
this change. Since we're in a code freeze, I'm doing the smallest
thing possible to try and fix bugs on FreeBSD, rather than cleaning up
the entire problem. I'll follow up with a more complete fix after the
freeze.
Sean Farley <sean@farley.io> [Wed, 18 Jan 2017 18:25:51 -0800] rev 30848
ui: rename tmpdir parameter to more specific repopath
This was requested by Augie and I agree that repopath is more
descriptive.
Yuya Nishihara <yuya@tcha.org> [Thu, 19 Jan 2017 23:01:32 +0900] rev 30847
pager: wrap _runcommand() no matter if stdout is redirected
We've made chg utilize the common code path implemented in pager.py (by
815e1cefd082 and 493935e0327a), but the chg server does not always start
with a tty. Because of this, uisetup() of the pager extension could be
skipped on the chg server.
Kudos given to Sean Farley for dogfooding new chg and spotting this problem.
Kostia Balytskyi <ikostia@fb.com> [Thu, 19 Jan 2017 09:48:40 -0800] rev 30846
shelve: make unshelve not crash when there are missing files (issue4176)
This patch makes it possible to unshelve while having missing files
in your repo as long as shelved changes don't touch those missing files.
It also makes error message better otherwise.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 18 Jan 2017 22:45:07 -0800] rev 30845
statprof: require input file
statprof has a __main__ handler that allows viewing of previously
written data files. As Yuya pointed out during review, f42cd5434cc2
broke this. This patch fixes that.
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 23:43:41 -0500] rev 30844
tests: work around FreeBSD's unzip having slightly different output
According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's
what comes as /usr/bin/unzip, so we may as well cater to it since it's
easy.
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 23:34:35 -0500] rev 30843
contrib: fix check-commit to not reject commits from `hg sign` and `hg tag`
I'm tired of having a spurious red build every time we do a
release. Fix it once and for all.
Augie Fackler <raf@durin42.com> [Wed, 18 Jan 2017 20:03:00 -0500] rev 30842
Added signature for changeset a1dd2c0c479e
Augie Fackler <raf@durin42.com> [Wed, 18 Jan 2017 20:02:58 -0500] rev 30841
Added tag 4.1-rc for changeset a1dd2c0c479e
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 11:54:51 -0500] rev 30840
tests: fix up some http tests for no-zstd case
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 11:43:36 -0500] rev 30839
freeze: merge default into stable for 4.1 code freeze
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:17:39 -0800] rev 30838
patchbomb: add tmpdir parameter to ui.edit call
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:15:57 -0800] rev 30837
histedit: add tmpdir parameter to ui.edit call
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:15:21 -0800] rev 30836
cmdutil: add tmpdir parament to ui.edit calls
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:05:22 -0800] rev 30835
ui: add a parameter to set the temporary directory for edit
Until callsites are updated, this will have no effect. Once callsites
are updated, specifying experimental.editortmpinhg will create editor
temporary files in a subdirectory of .hg, which will make it easier
for tool integrations to determine what repository is in play when
they're asked to edit an hg-related file.
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 18 Jan 2017 03:44:19 +0530] rev 30834
help: update help for `hg update` which was misleading (issue5427)
Matt Harbison <matt_harbison@yahoo.com> [Tue, 17 Jan 2017 23:12:54 -0500] rev 30833
templater: add '{envvars}' to access environment variables
Since the option for ui.exportableenviron is experimental, so is this template
until the underlying API is sorted out.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 17 Jan 2017 23:05:12 -0500] rev 30832
ui: introduce an experimental dict of exportable environment variables
Care needs to be taken to prevent leaking potentially sensitive environment
variables through hgweb, if template support for environment variables is to be
introduced. There are a few ideas about the API for preventing accidental
leaking [1]. Option 3 seems best from the POV of not needing to configure
anything in the normal case. I couldn't figure out how to do that, so guard it
with an experimental option for now.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-January/092383.html
Anton Shestakov <av6@dwimlabs.net> [Tue, 17 Jan 2017 13:44:53 +0800] rev 30831
tests: test experimental.spacemovesdown config for commit -i
The feature is still very experimental, but at least its behavior is captured
in the test.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 17 Jan 2017 10:17:13 -0800] rev 30830
zstd: prevent potential free() of uninitialized memory
This is a cherry pick of an upstream fix. The free() of uninitialed
memory could likely only occur if a malloc() inside zstd fails.
The patched functions aren't currently used by Mercurial. But I don't
like leaving footguns sitting around.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:25:02 -0800] rev 30829
revlog: give EXTSTORED flag value to narrowhg
Narrowhg has been using "1 << 14" as its revlog flag value for a long
time. We (Google) have many repos with that value in production
already. When the same value was reserved for EXTSTORED, it made those
repos invalid. Upgrading them will be a little painful. We should
clearly have reserved the value for narrowhg a long time ago. Since
the EXTSTORED flag is not yet in any release and Facebook also says
they have not started using it in production, so it should be okay to
change it. This patch gives the current value (1 << 14) back to
narrowhg and gives a new value (1 << 13) to EXTSTORED.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:45:10 -0800] rev 30828
help: don't let tools reflow revlog flags list
Before this change, the text about revlog flags was reflowed into a
single paragraph, which made it a bit hard to read. I don't even know
the rules around this, but adding a blank line before each flag seems
to prevent the reflowing.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:29:06 -0800] rev 30827
help: format revlog.txt more closely to result
The rendered text has spaces before each item in the list
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 17 Jan 2017 09:19:24 +0100] rev 30826
hgweb: simplify calculation of first revision in filelog command
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 17 Jan 2017 09:17:29 +0100] rev 30825
hgweb: restore ascending iteration on revs in filelog web command
Follow-up on 96f811bceb85. Adjust back the "parity" generator's offset to keep
rendering the same.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 16 Jan 2017 09:22:32 +0100] rev 30824
context: extract _changesinrange() out of blockancestors()
We'll need it to write a blockdescendants function in next changeset.
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Jan 2017 01:23:07 +0530] rev 30823
shelve: allow multiple shelves with --patch and --stat
Before this patch, there was a single way to see multiple shelves using
`--patch --list` which show all the shelves. Doing `--patch s1 s2` returns an
error. This patch allows to show multiple shelves using `--patch` and `--stat`.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 14 Jan 2017 19:41:43 -0800] rev 30822
zstd: vendor python-zstandard 0.6.0
Commit 63c68d6f5fc8de4afd9bde81b13b537beb4e47e8 from
https://github.com/indygreg/python-zstandard is imported without
modifications (other than removing unwanted files).
This includes minor performance and feature improvements. It also
changes the vendored zstd library from 1.1.1 to 1.1.2.
# no-check-commit
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Jan 2017 20:05:15 +0530] rev 30821
util: add length argument to util.buffer()
util.buffer() either returns inbuilt buffer function or defines a new one which
slices. The inbuilt buffer() also has a length argument which is missing from
the ones we defined. This patch adds that length argument.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 15 Jan 2017 13:17:05 +0530] rev 30820
py3: replace pycompat.getenv with encoding.environ.get
pycompat.getenv returns os.getenvb on py3 which is not available on Windows.
This patch replaces them with encoding.environ.get and checks to ensure no
new instances of os.getenv or os.setenv are introduced.
Yuya Nishihara <yuya@tcha.org> [Sun, 15 Jan 2017 16:33:15 +0900] rev 30819
patch: check length of git index header only if integer is specified
Otherwise TypeError would be raised. Follows up d1901c4c8ec0.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 20:16:56 -0800] rev 30818
localrepo: experimental support for non-zlib revlog compression
The final part of integrating the compression manager APIs into
revlog storage is the plumbing for repositories to advertise they
are using non-zlib storage and for revlogs to instantiate a non-zlib
compression engine.
The main intent of the compression manager work was to zstd all
of the things. Adding zstd to revlogs has proved to be more involved
than other places because revlogs are... special. Very small inputs
and the use of delta chains (which are themselves a form of
compression) are a completely different use case from streaming
compression, which bundles and the wire protocol employ. I've
conducted numerous experiments with zstd in revlogs and have yet
to formalize compression settings and a storage architecture that
I'm confident I won't regret later. In other words, I'm not yet
ready to commit to a new mechanism for using zstd - or any other
compression format - in revlogs.
That being said, having some support for zstd (and other compression
formats) in revlogs in core is beneficial. It can allow others to
conduct experiments.
This patch introduces *highly experimental* support for non-zlib
compression formats in revlogs. Introduced is a config option to
control which compression engine to use. Also introduced is a namespace
of "exp-compression-*" requirements to denote support for non-zlib
compression in revlogs. I've prefixed the namespace with "exp-"
(short for "experimental") because I'm not confident of the
requirements "schema" and in no way want to give the illusion of
supporting these requirements in the future. I fully intend to drop
support for these requirements once we figure out what we're doing
with zstd in revlogs.
A good portion of the patch is teaching the requirements system
about registered compression engines and passing the requested
compression engine as an opener option so revlogs can instantiate
the proper compression engine for new operations.
That's a verbose way of saying "we can now use zstd in revlogs!"
On an `hg pull` conversion of the mozilla-unified repo with no extra
redelta settings (like aggressivemergedeltas), we can see the impact
of zstd vs zlib in revlogs:
$ hg perfrevlogchunks -c
! chunk
! wall 2.032052 comb 2.040000 user 1.990000 sys 0.050000 (best of 5)
! wall 1.866360 comb 1.860000 user 1.820000 sys 0.040000 (best of 6)
! chunk batch
! wall 1.877261 comb 1.870000 user 1.860000 sys 0.010000 (best of 6)
! wall 1.705410 comb 1.710000 user 1.690000 sys 0.020000 (best of 6)
$ hg perfrevlogchunks -m
! chunk
! wall 2.721427 comb 2.720000 user 2.640000 sys 0.080000 (best of 4)
! wall 2.035076 comb 2.030000 user 1.950000 sys 0.080000 (best of 5)
! chunk batch
! wall 2.614561 comb 2.620000 user 2.580000 sys 0.040000 (best of 4)
! wall 1.910252 comb 1.910000 user 1.880000 sys 0.030000 (best of 6)
$ hg perfrevlog -c -d 1
! wall 4.812885 comb 4.820000 user 4.800000 sys 0.020000 (best of 3)
! wall 4.699621 comb 4.710000 user 4.700000 sys 0.010000 (best of 3)
$ hg perfrevlog -m -d 1000
! wall 34.252800 comb 34.250000 user 33.730000 sys 0.520000 (best of 3)
! wall 24.094999 comb 24.090000 user 23.320000 sys 0.770000 (best of 3)
Only modest wins for the changelog. But manifest reading is
significantly faster. What's going on?
One reason might be data volume. zstd decompresses faster. So given
more bytes, it will put more distance between it and zlib.
Another reason is size. In the current design, zstd revlogs are
*larger*:
debugcreatestreamclonebundle (size in bytes)
zlib: 1,638,852,492
zstd: 1,680,601,332
I haven't investigated this fully, but I reckon a significant cause of
larger revlogs is that the zstd frame/header has more bytes than
zlib's. For very small inputs or data that doesn't compress well, we'll
tend to store more uncompressed chunks than with zlib (because the
compressed size isn't smaller than original). This will make revlog
reading faster because it is doing less decompression.
Moving on to bundle performance:
$ hg bundle -a -t none-v2 (total CPU time)
zlib: 102.79s
zstd: 97.75s
So, marginal CPU decrease for reading all chunks in all revlogs
(this is somewhat disappointing).
$ hg bundle -a -t <engine>-v2 (total CPU time)
zlib: 191.59s
zstd: 115.36s
This last test effectively measures the difference between zlib->zlib
and zstd->zstd for revlogs to bundle. This is a rough approximation of
what a server does during `hg clone`.
There are some promising results for zstd. But not enough for me to
feel comfortable advertising it to users. We'll get there...
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 19:58:00 -0800] rev 30817
revlog: use compression engine APIs for decompression
Now that compression engines declare their header in revlog chunks
and can decompress revlog chunks, we refactor revlog.decompress()
to use them.
Making full use of the property that revlog compressor objects are
reusable, revlog instances now maintain a dict mapping an engine's
revlog header to a compressor object. This is not only a performance
optimization for engines where compressor object reuse can result in
better performance, but it also serves as a cache of header values
so we don't need to perform redundant lookups against the compression
engine manager. (Yes, I measured and the overhead of a function call
versus a dict lookup was observed.)
Replacing the previous inline lookup table with a dict lookup was
measured to make chunk reading ~2.5% slower on changelogs and ~4.5%
slower on manifests. So, the inline lookup table has been mostly
preserved so we don't lose performance. This is unfortunate. But
many decompression operations complete in microseconds, so Python
attribute lookup, dict lookup, and function calls do matter.
The impact of this change on mozilla-unified is as follows:
$ hg perfrevlogchunks -c
! chunk
! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6)
! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6)
! chunk batch
! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6)
! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)
$ hg perfrevlogchunks -m
! chunk
! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4)
! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4)
! chunk batch
! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5)
! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4)
Changelog chunk reading is slightly faster but manifest reading is
slower. What gives?
On this repo, 99.85% of changelog entries are zlib compressed (the 'x'
header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch
swapped the test order of 'x' and '\0' so now 'x' is tested first. This
makes changelogs faster since they almost always hit the first branch.
This makes a significant percentage of manifest '\0' chunks slower
because that code path now performs an extra test. Yes, I too can't
believe we're able to measure the impact of an if..elif with simple
string compares. I reckon this code would benefit from being written
in C...