Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 18 Sep 2018 10:46:19 -0700] rev 39687
tests: add debug output in test-narrow-widen-no-ellipsis.t
This will help us in understanding the upcoming patches better.
Differential Revision: https://phab.mercurial-scm.org/D4637
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 17 Sep 2018 18:21:17 +0300] rev 39686
changegroup: improve the devel-warn to specify changelog was empty
Right now, the develwarn says "applied empty changegroup" which is not correct
because we can send a changegroup without changelog with just manifest and
filelogs and it will still say the same.
Let's fix this to say that we are applying empty changelog from changegroup. In
future patches I am will be adding functionality to send a changegroup from the
server without an empty changelog.
Differential Revision: https://phab.mercurial-scm.org/D4636
Anton Shestakov <av6@dwimlabs.net> [Mon, 17 Sep 2018 13:21:46 +0800] rev 39685
zsh_completion: add -b/--branch and -B/--bookmark(s) flags properly
_hg_branch_bmark_opts used to add these two flags, but had the same
descriptions for the flags regardless of what command took them and didn't
allow specifying flags more than once (no '*' at the start). Even more
importantly, it assumed that -B was always expecting an argument (i.e.
--bookmark=foo), but in case of incoming and outgoing it's not so (--bookmarks
is self-sufficient).
Differential Revision: https://phab.mercurial-scm.org/D4612
spectral <spectral@google.com> [Fri, 14 Sep 2018 16:29:51 -0700] rev 39684
narrow: when writing treemanifests, skip inspecting directories outside narrow
This provides significant speed benefits when narrow and treemanifests are in
use, see the timing numbers below. Note that like previously, differences of <5%
are considered noise.
The below timing numbers are in the same style as previously (example:
ee7ee0c516ca). 'before' is 9db85644, and does not include that example commit's
improvements.
diff --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.327 s +- 0.051 s | 1.296 s +- 0.009 s | 97.7%
m-u | | x | 1.310 s +- 0.020 s | 1.295 s +- 0.015 s | 98.9%
m-u | x | | 1.295 s +- 0.018 s | 1.296 s +- 0.007 s | 100.1%
m-u | x | x | 83.5 ms +- 0.8 ms | 84.1 ms +- 0.8 ms | 100.7%
l-d-r | | | 205.1 ms +- 3.5 ms | 205.0 ms +- 3.8 ms | 100.0%
l-d-r | | x | 194.2 ms +- 5.6 ms | 192.3 ms +- 4.3 ms | 99.0%
l-d-r | x | | 99.1 ms +- 2.2 ms | 97.8 ms +- 0.9 ms | 98.7%
l-d-r | x | x | 66.2 ms +- 1.0 ms | 67.2 ms +- 2.7 ms | 101.5%
diff -c . --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 233.9 ms +- 1.9 ms | 235.6 ms +- 5.1 ms | 100.7%
m-u | | x | 151.4 ms +- 1.2 ms | 152.2 ms +- 2.0 ms | 100.5%
m-u | x | | 234.8 ms +- 2.7 ms | 235.0 ms +- 2.7 ms | 100.1%
m-u | x | x | 127.8 ms +- 2.1 ms | 126.0 ms +- 1.1 ms | 98.6%
l-d-r | | | 82.5 ms +- 1.6 ms | 82.3 ms +- 2.0 ms | 99.8%
l-d-r | | x | 3.742 s +- 0.017 s | 3.819 s +- 0.208 s | 102.1%
l-d-r | x | | 84.4 ms +- 1.5 ms | 83.2 ms +- 1.0 ms | 98.6%
l-d-r | x | x | 751.2 ms +- 5.0 ms | 755.8 ms +- 12.9 ms | 100.6%
rebase -r . --keep -d .^^:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 5.519 s +- 0.038 s | 5.526 s +- 0.057 s | 100.1%
m-u | | x | 5.588 s +- 0.048 s | 5.607 s +- 0.061 s | 100.3%
m-u | x | | 5.520 s +- 0.044 s | 5.546 s +- 0.059 s | 100.5%
m-u | x | x | 586.6 ms +- 12.8 ms | 554.9 ms +- 21.2 ms | 94.6% <--
l-d-r | | | 629.8 ms +- 5.5 ms | 627.4 ms +- 6.6 ms | 99.6%
l-d-r | | x | 6.165 s +- 0.058 s | 6.255 s +- 0.303 s | 101.5%
l-d-r | x | | 270.2 ms +- 2.3 ms | 271.4 ms +- 2.7 ms | 100.4%
l-d-r | x | x | 4.700 s +- 0.025 s | 1.651 s +- 0.016 s | 35.1% <--
status --change . --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 215.4 ms +- 2.3 ms | 216.5 ms +- 4.2 ms | 100.5%
m-u | | x | 132.9 ms +- 1.2 ms | 132.0 ms +- 1.4 ms | 99.3%
m-u | x | | 217.0 ms +- 1.9 ms | 215.4 ms +- 1.9 ms | 99.3%
m-u | x | x | 108.6 ms +- 1.0 ms | 108.2 ms +- 1.5 ms | 99.6%
l-d-r | | | 80.0 ms +- 1.3 ms | 80.5 ms +- 1.1 ms | 100.6%
l-d-r | | x | 3.916 s +- 0.187 s | 3.966 s +- 0.236 s | 101.3%
l-d-r | x | | 84.4 ms +- 3.1 ms | 83.9 ms +- 1.1 ms | 99.4%
l-d-r | x | x | 758.0 ms +- 8.2 ms | 753.5 ms +- 5.0 ms | 99.4%
status --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.905 s +- 0.025 s | 1.910 s +- 0.044 s | 100.3%
m-u | | x | 1.892 s +- 0.009 s | 1.895 s +- 0.012 s | 100.2%
m-u | x | | 1.891 s +- 0.012 s | 1.902 s +- 0.018 s | 100.6%
m-u | x | x | 93.3 ms +- 0.9 ms | 93.4 ms +- 0.8 ms | 100.1%
l-d-r | | | 570.7 ms +- 7.8 ms | 571.9 ms +- 18.5 ms | 100.2%
l-d-r | | x | 561.5 ms +- 5.2 ms | 562.9 ms +- 6.1 ms | 100.2%
l-d-r | x | | 171.7 ms +- 2.6 ms | 171.9 ms +- 1.2 ms | 100.1%
l-d-r | x | x | 142.7 ms +- 2.0 ms | 140.3 ms +- 1.0 ms | 98.3%
update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 3.126 s +- 0.016 s | 3.128 s +- 0.015 s | 100.1%
m-u | | x | 3.014 s +- 0.068 s | 3.008 s +- 0.031 s | 99.8%
m-u | x | | 3.143 s +- 0.037 s | 3.184 s +- 0.086 s | 101.3%
m-u | x | x | 308.0 ms +- 1.8 ms | 308.1 ms +- 5.7 ms | 100.0%
l-d-r | | | 430.8 ms +- 4.5 ms | 436.4 ms +- 8.7 ms | 101.3%
l-d-r | | x | 9.676 s +- 0.127 s | 9.945 s +- 0.272 s | 102.8%
l-d-r | x | | 254.2 ms +- 3.3 ms | 255.7 ms +- 3.1 ms | 100.6%
l-d-r | x | x | 1.571 s +- 0.030 s | 1.555 s +- 0.014 s | 99.0%
Differential Revision: https://phab.mercurial-scm.org/D4606
Augie Fackler <augie@google.com> [Mon, 17 Sep 2018 15:16:20 -0400] rev 39683
tests: fix a couple of drawdag.py references
Differential Revision: https://phab.mercurial-scm.org/D4635
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 14 Sep 2018 23:51:21 +0300] rev 39682
py3: fix kwargs handling in hgext/fastannotate.py
Differential Revision: https://phab.mercurial-scm.org/D4588
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 17 Sep 2018 15:55:18 +0300] rev 39681
narrow: use diffmatcher to send only new filelogs in non-ellipses widening
Before this patch, when we widen a non-ellipses narrow clone, we downloads all
the filelogs matching the resulting new matcher. This is same as the ellipses
case but can be improved because, we don't pull new csets in non-ellipses cases,
we can only download the new added files instead of downloading all the files
which matches the new matcher.
So, we only download files which matches the new matcher but does not matches
the old matcher. There exists a match.differencematcher() which is used here.
This will lead to significant amount of speedup in extending a non-ellipses
narrow copy on large repos because we will download and process only the new
required filelogs.
The tests changes demonstrate that we are downloading now less files.
Thanks to Augie for pointing that functionality of differencematcher exists in
core.
Differential Revision: https://phab.mercurial-scm.org/D4614
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 17 Sep 2018 15:27:39 +0300] rev 39680
py3: add missing b'' prefixes in couple of test files
These were missed in the earlier patch and caught by Yuya.
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D4613
Matt Harbison <matt_harbison@yahoo.com> [Sun, 16 Sep 2018 23:13:05 -0400] rev 39679
run-tests: convert the remaining os.system() call to Unicode
I wasn't able to hit this path in 543a788eea2d, but I have now when I
accidentally left off `--local`.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Sep 2018 13:31:41 -0400] rev 39678
py3: partially fix pager spawning on Windows
Previously, spinning up the pager crashed because the command and environment
was in bytes. (See also 543a788eea2d.) Now it aborts with an invalid handle:
$ HGMODULEPOLICY=py py -3 ../hg --traceback --config extensions.evolve=!
Traceback (most recent call last):
File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 967, in _write
self.fout.write(''.join(msgs))
File "c:\Users\Matt\projects\hg\mercurial\windows.py", line 173, in write
self.fp.write(s[start:end])
OSError: [WinError 6] The handle is invalid
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\Users\Matt\projects\hg\mercurial\scmutil.py", line 164, in callcatch
return func()
File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 350, in _runcatchfunc
return _dispatch(req)
File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 930, in _dispatch
return commands.help_(ui, 'shortlist')
File "c:\Users\Matt\projects\hg\mercurial\commands.py", line 2930, in help_
ui.write(formatted)
File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 948, in write
self._writenobuf(*args, **opts)
File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 960, in _writenobuf
self._write(*msgs, **opts)
File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 969, in _write
raise error.StdioError(err)
mercurial.error.StdioError: [Errno 9] The handle is invalid
abort: The handle is invalid
The interesting bit here is that the abort message is marked with ANSI color,
but the OSError is not.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Sep 2018 10:35:00 +0900] rev 39677
censor: rename loop variable to silence pyflakes warning
hgext/censor.py:92: list comprehension redefines 'c' from line 88
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 16 Sep 2018 20:58:51 +0530] rev 39676
py3: add b'' prefixes in tests/test-hgweb-no-request-uri.t
# skip-blame because just b'' prefixes.
Differential Revision: https://phab.mercurial-scm.org/D4611
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 16 Sep 2018 20:49:37 +0530] rev 39675
py3: add b'' prefixes in tests/test-hgweb-no-path-info.t
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4610
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 16 Sep 2018 20:20:59 +0530] rev 39674
py3: add b'' prefixes in tests/test-hgweb-non-interactive.t
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D4609
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 16 Sep 2018 19:58:01 +0530] rev 39673
py3: use codecs.encode() to encode in rot-13 encoding
The other occurence will need some more love as description is bytes by default
and we need to decode it and then encode it.
Differential Revision: https://phab.mercurial-scm.org/D4608
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 16 Sep 2018 19:18:15 +0530] rev 39672
py3: add two passing tests to whitelist found by buildbot
The buildbot found these two new passing tests on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4607
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 01:36:43 -0400] rev 39671
phabricator: mark extension as experimental for now
I don't want us to commit to this having a stable interface just yet.
Differential Revision: https://phab.mercurial-scm.org/D4605
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 01:16:31 -0400] rev 39670
phabricator: fix templating bug by using hybriddict
Differential Revision: https://phab.mercurial-scm.org/D4604
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 01:13:37 -0400] rev 39669
phabricator: add tests of templatekeyword
Having tests is paying off: I found a bug and now it'll be easy to
fix!
Differential Revision: https://phab.mercurial-scm.org/D4603
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 00:46:17 -0400] rev 39668
phabricator: move extension from contrib to hgext
It's well-enough tested now and widely enough used I think we should
ship it.
Differential Revision: https://phab.mercurial-scm.org/D4602
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 00:50:21 -0400] rev 39667
tests: add some basic tests of phabricator interactions
This uses the vcr library to avoid hitting phabricator on every test
execution. In order to generate new recordings (vcr calls them
cassettes) just remove the appropriate json file, and the test will
regenerate it. It's not my favorite way to test things, but it'll let
us have test coverage on the phabricator extension that'll make it
resilient to refactors in core and let us move it to hgext.
In the future, it'd probably be better to have a docker container we
can spin up for creating the vcr recordings, but for now this is
enough better than nothing I'm going to declare victory.
Coverage reports about 73% of the extension is now covered.
Differential Revision: https://phab.mercurial-scm.org/D4601
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 00:20:03 -0400] rev 39666
phabricator: add support for using the vcr library to mock interactions
I'll use this in an upcoming test. The decorator dancing in this is
more complicated than I'd like, but it beats repeating all this code
everywhere.
Differential Revision: https://phab.mercurial-scm.org/D4600
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 00:19:09 -0400] rev 39665
keepalive: work around slight deficiency in vcr
VCR's response type doesn't define the will_close attribute. Let's
just have keepalive default to closing the socket if the will_close
attribute is missing.
Differential Revision: https://phab.mercurial-scm.org/D4599
Augie Fackler <raf@durin42.com> [Sat, 15 Sep 2018 00:18:16 -0400] rev 39664
hghave: add a checker for the vcr HTTP record/replay library
I'm going to use this to write some tests of the phabricator
extension.
Differential Revision: https://phab.mercurial-scm.org/D4598
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Sep 2018 00:04:06 -0400] rev 39663
py3: allow run-tests.py to run on Windows
This is now functional:
HGMODULEPOLICY=py py -3 run-tests.py --local test-help.t --pure --view bcompare
However, on this machine without a C compiler, it tries to load cext anyway, and
blows up. I haven't looked into why, other than to see that it does set the
environment variable. When the test exits though, I see it can't find
killdaemons.py, get-with-headers.py, etc.
I have no idea why these changes are needed, given that it runs on Linux. But
os.system() is insisting that it take a str, and subprocess.Popen() blows up
without str:
Errored test-help.t: Traceback (most recent call last):
File "run-tests.py", line 810, in run
self.runTest()
File "run-tests.py", line 858, in runTest
ret, out = self._run(env)
File "run-tests.py", line 1268, in _run
exitcode, output = self._runcommand(cmd, env)
File "run-tests.py", line 1141, in _runcommand
env=env)
File "C:\Program Files\Python37\lib\subprocess.py", line 756, in __init__
restore_signals, start_new_session)
File "C:\Program Files\Python37\lib\subprocess.py", line 1100, in _execute_child
args = list2cmdline(args)
File "C:\Program Files\Python37\lib\subprocess.py", line 511, in list2cmdline
needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'int' is not iterable
This is exactly how it crashes when trying to spin up a pager too. I left one
instance of os.system() unchanged in _installhg(), because it doesn't get there.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 14 Sep 2018 23:04:18 -0400] rev 39662
py3: ensure run-tests environment is uniformly str
subprocess.popen() was crashing, and when I printed out `env`, all of the keys
and most of the values were str. Except these.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 14 Sep 2018 22:57:35 -0400] rev 39661
py3: ensure run-tests.osenvironb is actually bytes
Windows doesn't have os.environb, so it was falling back to the Unicode form,
and all of the accesses are trying to use bytes.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 13 Sep 2018 22:07:00 -0400] rev 39660
py3: fix str vs bytes in enough places to run `hg version` on Windows
I don't have Visual Studio 2015 at home, but this now works with a handful of
extensions (blackbox, extdiff, patchbomb, phabricator and rebase, but not
evolve):
$ HGMODULEPOLICY=py py -3 ../hg version
Enabling the evolve extension causes the usual "failed to import ..." line, but
then print this before the usual version output:
('commit', '[b'debugancestor', b'debugapplystreamclonebundle', ...,
b'verify', b'version']')
... where the elided part seems to be every command and alias known.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 13 Sep 2018 20:54:53 -0400] rev 39659
windows: open registry keys using unicode names
Python3 complained it must be str. While here, use a context manager to close
the key- it wouldn't wrap at 80 characters the old way, and would have had to
move anyway.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 13 Sep 2018 00:39:02 -0400] rev 39658
py3: byteify strings in pycompat
These surfaced when disabling the source transformer to debug the problems in
win32.py. ./contrib/byteify-strings.py found a couple false positives, so I
marked them with r'' explicitly (in case I'm wrong).
# skip-blame since this is just b'' and r'' prefixing
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 30 Aug 2018 14:55:34 -0700] rev 39657
wireprotov2: let clients drive delta behavior
Previously, the "manifestdata" and "filedata" commands assumed the
receiver had all parent revisions for requested nodes. Unless the
revision had no parents, they emitted a delta instead of a fulltext.
This strategy isn't appropriate for shallow clones and for clients
that only want to access fulltext revision data for a single node
without fetching their parent revisions.
This commit adds an "haveparents" argument to the "manifestdata"
and "filedata" commands that controls delta generation behavior.
Unless "haveparents" is set, the server assumes that the client
doesn't have parent revisions unless they were previously sent
as part of the current group of revisions.
This change allows the fulltext revision data of any individual
revision to be obtained. This will facilitate shallow clones
and other data retrieval strategies that don't require all previous
revisions of an entity to be fetched.
Differential Revision: https://phab.mercurial-scm.org/D4492
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 04 Sep 2018 10:42:24 -0700] rev 39656
exchangev2: fetch file revisions
Now that the server has an API for fetching file data, we can call into
it to fetch file revisions.
The implementation is relatively straightforward: we examine the
manifests that we fetched and find all new file revisions referenced
by them. We build up a mapping from file path to file nodes to
manifest node. (The mapping to first manifest node allows us to
map back to first changelog node/revision, which is used for the
linkrev.)
Once that map is built up, we iterate over it in a deterministic
manner and fetch and store file data. The code is very similar
to manifest fetching. So similar that we could probably extract the
common bits into a generic function.
With file data retrieval implemented, `hg clone` and `hg pull` are
effectively feature complete, at least as far as the completeness
of data transfer for essential repository data (changesets, manifests,
files, phases, and bookmarks). We're still missing support for
obsolescence markers, the hgtags fnodes cache, and the branchmap
cache. But these are non-essential for the moment (and will be
implemented later).
This is a good point to assess the state of exchangev2 in terms of
performance. I ran a local `hg clone` for the mozilla-unified
repository using both version 1 and version 2 of the wire protocols
and exchange methods. This is effectively comparing the performance
of the wire protocol overhead and "getbundle" versus domain-specific
commands. Wire protocol version 2 doesn't have compression implemented
yet. So I tested version 1 with `server.compressionengines=none` to
remove compression overhead from the equation.
server
before: user 220.420+0.000 sys 14.420+0.000
after: user 321.980+0.000 sys 18.990+0.000
client
before: real 561.650 secs (user 497.670+0.000 sys 28.160+0.000)
after: real 1226.260 secs (user 944.240+0.000 sys 354.150+0.000)
We have substantial regressions on both client and server. This
is obviously not desirable. I'm aware of some reasons:
* Lack of hgtagsfnodes transfer (contributes significant CPU to
client).
* Lack of branch cache transfer (contributes significant CPU to
client).
* Little to no profiling / optimization performed on wire protocol
version 2 code.
* There appears to be a memory leak on the client and that is likely
causing swapping on my machine.
* Using multiple threads on the client may be counter-productive because
Python.
* We're not compressing on the server.
* We're tracking file nodes on the client via manifest diffing
rather than using linkrev shortcuts on the server.
I'm pretty confident that most of these issues are addressable.
But even if we can't get wire protocol version 2 on performance parity
with "getbundle," I still think it is important to have the set of low
level data-specific retrieval commands that we have implemented so
far. This is because the existence of such commands allows flexibility
in how clients access server data.
Differential Revision: https://phab.mercurial-scm.org/D4491