Gregory Szorc <gregory.szorc@gmail.com> [Wed, 12 Sep 2018 09:58:23 -0700] rev 39645
exchangev2: start to implement pull with wire protocol v2
Wire protocol version 2 will take a substantially different
approach to exchange than version 1 (at least as far as pulling
is concerned).
This commit establishes a new exchangev2 module for holding
code related to exchange using wire protocol v2. I could have
added things to the existing exchange module. But it is already
quite big. And doing things inline isn't in question because
the existing code is already littered with conditional code
for various states of support for the existing wire protocol
as it evolved over 10+ years. A new module gives us a chance
to make a clean break.
This approach does mean we'll end up writing some duplicate
code. And there's a significant chance we'll miss functionality
as code is ported. The plan is to eventually add #testcase's
to existing tests so the new wire protocol is tested side-by-side
with the existing one. This will hopefully tease out any
features that weren't ported properly. But before we get there,
we need to build up support for the new exchange methods.
Our journey towards implementing a new exchange begins with pulling.
And pulling begins with discovery.
The discovery code added to exchangev2 is heavily drawn from
the following functions:
* exchange._pulldiscoverychangegroup
* discovery.findcommonincoming
For now, we build on top of existing discovery mechanisms. The
new wire protocol should be capable of doing things more efficiently.
But I'd rather defer on this problem.
To foster the transition, we invent a fake capability on the HTTPv2
peer and have the main pull code in exchange.py call into exchangev2
when the new wire protocol is being used.
Differential Revision: https://phab.mercurial-scm.org/D4480
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 21 Aug 2018 15:33:11 -0700] rev 39644
httppeer: expose capabilities for each command
This will help code using peers to sniff out exactly what servers
support.
Differential Revision: https://phab.mercurial-scm.org/D4436
spectral <spectral@google.com> [Thu, 13 Sep 2018 22:48:27 -0700] rev 39643
narrow: intersect provided matcher with narrowmatcher in `hg diff`
This provides significant speedups when running diff, and no change in behavior
that I'm aware of (or that the tests found). I tested with a repo that I started
using narrow in after it was created and attempted to run `hg diff -c .` and
similar commands in it on a commit that had files not in the narrowspec.
Timing numbers below, using a similar setup as my previous commits.
before=9db85644, m-u is mozilla-unified at eb39298e432d (flatmanifest) and
0553b7f29eaf (treemanifest). l-d-r is a repo simulating a situation I've
encountered where there's one directory with 30k+ subdirectories. N means
narrow, T means treemanifest. The narrowspec is pretty small when in use, and
importantly the narrowspec is applied *after* doing the initial checkout
(without narrowing), so all of these files exist in the filesystem, which is not
normally the case if someone has been using narrow for the entire life of the
clone.
Anything less than a 5% difference in performance is most likely noise.
diff --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.292 s +- 0.009 s | 1.295 s +- 0.010 s | 100.2%
m-u | | x | 1.296 s +- 0.042 s | 1.299 s +- 0.026 s | 100.2%
m-u | x | | 1.292 s +- 0.010 s | 1.297 s +- 0.021 s | 100.4%
m-u | x | x | 84.2 ms +- 1.2 ms | 83.6 ms +- 0.2 ms | 99.3%
l-d-r | | | 188.7 ms +- 2.7 ms | 188.8 ms +- 2.0 ms | 100.1%
l-d-r | | x | 189.9 ms +- 1.5 ms | 189.4 ms +- 1.2 ms | 99.7%
l-d-r | x | | 97.1 ms +- 1.0 ms | 87.1 ms +- 1.0 ms | 89.7% <--
l-d-r | x | x | 96.9 ms +- 0.8 ms | 87.2 ms +- 0.7 ms | 90.0% <--
diff -c . --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 231.6 ms +- 3.1 ms | 228.9 ms +- 1.6 ms | 98.8%
m-u | | x | 150.5 ms +- 1.7 ms | 150.7 ms +- 1.4 ms | 100.1%
m-u | x | | 233.7 ms +- 2.4 ms | 232.2 ms +- 1.9 ms | 99.4%
m-u | x | x | 126.1 ms +- 1.2 ms | 126.8 ms +- 1.2 ms | 100.6%
l-d-r | | | 82.1 ms +- 2.0 ms | 81.8 ms +- 1.4 ms | 99.6%
l-d-r | | x | 3.732 s +- 0.020 s | 3.746 s +- 0.027 s | 100.4%
l-d-r | x | | 83.1 ms +- 0.8 ms | 107.6 ms +- 2.4 ms | 129.5% <--
l-d-r | x | x | 758.2 ms +- 38.8 ms | 188.5 ms +- 1.8 ms | 24.9% <--
rebase -r . --keep -d .^^:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 5.532 s +- 0.087 s | 5.496 s +- 0.016 s | 99.3%
m-u | | x | 5.554 s +- 0.061 s | 5.532 s +- 0.013 s | 99.6%
m-u | x | | 5.602 s +- 0.134 s | 5.508 s +- 0.035 s | 98.3%
m-u | x | x | 582.2 ms +- 15.2 ms | 572.9 ms +- 12.0 ms | 98.4%
l-d-r | | | 629.5 ms +- 12.3 ms | 622.5 ms +- 7.3 ms | 98.9%
l-d-r | | x | 6.173 s +- 0.062 s | 6.185 s +- 0.076 s | 100.2%
l-d-r | x | | 274.5 ms +- 10.0 ms | 272.1 ms +- 6.2 ms | 99.1%
l-d-r | x | x | 4.835 s +- 0.056 s | 4.826 s +- 0.034 s | 99.8%
status --change . --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 214.4 ms +- 1.4 ms | 212.2 ms +- 1.7 ms | 99.0%
m-u | | x | 130.9 ms +- 1.2 ms | 131.7 ms +- 1.1 ms | 100.6%
m-u | x | | 215.0 ms +- 2.1 ms | 214.9 ms +- 2.7 ms | 100.0%
m-u | x | x | 109.5 ms +- 2.3 ms | 107.8 ms +- 0.9 ms | 98.4%
l-d-r | | | 79.6 ms +- 0.9 ms | 79.8 ms +- 1.6 ms | 100.3%
l-d-r | | x | 3.799 s +- 0.037 s | 3.928 s +- 0.021 s | 103.4% <--?
l-d-r | x | | 82.7 ms +- 0.7 ms | 83.2 ms +- 1.0 ms | 100.6%
l-d-r | x | x | 746.8 ms +- 6.1 ms | 739.0 ms +- 4.2 ms | 99.0%
status --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.884 s +- 0.012 s | 1.885 s +- 0.013 s | 100.1%
m-u | | x | 1.897 s +- 0.027 s | 1.909 s +- 0.077 s | 100.6%
m-u | x | | 1.886 s +- 0.021 s | 1.891 s +- 0.030 s | 100.3%
m-u | x | x | 92.0 ms +- 0.7 ms | 92.4 ms +- 0.4 ms | 100.4%
l-d-r | | | 570.3 ms +- 18.7 ms | 552.2 ms +- 4.5 ms | 96.8%
l-d-r | | x | 568.9 ms +- 16.1 ms | 567.2 ms +- 11.9 ms | 99.7%
l-d-r | x | | 171.1 ms +- 2.5 ms | 170.4 ms +- 1.2 ms | 99.6%
l-d-r | x | x | 171.6 ms +- 3.4 ms | 171.5 ms +- 1.7 ms | 99.9%
update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 3.107 s +- 0.017 s | 3.116 s +- 0.012 s | 100.3%
m-u | | x | 2.943 s +- 0.010 s | 2.945 s +- 0.019 s | 100.1%
m-u | x | | 3.116 s +- 0.033 s | 3.118 s +- 0.027 s | 100.1%
m-u | x | x | 318.5 ms +- 2.7 ms | 320.8 ms +- 4.8 ms | 100.7%
l-d-r | | | 428.9 ms +- 4.4 ms | 429.5 ms +- 4.0 ms | 100.1%
l-d-r | | x | 9.593 s +- 0.081 s | 9.869 s +- 0.043 s | 102.9%
l-d-r | x | | 253.2 ms +- 3.6 ms | 254.0 ms +- 2.8 ms | 100.3%
l-d-r | x | x | 1.613 s +- 0.009 s | 1.630 s +- 0.017 s | 101.1%
Differential Revision: https://phab.mercurial-scm.org/D4587
Yuya Nishihara <yuya@tcha.org> [Sat, 01 Sep 2018 12:15:02 +0900] rev 39642
identify: change {parents} to a list of nodes (BC)
This is a part of the name unification. {parents} is a list of nodes in
"hg log -Tjson" output. Since {rev} can be computed from (repo, node) pair,
we no longer need to put it to provide {rev} to user templates.
https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary
Yuya Nishihara <yuya@tcha.org> [Sat, 01 Sep 2018 12:09:22 +0900] rev 39641
identify: use fm.hexfunc thoroughly
This fixes the length of {id} in JSON and template outputs.
Yuya Nishihara <yuya@tcha.org> [Sat, 01 Sep 2018 15:52:18 +0900] rev 39640
formatter: replace contexthint() with demand loading of ctx object
And pass in repo instead to resolve ctx from (repo, node) pair.
Yuya Nishihara <yuya@tcha.org> [Thu, 07 Jun 2018 21:48:11 +0900] rev 39639
formatter: populate ctx from repo and node value
This will basically replace the fm.contexthint() API. I originally thought
this would be too complicated, and I wrote 8399438bc7ef "formatter: provide
hint of context keys required by template" because of that. However, I had
to add a similar mechanism for fctx templates, and the overall machinery
became way simpler than my original patch.
The test output slightly changed as {author} is no longer available in
the {manifest} context, which isn't the point this test targeted on.
Augie Fackler <augie@google.com> [Fri, 14 Sep 2018 18:18:46 -0400] rev 39638
merge with stable
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 15 Sep 2018 00:37:20 +0300] rev 39637
py3: call hgweb.hgweb() with bytes values
# skip-blame because just b'' prefixes
I believe this should fix some tests.
Differential Revision: https://phab.mercurial-scm.org/D4594
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 15 Sep 2018 00:24:05 +0300] rev 39636
py3: use '%d' for integers instead of '%s'
Differential Revision: https://phab.mercurial-scm.org/D4593
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 15 Sep 2018 00:17:56 +0300] rev 39635
py3: use "%f" for floats instead of "%s"
Differential Revision: https://phab.mercurial-scm.org/D4592
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 15 Sep 2018 00:01:52 +0300] rev 39634
py3: suppress the return value from .write() call
Differential Revision: https://phab.mercurial-scm.org/D4591
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 15 Sep 2018 00:01:20 +0300] rev 39633
py3: add b'' prefixes in tests/test-diff-color.t
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4590
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 14 Sep 2018 23:59:41 +0300] rev 39632
py3: slice through bytes to prevent getting ascii value
I still don't know why python-dev thought it was a nice idea to do this.
Differential Revision: https://phab.mercurial-scm.org/D4589
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Thu, 13 Sep 2018 16:22:53 -0400] rev 39631
censor: use a reasonable amount of memory
Before this change, trying to censor some random revision uses an ever
increasing amount of memory (I stopped at 20GB, but it was by no means
finished), presumably because these contexts have a lot of
information that is kept alive.
After this change, the memory usage plateaus quickly.
Differential Revision: https://phab.mercurial-scm.org/D4582
Yuya Nishihara <yuya@tcha.org> [Fri, 14 Sep 2018 22:25:44 +0900] rev 39630
help: add internals.wireprotocolrpc to the table