narrow: when writing treemanifests, skip inspecting directories outside narrow This provides significant speed benefits when narrow and treemanifests are in use, see the timing numbers below. Note that like previously, differences of <5% are considered noise. The below timing numbers are in the same style as previously (example: ee7ee0c516ca). 'before' is 9db85644, and does not include that example commit's improvements. diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.327 s +- 0.051 s | 1.296 s +- 0.009 s | 97.7% m-u | | x | 1.310 s +- 0.020 s | 1.295 s +- 0.015 s | 98.9% m-u | x | | 1.295 s +- 0.018 s | 1.296 s +- 0.007 s | 100.1% m-u | x | x | 83.5 ms +- 0.8 ms | 84.1 ms +- 0.8 ms | 100.7% l-d-r | | | 205.1 ms +- 3.5 ms | 205.0 ms +- 3.8 ms | 100.0% l-d-r | | x | 194.2 ms +- 5.6 ms | 192.3 ms +- 4.3 ms | 99.0% l-d-r | x | | 99.1 ms +- 2.2 ms | 97.8 ms +- 0.9 ms | 98.7% l-d-r | x | x | 66.2 ms +- 1.0 ms | 67.2 ms +- 2.7 ms | 101.5% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 233.9 ms +- 1.9 ms | 235.6 ms +- 5.1 ms | 100.7% m-u | | x | 151.4 ms +- 1.2 ms | 152.2 ms +- 2.0 ms | 100.5% m-u | x | | 234.8 ms +- 2.7 ms | 235.0 ms +- 2.7 ms | 100.1% m-u | x | x | 127.8 ms +- 2.1 ms | 126.0 ms +- 1.1 ms | 98.6% l-d-r | | | 82.5 ms +- 1.6 ms | 82.3 ms +- 2.0 ms | 99.8% l-d-r | | x | 3.742 s +- 0.017 s | 3.819 s +- 0.208 s | 102.1% l-d-r | x | | 84.4 ms +- 1.5 ms | 83.2 ms +- 1.0 ms | 98.6% l-d-r | x | x | 751.2 ms +- 5.0 ms | 755.8 ms +- 12.9 ms | 100.6% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.519 s +- 0.038 s | 5.526 s +- 0.057 s | 100.1% m-u | | x | 5.588 s +- 0.048 s | 5.607 s +- 0.061 s | 100.3% m-u | x | | 5.520 s +- 0.044 s | 5.546 s +- 0.059 s | 100.5% m-u | x | x | 586.6 ms +- 12.8 ms | 554.9 ms +- 21.2 ms | 94.6% <-- l-d-r | | | 629.8 ms +- 5.5 ms | 627.4 ms +- 6.6 ms | 99.6% l-d-r | | x | 6.165 s +- 0.058 s | 6.255 s +- 0.303 s | 101.5% l-d-r | x | | 270.2 ms +- 2.3 ms | 271.4 ms +- 2.7 ms | 100.4% l-d-r | x | x | 4.700 s +- 0.025 s | 1.651 s +- 0.016 s | 35.1% <-- status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 215.4 ms +- 2.3 ms | 216.5 ms +- 4.2 ms | 100.5% m-u | | x | 132.9 ms +- 1.2 ms | 132.0 ms +- 1.4 ms | 99.3% m-u | x | | 217.0 ms +- 1.9 ms | 215.4 ms +- 1.9 ms | 99.3% m-u | x | x | 108.6 ms +- 1.0 ms | 108.2 ms +- 1.5 ms | 99.6% l-d-r | | | 80.0 ms +- 1.3 ms | 80.5 ms +- 1.1 ms | 100.6% l-d-r | | x | 3.916 s +- 0.187 s | 3.966 s +- 0.236 s | 101.3% l-d-r | x | | 84.4 ms +- 3.1 ms | 83.9 ms +- 1.1 ms | 99.4% l-d-r | x | x | 758.0 ms +- 8.2 ms | 753.5 ms +- 5.0 ms | 99.4% status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.905 s +- 0.025 s | 1.910 s +- 0.044 s | 100.3% m-u | | x | 1.892 s +- 0.009 s | 1.895 s +- 0.012 s | 100.2% m-u | x | | 1.891 s +- 0.012 s | 1.902 s +- 0.018 s | 100.6% m-u | x | x | 93.3 ms +- 0.9 ms | 93.4 ms +- 0.8 ms | 100.1% l-d-r | | | 570.7 ms +- 7.8 ms | 571.9 ms +- 18.5 ms | 100.2% l-d-r | | x | 561.5 ms +- 5.2 ms | 562.9 ms +- 6.1 ms | 100.2% l-d-r | x | | 171.7 ms +- 2.6 ms | 171.9 ms +- 1.2 ms | 100.1% l-d-r | x | x | 142.7 ms +- 2.0 ms | 140.3 ms +- 1.0 ms | 98.3% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.126 s +- 0.016 s | 3.128 s +- 0.015 s | 100.1% m-u | | x | 3.014 s +- 0.068 s | 3.008 s +- 0.031 s | 99.8% m-u | x | | 3.143 s +- 0.037 s | 3.184 s +- 0.086 s | 101.3% m-u | x | x | 308.0 ms +- 1.8 ms | 308.1 ms +- 5.7 ms | 100.0% l-d-r | | | 430.8 ms +- 4.5 ms | 436.4 ms +- 8.7 ms | 101.3% l-d-r | | x | 9.676 s +- 0.127 s | 9.945 s +- 0.272 s | 102.8% l-d-r | x | | 254.2 ms +- 3.3 ms | 255.7 ms +- 3.1 ms | 100.6% l-d-r | x | x | 1.571 s +- 0.030 s | 1.555 s +- 0.014 s | 99.0% Differential Revision: https://phab.mercurial-scm.org/D4606

tests: fix a couple of drawdag.py references Differential Revision: https://phab.mercurial-scm.org/D4635

py3: fix kwargs handling in hgext/fastannotate.py Differential Revision: https://phab.mercurial-scm.org/D4588

narrow: use diffmatcher to send only new filelogs in non-ellipses widening Before this patch, when we widen a non-ellipses narrow clone, we downloads all the filelogs matching the resulting new matcher. This is same as the ellipses case but can be improved because, we don't pull new csets in non-ellipses cases, we can only download the new added files instead of downloading all the files which matches the new matcher. So, we only download files which matches the new matcher but does not matches the old matcher. There exists a match.differencematcher() which is used here. This will lead to significant amount of speedup in extending a non-ellipses narrow copy on large repos because we will download and process only the new required filelogs. The tests changes demonstrate that we are downloading now less files. Thanks to Augie for pointing that functionality of differencematcher exists in core. Differential Revision: https://phab.mercurial-scm.org/D4614

py3: add missing b'' prefixes in couple of test files These were missed in the earlier patch and caught by Yuya. # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4613

run-tests: convert the remaining os.system() call to Unicode I wasn't able to hit this path in 543a788eea2d, but I have now when I accidentally left off `--local`.

py3: partially fix pager spawning on Windows Previously, spinning up the pager crashed because the command and environment was in bytes. (See also 543a788eea2d.) Now it aborts with an invalid handle: $ HGMODULEPOLICY=py py -3 ../hg --traceback --config extensions.evolve=! Traceback (most recent call last): File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 967, in _write self.fout.write(''.join(msgs)) File "c:\Users\Matt\projects\hg\mercurial\windows.py", line 173, in write self.fp.write(s[start:end]) OSError: [WinError 6] The handle is invalid During handling of the above exception, another exception occurred: Traceback (most recent call last): File "c:\Users\Matt\projects\hg\mercurial\scmutil.py", line 164, in callcatch return func() File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 350, in _runcatchfunc return _dispatch(req) File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 930, in _dispatch return commands.help_(ui, 'shortlist') File "c:\Users\Matt\projects\hg\mercurial\commands.py", line 2930, in help_ ui.write(formatted) File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 948, in write self._writenobuf(*args, **opts) File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 960, in _writenobuf self._write(*msgs, **opts) File "c:\Users\Matt\projects\hg\mercurial\ui.py", line 969, in _write raise error.StdioError(err) mercurial.error.StdioError: [Errno 9] The handle is invalid abort: The handle is invalid The interesting bit here is that the abort message is marked with ANSI color, but the OSError is not.

censor: rename loop variable to silence pyflakes warning hgext/censor.py:92: list comprehension redefines 'c' from line 88

py3: add b'' prefixes in tests/test-hgweb-no-request-uri.t # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D4611

py3: add b'' prefixes in tests/test-hgweb-no-path-info.t # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4610

py3: add b'' prefixes in tests/test-hgweb-non-interactive.t # skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4609

py3: use codecs.encode() to encode in rot-13 encoding The other occurence will need some more love as description is bytes by default and we need to decode it and then encode it. Differential Revision: https://phab.mercurial-scm.org/D4608

py3: add two passing tests to whitelist found by buildbot The buildbot found these two new passing tests on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4607

phabricator: mark extension as experimental for now I don't want us to commit to this having a stable interface just yet. Differential Revision: https://phab.mercurial-scm.org/D4605

phabricator: fix templating bug by using hybriddict Differential Revision: https://phab.mercurial-scm.org/D4604

phabricator: add tests of templatekeyword Having tests is paying off: I found a bug and now it'll be easy to fix! Differential Revision: https://phab.mercurial-scm.org/D4603

phabricator: move extension from contrib to hgext It's well-enough tested now and widely enough used I think we should ship it. Differential Revision: https://phab.mercurial-scm.org/D4602

tests: add some basic tests of phabricator interactions This uses the vcr library to avoid hitting phabricator on every test execution. In order to generate new recordings (vcr calls them cassettes) just remove the appropriate json file, and the test will regenerate it. It's not my favorite way to test things, but it'll let us have test coverage on the phabricator extension that'll make it resilient to refactors in core and let us move it to hgext. In the future, it'd probably be better to have a docker container we can spin up for creating the vcr recordings, but for now this is enough better than nothing I'm going to declare victory. Coverage reports about 73% of the extension is now covered. Differential Revision: https://phab.mercurial-scm.org/D4601

phabricator: add support for using the vcr library to mock interactions I'll use this in an upcoming test. The decorator dancing in this is more complicated than I'd like, but it beats repeating all this code everywhere. Differential Revision: https://phab.mercurial-scm.org/D4600

keepalive: work around slight deficiency in vcr VCR's response type doesn't define the will_close attribute. Let's just have keepalive default to closing the socket if the will_close attribute is missing. Differential Revision: https://phab.mercurial-scm.org/D4599

hghave: add a checker for the vcr HTTP record/replay library I'm going to use this to write some tests of the phabricator extension. Differential Revision: https://phab.mercurial-scm.org/D4598

py3: allow run-tests.py to run on Windows This is now functional: HGMODULEPOLICY=py py -3 run-tests.py --local test-help.t --pure --view bcompare However, on this machine without a C compiler, it tries to load cext anyway, and blows up. I haven't looked into why, other than to see that it does set the environment variable. When the test exits though, I see it can't find killdaemons.py, get-with-headers.py, etc. I have no idea why these changes are needed, given that it runs on Linux. But os.system() is insisting that it take a str, and subprocess.Popen() blows up without str: Errored test-help.t: Traceback (most recent call last): File "run-tests.py", line 810, in run self.runTest() File "run-tests.py", line 858, in runTest ret, out = self._run(env) File "run-tests.py", line 1268, in _run exitcode, output = self._runcommand(cmd, env) File "run-tests.py", line 1141, in _runcommand env=env) File "C:\Program Files\Python37\lib\subprocess.py", line 756, in __init__ restore_signals, start_new_session) File "C:\Program Files\Python37\lib\subprocess.py", line 1100, in _execute_child args = list2cmdline(args) File "C:\Program Files\Python37\lib\subprocess.py", line 511, in list2cmdline needquote = (" " in arg) or ("\t" in arg) or not arg TypeError: argument of type 'int' is not iterable This is exactly how it crashes when trying to spin up a pager too. I left one instance of os.system() unchanged in _installhg(), because it doesn't get there.

py3: ensure run-tests environment is uniformly str subprocess.popen() was crashing, and when I printed out `env`, all of the keys and most of the values were str. Except these.

py3: ensure run-tests.osenvironb is actually bytes Windows doesn't have os.environb, so it was falling back to the Unicode form, and all of the accesses are trying to use bytes.