Yuya Nishihara <yuya@tcha.org> [Sat, 13 Oct 2018 14:11:12 +0200] rev 40250
py3: do I/O in bytes in test-help.t
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 07:55:34 -0400] rev 40249
tests: accept slightly different zip file in Python 3
I added some `unzip -t` here and I *think* the only change is from
Python 3 having more data in the zip file headers or something. Sigh.
Differential Revision: https://phab.mercurial-scm.org/D5075
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 07:53:51 -0400] rev 40248
webcommands: use stringutil.pprint() to repr invalid archive types
Differential Revision: https://phab.mercurial-scm.org/D5074
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 07:51:22 -0400] rev 40247
archival: don't try and fsdecode non-{bytes,str} objects
This function accepts both bytes and file-like objects.
Differential Revision: https://phab.mercurial-scm.org/D5073
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 06:34:53 -0400] rev 40246
tests: fix last failure in test-tools.t
Differential Revision: https://phab.mercurial-scm.org/D5072
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 13 Oct 2018 12:20:24 +0200] rev 40245
run-tests: run tests with as many processes as cores by default
This seems like a useful default behavior so tests run faster by default*
* Except in special circumstances where the OS/filesystem doesn't scale well
to many CPU cores (like APFS *cough* *cough*).
Differential Revision: https://phab.mercurial-scm.org/D5071
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 13 Oct 2018 12:11:45 +0200] rev 40244
run-tests: print number of tests and parallel process count
This seems like a useful output message to have.
I also sneak in a change to lower the parallel process count if it
is larger than the number of tests, as that makes no sense and output
saying we're running more tests in parallel than there exists tests would
be wonky.
Differential Revision: https://phab.mercurial-scm.org/D5070
Yuya Nishihara <yuya@tcha.org> [Sat, 13 Oct 2018 12:29:43 +0200] rev 40243
releasenotes: use stringutil.wrap() instead of handcrafted TextWrapper wrapper
It's silly to splitlines() a joined string, but we don't care the performance
here.
Martin von Zweigbergk <martinvonz@google.com> [Sat, 13 Oct 2018 00:22:05 -0700] rev 40242
match: optimize matcher when all patterns are of rootfilesin kind
Internally at Google, we use narrowspecs with only rootfilesin-kind
patterns. Sometimes there are thousands of such patterns
(i.e. thousands of tracked directories). In such cases, it can take
quite long to build and evaluate the resulting matcher.
This patch optimizes matchers that have only patterns of rootfilesin
so it instead of creating a regular expression, it matches the given
file's directory against the set of directories.
In a repo with ~3600 tracked directories, it takes about 1.35 s to
build the matcher and 2.7 s to walk the dirstate before this
patch. After, it takes 0.04 s to create the matcher and 0.87 s to walk
the dirstate.
It may be worthwhile to do similar optimizations for e.g. patterns of
type "kind:", but that's not a priority for us right now.
Differential Revision: https://phab.mercurial-scm.org/D5058
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 06:02:27 -0400] rev 40241
churn: use integer division consistently
This results in slight output changes, but it's at least consistent
between Python 2 and 3. Since the output is just bar graphs anyway,
I'm content with the changes.
Differential Revision: https://phab.mercurial-scm.org/D5063
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 05:58:16 -0400] rev 40240
churn: fix stack traces on Python 3
Differential Revision: https://phab.mercurial-scm.org/D5062
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 04:00:57 -0400] rev 40239
py3: moar passing tests
Differential Revision: https://phab.mercurial-scm.org/D5055
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 12:52:49 -0400] rev 40238
py3: one new passing tests
Differential Revision: https://phab.mercurial-scm.org/D5006
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 12:38:37 -0400] rev 40237
tests: expand our coverage of errors in Python 3 for bad extensions
What a mess.
Differential Revision: https://phab.mercurial-scm.org/D5005
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 12:32:12 -0400] rev 40236
tests: add lots of globs and conditional output lines
The test now passes on Python 3. I'm going to do one follow-up where
I'll adjust some grep calls to improve our checking of things on
Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5004
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 04:00:47 -0400] rev 40235
releasenotes: fix remaining bytes/unicode issues caught by tests
All tests now pass.
Differential Revision: https://phab.mercurial-scm.org/D5054
Augie Fackler <augie@google.com> [Sat, 13 Oct 2018 03:27:21 -0400] rev 40234
relnotes: port to Python 3
The big annoyance here was having to feed textwrap unicodes instead of
bytes, but it all seems to work.
Differential Revision: https://phab.mercurial-scm.org/D5053
Yuya Nishihara <yuya@tcha.org> [Sat, 13 Oct 2018 11:01:38 +0200] rev 40233
py3: use __code__ instead of func_code to test if func is cachable
Surprisingly, this fixes test-cache-abuse.t, which would generate a different
revbranch cache if filtered set had no cache.
Yuya Nishihara <yuya@tcha.org> [Sat, 13 Oct 2018 10:58:06 +0200] rev 40232
py3: pass in system string to vars(branchmap).__contains__()
Mark Thomas <mbthomas@fb.com> [Sat, 13 Oct 2018 08:48:59 +0000] rev 40231
test-bookmarks-pushpull: use correct tmp dir for no-bm-move.sh
test-bookmarks-pushpull.t writes a temporary file in $TESTDIR instead of
$TESTTMP. Make it use $TESTTMP instead.
Differential Revision: https://phab.mercurial-scm.org/D5057
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 05:17:28 +0300] rev 40230
py3: add some glob to make output compatible with python 3
This makes test-pathconflicts-basic.t and test-origbackup-conflict.t passing
on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5042
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 05:14:23 +0300] rev 40229
py3: add a b'' prefix in tests/test-requires.t
This makes the test pass on Python 3.
# skip-blame because just b'' prefix.
Differential Revision: https://phab.mercurial-scm.org/D5041
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 05:12:20 +0300] rev 40228
py3: use '%d' for rev nums instead of '%s'
This makes test-close-head.t pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5040
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 05:03:24 +0300] rev 40227
py3: more globing of things to make output compatible between py2 and py3
This also makes the test-ssh-repoerror.t pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5038
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:54:35 +0300] rev 40226
py3: use stringutil.pprint() instead of '%r'
This makes test-pager-legacy.t pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5037
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:51:09 +0300] rev 40225
py3: use '%d' for integers instead of '%s'
Differential Revision: https://phab.mercurial-scm.org/D5036
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:47:51 +0300] rev 40224
py3: add some py3 specific output to test
This makes the test work on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5035
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:42:14 +0300] rev 40223
py3: use stringutil.pprint() in color.py instead of '%r'
This fixes the test-status-color.t on python 3.
Differential Revision: https://phab.mercurial-scm.org/D5034
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:36:04 +0300] rev 40222
py3: add b'' prefixes in tests/test-progress.t
This makes the test pass on Python 3.
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5033
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:29:03 +0300] rev 40221
py3: add b'' prefixes to tests/test-keyword.t
This test is now pretty close to passing on Python 3.
# skip-blame because just b'' prefixes.
Differential Revision: https://phab.mercurial-scm.org/D5032
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:24:19 +0300] rev 40220
py3: use stringutil.pprint() if we are printing bool values
Differential Revision: https://phab.mercurial-scm.org/D5031
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:21:02 +0300] rev 40219
py3: glob some difference between py2 and py3 output
On py2, the directory names are under quotes and on py3 they are not and I don't
know why.
Differential Revision: https://phab.mercurial-scm.org/D5030
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:09:34 +0300] rev 40218
py3: byteify tests/wireprotosimplecache.py
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5029
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 04:08:53 +0300] rev 40217
py3: add b'' prefixes in tests/test-help.t
This takes us close to make the test passing.
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5028
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 03:59:15 +0300] rev 40216
py3: use sys.stdout.buffer for binary output in tests/svnxml.py
Just following 6c0b1d964537ab62d76c208f5f04ab414814c94e here.
Differential Revision: https://phab.mercurial-scm.org/D5027
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 03:58:20 +0300] rev 40215
py3: use '%s' instead of %r in hgext/convert/subversion.py
Using '%r' on bytes in python 3 adds b'' prefixes to output which is bad. I
don't see a reason why we want to repr() the string here, so just switched to
'%s'. This fixes some output failure in one of the subversion test.
Differential Revision: https://phab.mercurial-scm.org/D5026
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 03:46:57 +0300] rev 40214
py3: make test-contrib-perf.t work on python 3
Differential Revision: https://phab.mercurial-scm.org/D5025
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 03:32:12 +0300] rev 40213
py3: add 3 more passing tests to whitelist caught by buildbot
Differential Revision: https://phab.mercurial-scm.org/D5024
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 00:35:37 +0300] rev 40212
py3: add some r'' prefixes in hgext/narrow/narrowwirepeer.py
This fixes test-narrow-widen-no-ellipsis.t on Python 3.
# skip-blame because just r'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5023
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 19:49:02 +0200] rev 40211
py3: r'' prefix default values for mimetypes.guess_mime()
As suggested by @yuja in D4967.
Differential Revision: https://phab.mercurial-scm.org/D5020
Mark Thomas <mbthomas@fb.com> [Fri, 12 Oct 2018 20:03:29 +0000] rev 40210
absorb: update help text
Update the help text to reflect the recent changes to absorb in D4999.
The `-p` option actually only affects the `-a` option: without `-a`, the
changes are printed; with `-a` the changes are only printed if `-p` is
specified. Reword the help text for `-p` to reflect this.
Differential Revision: https://phab.mercurial-scm.org/D5022
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Fri, 12 Oct 2018 16:45:46 +0900] rev 40209
fsmonitor: use vfs instead of opener (issue5938)
"opener" of localrepository object was dropped at Mercurial 4.3 (or
a7e210167c28). "vfs" should be used instead.
wlock is required to write into a file under .hg directory.
For efficiency, we should change _cmpsets() from:
1. acquire wlock
2. open log file under .hg directory with write mode
3. compare between result of watchman and Mercurial's dirstate logic
4. write out error info into a file, if error is detected
5. release wlock
to:
1. compare between result of watchman and Mercurial's dirstate logic
2. acquire wlock, if error is detected
3. open and write error info into a file
4. release wlock
But this is another issue.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Fri, 12 Oct 2018 16:07:08 +0900] rev 40208
tests: configure fsmonitor.mode=paranoid always if fsmonitor is used
This forces fsmonitor extension execute "paranoid" code path.
Strict speaking, we should make fsmonitor-run-tests.py accept own
specific options, but there is no code path, which is disabled in
"paranoid" mode, at least now. Therefore, this solution seems
reasonable enough.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Mon, 08 Oct 2018 13:12:06 +0900] rev 40207
tests: drop files from exclusion list in test-check-module-imports.t
Now, these files has no style issue at checking with import-checker.py.
BTW, tests/test-imports-checker.t is still excluded, because almost
all code fragments in it has intentional importing style violation.
Using NO_CHECK_EOF instead of EOF as heredoc limit mark can make
import-checker.py ignore such fragments. But keeping these fragments
checkable seems still useful: for example, test-imports-checker.t can
be used to test whether import-checker.py can detect erroneous code
fragment in test script as expected.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Mon, 08 Oct 2018 12:03:04 +0900] rev 40206
tests: use NO_CHECK_EOF for fragments used to test importing via extension
test-extension.t is excluded in test-check-module-imports.t, because
import-checker.py reports that some python code fragments in it does
not import modules in recommended style.
These code fragments are used to test importing modules relatively and
absolutely via extension (e.g. issue4029, issue5208, and so on). Test
specific package hierarchy becomes complicated for this purpose, and
it seems reasonable to avoid checking these code fragments with
import-checker.py.
But on the other hand, other code fragments in test-extension.t should
be checked by import-checker.py.
Therefore, this patch uses NO_CHECK_EOF limit mark only for python
code fragments, which are used to test importing via extension in
test-extension.t. NO_CHECK_EOF limit mark tells import-checker.py that
this code fragment should be ignored, via testparseutil.py.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Mon, 08 Oct 2018 10:30:59 +0900] rev 40205
tests: use NO_CHECK_EOF for fragments having intentional error
Some test scripts are excluded in test-check-module-imports.t, because
import-checker.py reports that code fragments embedded in these test
scripts have syntax error. But these syntax error is intentional.
This patch uses NO_CHECK_EOF instead of EOF as heredoc limit mark for
such fragments, in order to make import-checker.py ignore
them. NO_CHECK_EOF limit mark tells import-checker.py that this code
fragment should be ignored, via testparseutil.py.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Mon, 08 Oct 2018 10:45:36 +0900] rev 40204
tests: fix style issue of importing order in test-lock.py
test-lock.py is excluded in test-check-module-imports.t, because
import-checker.py reports that some stdlibs are imported after local
silenttestrunner module.
I can not found out any reason why tests/silenttestrunner.py should be
imported before some stdlibs from the point of functionality view.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Mon, 08 Oct 2018 10:10:06 +0900] rev 40203
tests: fix style issue of importing hgweb in embedded code fragments
Some test scripts are excluded in test-check-module-imports.t, because
import-checker.py reports that hgweb and/or hgwebdir of
mercurial.hgweb are not imported in recommended style.
To fix this issues, this patch make python code fragments embedded in
these files import hgweb from mercurial package at first, and refer
hgweb and hgwebdir via imported hgweb.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 18:20:17 +0200] rev 40202
py3: encode str to bytes
These fields are str on Python 2 and 3. This module doesn't import any Mercurial
modules. So I just did the str -> bytes inline.
Differential Revision: https://phab.mercurial-scm.org/D5010
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 19:00:24 +0200] rev 40201
py3: sprinkle statprof.py with utf-8 encoding
Trying to chase the long tail of unicode badness in this file due to
frame / code objects holding str everywhere.
Differential Revision: https://phab.mercurial-scm.org/D5016
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 19:17:22 +0200] rev 40200
py3: flush stdout
This avoids buffering due to mixed print() and sys.stdout.write() by
profiling code.
And with this, test-profile.t passes on Python 3 \o/.
Honestly, I'd be shocked if there weren't more Python 3 encoding issues lingering
in the profiling code. That code isn't super high quality and doesn't seem to have
thorough test coverage.
Differential Revision: https://phab.mercurial-scm.org/D5018
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 19:07:12 +0200] rev 40199
py3: use raw strings in statprof.py
This allows main() and argument parsing to work.
Differential Revision: https://phab.mercurial-scm.org/D5017
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 18:51:10 +0200] rev 40198
py3: switch from print(..., file=) to write()
Because Python 3's print() doesn't like bytes.
Differential Revision: https://phab.mercurial-scm.org/D5015
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 18:40:32 +0200] rev 40197
py3: use %d in a few places
Differential Revision: https://phab.mercurial-scm.org/D5014
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 18:37:42 +0200] rev 40196
py3: use raw strings for stack names
These are used for simple compares. We don't need to normalize to bytes.
As part of this, I expanded a set literal so entries are 1 per line.
Differential Revision: https://phab.mercurial-scm.org/D5013
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 18:30:40 +0200] rev 40195
py3: use sysbytes for converting code attributes
Differential Revision: https://phab.mercurial-scm.org/D5012
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 18:26:10 +0200] rev 40194
py3: use write() instead of print()
Because print() expects str and we want to write bytes.
There should be no functional changes as part of this.
Differential Revision: https://phab.mercurial-scm.org/D5011
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 17:35:54 +0200] rev 40193
py3: use %d to format ints
Differential Revision: https://phab.mercurial-scm.org/D5009
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 17:31:10 +0200] rev 40192
py3: convert sorting field to sysstr
This is used as part of an attribute name check and needs to be native str.
Differential Revision: https://phab.mercurial-scm.org/D5008
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 17:25:02 +0200] rev 40191
py3: encode json output to bytes and use write()
Differential Revision: https://phab.mercurial-scm.org/D5007
Mark Thomas <mbthomas@fb.com> [Fri, 12 Oct 2018 17:48:40 +0000] rev 40190
absorb: prompt user to accept absorb changes by default
Change the default absorb behaviour to print the changes out and then prompt
the user if they want to absorb the changes or not.
Providing the `-p` option still prints the result and exits. A new `-a` option
is provided which applies the changes without printing them, giving the
equivalent of the old behaviour.
Differential Revision: https://phab.mercurial-scm.org/D4999
Mark Thomas <mbthomas@fb.com> [Fri, 12 Oct 2018 15:46:52 +0000] rev 40189
templatefuncs: add truncate parameter to pad
Add a truncate option to pad that additionally truncates the text to the pad
width if it is wider.
Since color codes can cause a problem with this, when the text is truncated,
the color codes are also stripped. Users of the truncate option should label
the text outside the pad.
Differential Revision: https://phab.mercurial-scm.org/D5000
Mark Thomas <mbthomas@fb.com> [Fri, 12 Oct 2018 15:46:51 +0000] rev 40188
absorb: print summary of changesets affected
Differential Revision: https://phab.mercurial-scm.org/D4998
Mark Thomas <mbthomas@fb.com> [Fri, 12 Oct 2018 13:35:58 +0000] rev 40187
absorb: use a formatter to generate output
Change absorb to use a formatter to generate its output. This allows the use
of templates to customize the output.
Differential Revision: https://phab.mercurial-scm.org/D4997
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 12:31:17 -0400] rev 40186
tests: fix up test-bad-extension.t's inline extension
Upcoming patches will be simpler, I think.
Differential Revision: https://phab.mercurial-scm.org/D5003
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 11:44:27 -0400] rev 40185
polib: update to latest release 1.0.7 (upstream rev d75ce6dbbc2a)
# no-check-commit third-party code that doesn't match our style
Differential Revision: https://phab.mercurial-scm.org/D5001
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 10:01:09 -0400] rev 40184
tests: glob away some annoying py3 differences
Differential Revision: https://phab.mercurial-scm.org/D4994
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 10:00:43 -0400] rev 40183
tests: fix a repr on python3 in test-extension.t
Differential Revision: https://phab.mercurial-scm.org/D4993
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 10:00:14 -0400] rev 40182
tests: so many b prefixes in test-extension.t
# skip-blame because it's b prefixes
Differential Revision: https://phab.mercurial-scm.org/D4992
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 09:59:11 -0400] rev 40181
tests: ensure print() statements in test-extension.t all flush
Differential Revision: https://phab.mercurial-scm.org/D4991
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 12 Oct 2018 16:19:21 +0200] rev 40180
revlog: update pure nodecache start lookup offset on insertion
test-storage.py is failing in pure builds because the population of the node
cache isn't accounting for the new starting offset after a revlog insertion.
This commit updates the node cache start offset to account for insertions.
I'm not 100% convinced this is the ideal solution. But it works and seems
correct.
Differential Revision: https://phab.mercurial-scm.org/D4996
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 13:57:42 -0700] rev 40179
exchangev2: use filesdata
filesdata is a more efficient mechanism for bulk fetching files data for a
range of changesets. Let's use it in exchangev2.
With this change, a client performing a full clone of mozilla-unified
transmits substantially fewer bytes across the wire:
before: 139,124,863 bytes sent
after: 20,522,499 bytes sent
The bulk of the remaining bytes is likely the transfer of ~1M nodes for
changesets and manifests. We can eliminate this by making requests in
terms of node ranges instead of explicit node lists...
Differential Revision: https://phab.mercurial-scm.org/D4982
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 12:54:39 -0700] rev 40178
wireprotov2: define and implement "filesdata" command
Previously, the only way to access file revision data was the
"filedata" command. This command is useful to have. But, it only
allowed resolving revision data for a single file. This meant that
clients needed to send 1 command for each tracked path they were
seeking data on. Furthermore, those commands would need to enumerate
the exact file nodes they wanted data for.
This approach meant that clients were sending a lot of data to
remotes in order to request file data. e.g. if there were 1M
file revisions, we'd need at least 20,000,000 bytes just to encode
file nodes! Many clients on the internet don't have that kind of
upload capacity.
In order to limit the amount of data that clients must send, we'll
need more efficient ways to request repository data.
This commit defines and implements a new "filesdata" command. This
command allows the retrieval of data for multiple files by specifying
changeset revisions and optional file patterns. The command figures
out what file revisions are "relevant" and sends them in bulk.
The logic around choosing which file revisions to send in the case of
haveparents not being set is overly simple and will over-send files. We
will need more smarts here eventually. (Specifically, the client will
need to tell the server which revisions it knows about.) This work
is deferred until a later time.
Differential Revision: https://phab.mercurial-scm.org/D4981
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 10:31:36 -0700] rev 40177
wireprotov2: extract file object emission to own function
An upcoming commit will introduce another caller.
Differential Revision: https://phab.mercurial-scm.org/D4980
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 18:17:12 -0700] rev 40176
wireprotov2: change how revisions are specified to changesetdata
Right now, we have a handful of arguments for specifying the revisions
whose data should be returned. Defining how all these arguments
interact when various combinations are present is difficult.
This commit establishes a new, generic mechanism for specifying
revisions. Instead of a hodgepodge of arguments defining things, we
have a list of dicts that specify revision selectors. The final set
of revisions is a union of all these selectors.
We implement support for specifying revisions based on:
* An explicit list of changeset revisions
* An explicit list of changeset revisions plus ancestry depth
* A DAG range between changeset roots and heads
If you squint hard enough, this problem has already been solved by
revsets. But I'm reluctant to expose revsets to the wire protocol
because that would require servers to implement a revset parser.
Plus there are security and performance implications: the set
of revision selectors needs to be narrowly and specifically tailored
for what is appropriate to be executing on a server. Perhaps there
would be a way for us to express the "parse tree" of a revset
query, for example. I'm not sure. We can explore this space another
time. For now, the new mechanism should bring sufficient flexibility
while remaining relatively simple.
The selector "types" are prefixed with "changeset" because I plan
to add manifest and file-flavored selectors as well. This will enable
us to e.g. select file revisions based on a range of changeset
revisions.
Differential Revision: https://phab.mercurial-scm.org/D4979
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 17:54:14 -0700] rev 40175
wireprotov2: stop sending phase updates for base revisions
This feature is broken and doesn't work properly in all scenarios.
e.g. if we have the following DAGs:
client server
D draft
C draft C draft
B draft B public
A public A public
The current code would only send the phase data for C. The
client wouldn't see that B moved from draft to public.
This feature will be restored in a future commit. For now, it is
making refactoring of how revisions are specified in the wire protocol
a bit difficult...
Differential Revision: https://phab.mercurial-scm.org/D4978
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 09:47:52 +0200] rev 40174
debugcommands: support wrapping long lines
If a line within a block is indented more than the line that came before,
we automatically concatenate it with the previous line. This allows us to
pretty format data. This will make tests easier to read.
At some point we may just want to evaluate entire blocks as Python
code or something, as even with this change, things aren't perfect, as we
can't e.g. have formatting like:
foo eval:[
True
]
But this is strictly better than before, where we couldn't wrap long lines.
Differential Revision: https://phab.mercurial-scm.org/D4977
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 13:17:00 -0700] rev 40173
exchangev2: honor server advertised manifestdata recommended batch size
Let's plug the client up to the server-advertised recommended batch size
for manifestdata requests.
Differential Revision: https://phab.mercurial-scm.org/D4976
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 17:45:51 -0700] rev 40172
wireprotov2: advertise recommended batch size for requests
Currently, exchangev2 hardcodes the batch size for how many revisions
to fetch per command request. A single value is not appropriate
for every repository because some repositories may have a drastically
different "shape" from other repositories. e.g. a repo with lots of
small files may benefit from larger batch sizes than a repo with lots
of large files. And depending on caching used by the server, the server
may wish to control the number of commands (to e.g. mitigate overhead
of following content redirects).
This commit teaches wireprotov2 commands to declare extra metadata
which is advertised as part of the command descriptor. The manifestdata
command has been taught to advertise a recommended batch size for
requests.
Differential Revision: https://phab.mercurial-scm.org/D4975
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 13:07:28 -0700] rev 40171
httppeer: expose API descriptor on httpv2peer
The API descriptor in wireprotov2 is much more expressive than
space-delimited tokens and it will be difficult to define methods
to query it in all of the ways we'll want to query it. So let's
just declare defeat and expose the API descriptor on the peer
instance.
As part of this, we define a new interface for version 2 peers,
fulfilling a TODO in the process.
Differential Revision: https://phab.mercurial-scm.org/D4974
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 09:26:05 +0200] rev 40170
tests: use baseurl instead of advertisedbaseurl
The distinction matters for e.g. hosts behind load balancers. But for
the test environment, it doesn't matter.
For whatever reason, advertisedbaseurl is resolving to
http://1.0.0.127.in-addr.arpa:$HGPORT on my MBP. This hostname fails
to resolve, causing the test to fail. No clue what's up with that
behavior.
Differential Revision: https://phab.mercurial-scm.org/D4973
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 09:23:55 -0400] rev 40169
py3: another one started passing
Differential Revision: https://phab.mercurial-scm.org/D4990
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 06:55:11 -0400] rev 40168
py3: one more passing test
Differential Revision: https://phab.mercurial-scm.org/D4989
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 06:50:01 -0400] rev 40167
scmutil: wrap locker information in bytestr before repr()ing it
Differential Revision: https://phab.mercurial-scm.org/D4988
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 06:50:46 -0400] rev 40166
py3: more passing tests
Differential Revision: https://phab.mercurial-scm.org/D4987
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 06:35:55 -0400] rev 40165
py3: whitelist another passing test
Differential Revision: https://phab.mercurial-scm.org/D4986
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 06:22:15 -0400] rev 40164
obsolete: convert error string to a sysstr
This should be okay because we're just %-formatting a hash into a
localized string.
Differential Revision: https://phab.mercurial-scm.org/D4985
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 06:39:32 -0400] rev 40163
tests: ensure the python 3 passing test list stays sorted
Differential Revision: https://phab.mercurial-scm.org/D4984
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 06:35:33 -0400] rev 40162
py3: sort passing test list
Differential Revision: https://phab.mercurial-scm.org/D4983
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 23:07:23 +0200] rev 40161
py3: tweak stdout writing in test-hgweb-no-path-info.t
We want to write bytes for convenience. This requires sys.stdout.buffer.
But using sys.stdout.buffer introducing buffered output. So we sprinkle
code with sys.stdout.flush() to force immediate writes.
After all that, Python 3 was emitting b'' prefixed output for errors.
So we only print errors if there were some. There aren't, so b'' don't
come into play and output is identical in Python 2 and 3.
Differential Revision: https://phab.mercurial-scm.org/D4972
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 22:53:44 +0200] rev 40160
py3: use .+ instead of .* in regexp pattern
Python 3.7 changed the behavior of re.sub(). See
https://bugs.python.org/issue33585.
The new code should work on old and new Pythons.
Differential Revision: https://phab.mercurial-scm.org/D4971
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 22:26:12 +0200] rev 40159
py3: coerce bytestr to bytes to appease urllib.parse.quote_from_bytes()
Differential Revision: https://phab.mercurial-scm.org/D4969
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 21:47:39 +0200] rev 40158
py3: pass str and return bytes from mimetypes.guess_type()
This function wants a str (which represents a path) and returns a str.
We normalize input to str and output to bytes.
Differential Revision: https://phab.mercurial-scm.org/D4967
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 22:27:52 +0200] rev 40157
py3: use bytes literal in test-hgweb-json.t
# skip-blame just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D4970
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 22:01:54 +0200] rev 40156
py3: convert diff opcode name to bytes
Differential Revision: https://phab.mercurial-scm.org/D4968
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 21:22:43 +0200] rev 40155
py3: byteify hgweberror.py
# skip-blame just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4966
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 21:18:58 +0200] rev 40154
py3: encode JSON str to bytes
json.dumps() will emit UTF-8 str on Python 2 and 3. Use sysbytes to
force the .encode('utf-8') on Python 3 and no-op on Python 2.
Differential Revision: https://phab.mercurial-scm.org/D4965
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 11 Oct 2018 21:05:03 +0200] rev 40153
py3: use b'' in test-check-interfaces.py
# skip-blame just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4964
Yuya Nishihara <yuya@tcha.org> [Fri, 12 Oct 2018 06:22:43 +0200] rev 40152
revlog: optimize ancestors() to not check filtered revisions for each
While reviewing the Rust implementation, I noticed iter(ancestors) doesn't
need to check filtering state for each parent revision. And doing that appears
to have some measurable perf win.
$ hg perfancestors -R mercurial
(orig) wall 0.038093 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
(this) wall 0.024795 comb 0.020000 user 0.020000 sys 0.000000 (best of 117)
Matt Harbison <matt_harbison@yahoo.com> [Thu, 11 Oct 2018 21:51:17 -0400] rev 40151
phabricator: drop support for the legacy phabricator.auth.token config (BC)
The test for this broke in dc82ad1b7f77 when statistics started being tracked.
It wasn't noticed because none of the bots have the vcr module installed. It
looks like the custom_patches argument should patch in the custom
httpconnection, and I can't figure out what is going on.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 11 Oct 2018 22:39:11 -0400] rev 40150
absorb: print '{rev}:' as a prefix to the hash
I don't see a sane way to print the summary to identify the changed commit, but
this at least makes it a bit easier to identify a commit when a group of them
changes.
muxator <a.mux@inwind.it> [Thu, 11 Oct 2018 19:02:42 +0200] rev 40149
packaging: fix "make centos{5,6,7}"
Without this change, make centos{5,6,7} fails with error:
cp: cannot stat '<basedir>/rpmbuild/RPMS/*/*': No such file or directory
This change is the exact equivalent of bc4bbc42899a (which applied to Fedora).
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 10 Oct 2018 19:46:13 +0300] rev 40148
narrow: don't compress the bundle2 when sending 'error:abort'
This is similar to waht getbundle() does and also explicitly specifies that we
should get a compressed bundle2 in normal cases when not sending 'error:abort'.
Differential Revision: https://phab.mercurial-scm.org/D4934
Taapas Agrawal <taapas2897@gmail.com> [Thu, 11 Oct 2018 03:38:23 +0530] rev 40147
push: add "remote" to 'repository changed while pushing' messages (issue5971)
Differential Revision: https://phab.mercurial-scm.org/D4933
Augie Fackler <augie@google.com> [Thu, 11 Oct 2018 07:27:24 -0400] rev 40146
fuzz: try setting PYTHONNOUSERSITE=1 to avoid loading site-packages
Right now the fuzzer is crashing trying to look up the home dir for
uid 0, which is breaking in the fuzz environment.
Differential Revision: https://phab.mercurial-scm.org/D4936
Boris Feld <boris.feld@octobus.net> [Wed, 03 Oct 2018 10:46:56 +0200] rev 40145
perf: extract result formatting in its own function
This will make it easier to reuse in another changesets.
Boris Feld <boris.feld@octobus.net> [Tue, 02 Oct 2018 18:54:06 +0200] rev 40144
perf: extract the timing of a section in a context manager
This makes it easier to reuse it in other (future) part of the code that
requires their own time management.
Boris Feld <boris.feld@octobus.net> [Mon, 01 Oct 2018 17:23:54 +0200] rev 40143
pref: support negative indexing in perfrevlogrevisions
This is useful to check for a consistent final slice in multiple different repositories.
Boris Feld <boris.feld@octobus.net> [Mon, 01 Oct 2018 17:37:38 +0200] rev 40142
perf: accept formatter option for perfmanifest
Boris Feld <boris.feld@octobus.net> [Mon, 01 Oct 2018 17:53:47 +0200] rev 40141
perf: fix -T json
The previous code was mixing formatting and data, breaking `-T json` with
unexpected data. We fix the issue and add a test to prevent future regression.
Boris Feld <boris.feld@octobus.net> [Mon, 01 Oct 2018 17:37:53 +0200] rev 40140
formatter: more details on assertion failure
This is useful when the assertion fails.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 10 Oct 2018 23:19:42 -0700] rev 40139
wireprotov2: raise ProgrammingError on unknown action
Suggested by @durin42 in review of D4923.
Differential Revision: https://phab.mercurial-scm.org/D4935
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 17:24:28 -0700] rev 40138
wireprotov2: send content encoded frames from server
Now that we have support for negotiating encodings and configuring
an encoder, we can start sending content encoded frames from the
server.
This commit teaches the wireprotov2 server code to send content
encoded frames.
On the mozilla-unified repository with zstd enabled peers, this change
reduces the total amount of data transferred from server to client
drastically:
befor: 7,190,995,812 bytes
after: 1,605,508,691 bytes
Differential Revision: https://phab.mercurial-scm.org/D4927
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 15:19:32 -0700] rev 40137
wireprotov2: raise exception in objects() if future has been resolved
Differential Revision: https://phab.mercurial-scm.org/D4926
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 05 Oct 2018 23:49:18 +0000] rev 40136
wireprotov2: don't emit empty frames
Staring at logs revealed the presence of empty frames that should have
contained payload. Let's stop that from happening.
Differential Revision: https://phab.mercurial-scm.org/D4925
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 05 Oct 2018 10:29:36 -0700] rev 40135
wireprotov2: remove functions for creating response frames from bytes
All code in the actual server uses oncommandresponsereadyobjects().
Test code was ported to that method. This resulted in a handful of
subtle test changes.
Differential Revision: https://phab.mercurial-scm.org/D4924
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 05 Oct 2018 09:23:06 -0700] rev 40134
wireprotov2: handle noop action
This action can be returned from the client reactor. We should
handle it.
Differential Revision: https://phab.mercurial-scm.org/D4923
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 17:00:16 -0700] rev 40133
wireprotov2: send protocol settings frame from client
Now that we have client and server reactor support for protocol
settings and encoding frames, we can start to send them out over
the wire!
This commit teaches the client reactor to send out a protocol
settings frame when needed. The httpv2 peer has been taught to
gather a list of supported content encoders and to advertise them
through the client reactor.
Because the client is now sending new frame types by default, this
constitutes a compatibility break in the framing protocol. The
media type version has been bumped accordingly. This will ensure
existing clients won't attempt to send the new frames to old
servers not supporting this explicit media type. I'm not bothering
with the BC annotation because everything wireprotov2 is highly
experimental and nobody should be running a server yet.
Differential Revision: https://phab.mercurial-scm.org/D4922
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 17:10:59 -0700] rev 40132
wireprotov2: define and use stream encoders
Now that we have basic support for defining stream encoding, it is
time to start doing something with it.
We define various classes implementing stream encoders/decoders for
the defined encoding profiles. This is relatively straightforward.
We teach the inputstream and outputstream classes how to encode,
decode, and flush data.
We then teach the clientreactor how to filter received data through
the inputstream decoder.
One of the features of the framing format is that streams can span
requests. This is a differentiating feature from say HTTP/2, which
associates streams with requests. By allowing streams to span requests,
we can reuse compression context data across requests/responses. But
in order to do this, we need a mechanism to "flush" the encoder at
logical boundaries so that receivers receive all data where it is
expected. And a "flush" event is distinct from a "finish" event from
the perspective of certain compressors because a "flush" will retain
compression context state whereas a "finish" operation will not. This
is why encoders have both a flush() and a finish() and each uses
specific flushing semantics on the underlying compressor.
The added tests verify various behavior of decoders via clientreactor.
These tests do test some compression behavior via use of outputstream.
But for all intents and purposes, server reactor support for encoding
is not yet implemented.
Differential Revision: https://phab.mercurial-scm.org/D4921
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 04 Oct 2018 17:39:16 -0700] rev 40131
wireprotov2: establish dedicated classes for input and output streams
Streams are unidirectional. As part of implementing encoding/decoding
support, it became clear that it didn't make sense for a generic
"stream" class to hold functionality related to both encoding and
decoding. So we create new classes to represent the flavor of
stream.
Differential Revision: https://phab.mercurial-scm.org/D4920
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 04 Oct 2018 17:17:57 -0700] rev 40130
wireprotov2: pass ui into clientreactor and serverreactor
This will allow us to use config options to influence compression
settings.
Differential Revision: https://phab.mercurial-scm.org/D4919
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 04 Oct 2018 16:44:21 -0700] rev 40129
wireprotov2: handle stream encoding settings frames
Like what we just did for the server reactor, we teach the client
reactor to handle stream encoding settings frames. The code is
very similar.
We define a method on the stream class to handle processing the data
within the decoded frames. However, it doesn't yet do anything useful.
Differential Revision: https://phab.mercurial-scm.org/D4918
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 04 Oct 2018 15:43:21 -0700] rev 40128
wireprotov2: document client reactor actions
We should document these so consumers have an easier life.
Differential Revision: https://phab.mercurial-scm.org/D4917
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 04 Oct 2018 16:26:45 -0700] rev 40127
wireprotov2: handle sender protocol settings frames
We teach the server reactor to handle the optional sender protocol
settings frames, which can only be sent at the beginning of frame
exchange.
Right now, we simply decode the data and record the sender protocol
settings on the server reactor instance: we don't yet do anything
meaningful with the data.
Differential Revision: https://phab.mercurial-scm.org/D4916
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 04 Oct 2018 14:05:16 -0700] rev 40126
wireprotov2: update stream encoding specification
The encoding of data within streams in the frame-based protocol is
not yet defined or implemented. This means that all data in wire
protocol version 2 is currently being sent out raw, without
compression. That's obviously not ideal.
This commit formalizes the beginnings of stream encoding support
in the protocol.
I suspect we'll change behavior substantially in the future. My goal
is to get something landed so we can use compression. We can build
out more robust support later.
Because the frame type ID changed, this is strictly BC. But existing
code wasn't using the frame. I'll bump the framing protocol version
later once code is introduced to use the new frame.
Differential Revision: https://phab.mercurial-scm.org/D4915
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 04 Oct 2018 15:08:42 -0700] rev 40125
cborutil: cast bytearray to bytes
This code didn't like passing in bytearray instances. Let's cast
bytearray to bytes so it works.
Differential Revision: https://phab.mercurial-scm.org/D4914
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 17:06:24 -0700] rev 40124
tests: disable zstd in test
This makes the test pass in pure installs.
Differential Revision: https://phab.mercurial-scm.org/D4913
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 17:20:41 -0700] rev 40123
wireprotov2: remove "compression" from capabilities response
This is not used. And future commits will change how this mechanism
works. Let's remove it.
As a bonus, this fixes some test failures on pure installs (due to
zstd references).
Differential Revision: https://phab.mercurial-scm.org/D4912
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 08 Oct 2018 16:27:40 -0700] rev 40122
zstandard: vendor python-zstandard 0.10.1
This was just released.
The upstream source distribution from PyPI was extracted. Unwanted
files were removed.
The clang-format ignore list was updated to reflect the new source
of files.
setup.py was updated to pass a new argument to python-zstandard's
function for returning an Extension instance. Upstream had to change
to use relative paths because Python 3.7's packaging doesn't
seem to like absolute paths when defining sources, includes, etc.
The default relative path calculation is relative to setup_zstd.py
which is different from the directory of Mercurial's setup.py.
The project contains a vendored copy of zstandard 1.3.6. The old
version was 1.3.4.
The API should be backwards compatible and nothing in core should
need adjusted. However, there is a new "chunker" API that we
may find useful in places where we want to emit compressed chunks
of a fixed size.
There are a pair of bug fixes in 0.10.0 with regards to
compressobj() and decompressobj() when block flushing is used. I
actually found these bugs when introducing these APIs in Mercurial!
But existing Mercurial code is not affected because we don't
perform block flushing.
# no-check-commit because 3rd party code has different style guidelines
Differential Revision: https://phab.mercurial-scm.org/D4911
Yuya Nishihara <yuya@tcha.org> [Tue, 25 Sep 2018 20:55:03 +0900] rev 40121
rust-chg: install signal handlers to forward signals to server
I use sync::Once as a synchronization primitive because it's quite easy
to use, and is good enough to prevent data race in these C functions.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 22:19:49 +0900] rev 40120
rust-chg: remove SIGCHLD handler which won't work in oxidized chg
Since pager is managed by the Rust part, the C code doesn't know the pager
pid. I could make the Rust part teach the pid to C, but still installing
SIGCHLD handler seems horrible idea since we no longer use handcrafted
low-level process management functions.
Instead, I'm thinking of adding async handler to send SIGPIPE at the exit
of the pager.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 22:04:57 +0900] rev 40119
rust-chg: extract signal handlers from chg/procutil.c
abortmsgerrno() and debugmsg() are removed, and the public interface instead
returns success/error status. Since signal handlers can't propagate errors,
the result of kill() is just ignored.
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 23:19:49 +0900] rev 40118
help: document about "version" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 23:14:21 +0900] rev 40117
help: document about "tags" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 23:12:04 +0900] rev 40116
help: document about "status" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 23:05:00 +0900] rev 40115
help: document about "resolve" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 23:00:50 +0900] rev 40114
help: document about "paths" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 22:56:37 +0900] rev 40113
help: document about "identify" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 22:50:12 +0900] rev 40112
help: document about "grep" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 07 Oct 2018 17:35:25 +0900] rev 40111
chgserver: catch Abort while parsing early args to shut down cleanly
_loadnewui() calls dispatcher functions, which may raise Abort if unparsable
arguments are passed in. The server should catch such errors and translate
them to the "exit 255" instruction so the client can finish the IPC session
cleanly.
Spotted while porting the chg client to Rust.
Yuya Nishihara <yuya@tcha.org> [Fri, 05 Oct 2018 22:08:37 +0900] rev 40110
chg: upgrade client to use "setumask2" command
No compatibility code is added to the client side, since it's unlikely for
new client to communicate with the old server.
Yuya Nishihara <yuya@tcha.org> [Thu, 04 Oct 2018 23:25:55 +0900] rev 40109
chgserver: add "setumask2" command which uses correct message frame
The first 4 bytes should be a length field, not a value. Spotted while
porting chg functions to the Rust one.
muxator <a.mux@inwind.it> [Tue, 09 Oct 2018 22:29:10 +0200] rev 40108
packaging: "make deb" no longer fails
Release 4.7 rationalized the layout of the build scripts.
Unfortunately, while "make docker-ubuntu-*" and "make docker-debian-*" worked as
expected, "make deb" was broken.
Before this change "make deb" was failing with the following error:
You are not inside a Mercurial repository!
Or, after the latest changes:
You are inside <fullpath>, which is not the root of a Mercurial repository
Moreover, when "make deb" failed, the cleanup routine deleted the wrong
directory (contrib/packaging/debian instead of <reporoot>/debian) resulting in
a corrupted working copy that needed to be hg revert-ed.
After this change the docker targets continue to work, and the deb one is able
to finish.
muxator <a.mux@inwind.it> [Tue, 09 Oct 2018 22:24:38 +0200] rev 40107
packaging: cleanup() did not read the value of $CLEANUP
When the original author put CLEANUP in a conditional statement he was probably
willing to use it to control the "if". This change tries to restore that
behaviour: the "rm" clause is triggered if and only if CLEANUP is defined and
not empty.
muxator <a.mux@inwind.it> [Tue, 09 Oct 2018 22:18:35 +0200] rev 40106
packaging: builddeb's cleanup needs to expand PWD, safely
Single quotes would not expand the variable.
muxator <a.mux@inwind.it> [Tue, 09 Oct 2018 22:16:25 +0200] rev 40105
packaging: blindly factor out trap's cleanup function in builddeb
This commit blindly extracts builddeb's trap routine in a dedicated function.
While doing so, I think two bugs are exposed, which will be addressed in the
next commits:
- single quoting around '$CLEANUP' will always evaluate to the literal
'$CLEANUP' regardless of the variable's value. The "if" will always be true.
- the removal operation will not expand $PWD (and a variable expansion would
need double quotes, anyways.
muxator <a.mux@inwind.it> [Tue, 09 Oct 2018 21:40:49 +0200] rev 40104
packaging: print full path to the packages when builddeb finishes successfully
muxator <a.mux@inwind.it> [Tue, 09 Oct 2018 21:39:39 +0200] rev 40103
packaging: print more specific error messages when builddeb fails
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 09 Oct 2018 12:56:11 -0700] rev 40102
cmdutil: sort unresolved paths
I noticed that `hg status` was printing unresolved paths in a
non-deterministic order. This patch fixes that.
I'm not sure if the sorting should be done in
merge.mergestate.unresolved() instead. Either way fixes the
presentation issue.
Differential Revision: https://phab.mercurial-scm.org/D4929
Yuya Nishihara <yuya@tcha.org> [Tue, 09 Oct 2018 07:46:01 +0900] rev 40101
fuzz: report error if Python code raised exception
I think that's what we wanted to do, given the most of the code block is
surrounded by try-except. 'lazymanifest(mdata)' is moved to the try block
as it can fail.
Yuya Nishihara <yuya@tcha.org> [Tue, 09 Oct 2018 07:42:05 +0900] rev 40100
revlog: explicitly initialize static variables
I know .bss section is zero-filled, but explicit initialization should be
better as we rely on that.
Joerg Sonnenberger <joerg@bec.de> [Mon, 08 Oct 2018 21:53:32 +0200] rev 40099
tests: do not change sys.path, it can break loading cext.parsers
When running this tests with run-tests, the prefix would resolve
mercurial.cext to the source tree and the attempt to load
mercurial.cext.parsers would therefore fail since it doesn't exist in
it. With the regular search path from run-tests, it is picked up from
the temporary prefix correctly.
Differential Revision: https://phab.mercurial-scm.org/D4910
Joerg Sonnenberger <joerg@bec.de> [Mon, 08 Oct 2018 21:51:20 +0200] rev 40098
tests: deal with differences in tic from ncurses and NetBSD
Differential Revision: https://phab.mercurial-scm.org/D4909
Joerg Sonnenberger <joerg@bec.de> [Mon, 08 Oct 2018 20:07:13 +0200] rev 40097
closehead: fix close-head -r listification
Differential Revision: https://phab.mercurial-scm.org/D4908
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Thu, 23 Aug 2018 12:25:54 +0900] rev 40096
import-checker: use testparseutil.embedded() to centralize detection logic
This patch fixes issues of embedded() in import-checker.py below, too.
- overlook (or mis-detect) the end of inline script in doctest style
- overlook inline script in doctest style at the end of file
(and ignore invalid un-closed heredoc at the end of file, too)
- overlook code fragment in styles below
- "python <<EOF" (heredoc should be "cat > file <<EOF" style)
- "cat > foobar.py << ANYLIMIT" (limit mark should be "EOF")
- "cat << EOF > foobar.py" (filename should be placed before limit mark)
- "cat >> foobar.py << EOF" (appending is ignored)
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Thu, 23 Aug 2018 12:25:54 +0900] rev 40095
tests: use NO_CHECK_EOF as heredoc limit mark to omit checking code fragments
This patch uses NO_CHECK_EOF as heredoc limit mark instead of EOF, in
order to avoid checking all python code fragments in
test-contrib-check-code.t, because almost all of them has
un-recommended implementations intentionally.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Thu, 23 Aug 2018 12:25:54 +0900] rev 40094
contrib: add an utility module to parse test scripts
This patch centralizes the logic to pick up code fragments embedded in
*.t script, in order to:
- apply checking with patterns in check-code.py on such embedded
code fragments
Now, check-code.py completely ignores embedded code
fragments. I'll post another patch series to check them.
- replace similar code path in contrib/import-checker.py
Current import-checker.py has problems below. Fixing each of them
is a little difficult, because parsing logic and pattern strings
are tightly coupled.
- overlook (or mis-detect) the end of inline script in doctest
style
8a8dd6e4a97a fixed a part of this issue, but not enough.
- it overlooks inline script in doctest style at the end of file
(and ignores invalid un-closed heredoc at the end of file, too)
- it overlooks code fragment in styles below
- "python <<EOF" (heredoc should be "cat > file <<EOF" style)
- "cat > foobar.py << ANYLIMIT" (limit mark should be "EOF")
- "cat << EOF > foobar.py" (filename should be placed before limit mark)
- "cat >> foobar.py << EOF" (appending is ignored)
- it is not extensible for other than python code fragments
(e.g. shell script, hgrc file, and so on)
This new module can detect python code fragments in styles below:
- inline script in doctest style (starting by " >>> " line)
- python invocation with heredoc script ("python <<EOF")
- python script in heredoc style (redirected into ".py" file)
As an example of extensibility of new module, this patch also contains
implementation to pick up code fragment below. This will be useful to
add additional restriction for them, for example.
- shell script in heredoc style (redirected into ".sh" file)
- hgrc configuration in heredoc style (redirected into hgrc or $HGRCPATH)
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Thu, 23 Aug 2018 12:24:41 +0900] rev 40093
tests: use environment variable indirectly
Using environment variable directly in heredoc python code will cause
syntax error at checking module importation by import-checker.py
strictly, because "$varname" is invalid in Python syntax. "$varname"
becomes valid after environment variable substitution by shell at
writing text into file.
Current import-checker.py overlooks code fragment changed in this
patch, because of a restriction below for a line starting code
fragment.
- filename must be specified before limit mark
NG: cat <<EOF > FILE.py
OK: cat > FILE.py <<EOF
import-checker.py itself is fixed in subsequent patch.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Thu, 23 Aug 2018 12:20:41 +0900] rev 40092
tests: import multiple modules separately
Current import-checker.py overlooks code fragment changed in this
patch, because of restrictions below for a line starting code
fragment.
- filename must be specified before limit mark
NG: cat <<EOF > FILE.py
OK: cat > FILE.py <<EOF
- limit mark must not be quoted
NG: cat > FILE.py <<'EOF'
OK: cat > FILE.py <<EOF
import-checker.py itself is fixed in subsequent patch.
Augie Fackler <augie@google.com> [Mon, 08 Oct 2018 11:50:25 -0400] rev 40091
fuzz: allow manifest fuzzer to detect leaks
Huzzah!
Differential Revision: https://phab.mercurial-scm.org/D4907
Augie Fackler <augie@google.com> [Mon, 08 Oct 2018 11:47:25 -0400] rev 40090
fuzzers: init Python in LLVMFuzzerInitialize and intentionally leak it
This sidesteps leaks (or "leaks", I'm not sure) in CPython, and lets
our fuzzer work.
Differential Revision: https://phab.mercurial-scm.org/D4906
Augie Fackler <augie@google.com> [Mon, 08 Oct 2018 11:42:06 -0400] rev 40089
revlog: if the module is initialized more than once, don't leak nullentry
Caught (annoyingly) by the manifest fuzzer.
Differential Revision: https://phab.mercurial-scm.org/D4905
Martin von Zweigbergk <martinvonz@google.com> [Mon, 01 Oct 2018 14:31:15 -0700] rev 40088
narrow: move remaining narrow-limited dirstate walks to core
In most places we now filter at a higher level (the context object),
but there are few places that relied on the dirstate walk to be
filtered by the narrowspec. The important cases are those used by `hg
add` and `hg addremove`. This patch updates them to pass in a matcher
instead of relying on the dirstate to do the filtering. The dirstate
filtering is also dropped in narrowdirstate.py.
Not always filtering in the dirstate should be useful for a future `hg
status --include-outside-narrow` option.
These places now end up doing an unrestricted dirstate walk after this
patch:
* debugfileset
* perfwalk
* sparse (but restricted to sparse config)
* largefiles
I'll let anyone who cares about these cases adapt them to work with
narrow if necessary.
Differential Revision: https://phab.mercurial-scm.org/D4901
Martin von Zweigbergk <martinvonz@google.com> [Mon, 01 Oct 2018 10:11:00 -0700] rev 40087
narrow: allow repo.narrowmatch(match) to include exact matches from "match"
Differential Revision: https://phab.mercurial-scm.org/D4900
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Sep 2018 22:35:05 -0700] rev 40086
narrow: filter files by narrowspec in ctx.matches()
This has no effect yet because 1) for committed changes, ctx.matches()
just calls ctx.walk(), which we updated in the previous patch, and 2)
for the working copy, the filtering is also done in the overridden
dirstate.walk() in narrowdirstate.
Differential Revision: https://phab.mercurial-scm.org/D4899
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Sep 2018 17:09:15 -0700] rev 40085
narrow: only walk files within narrowspec also for committed revisions
Narrow has been walking only paths matching the narrowspec when
walking the working copy. We have not done the same filtering when
walking committed revisions (e.g. "hg files -r "), which seems a
little odd. Let's make it consistent.
Differential Revision: https://phab.mercurial-scm.org/D4898
Martin von Zweigbergk <martinvonz@google.com> [Thu, 27 Sep 2018 23:01:26 -0700] rev 40084
status: intersect matcher with narrow matcher instead of filtering afterwards
I seem to have done a very naive move of the code from the narrow
extension into core in e411774a2e0f (narrow: move status-filtering to
core and to ctx, 2018-08-02). It seems obvious that a better way is to
intersect the matchers.
Note that this means that when requesting status for the working
directory in a narrow repo, we now pass the narrow matcher (possibly
intersected with a user-provided matcher) into _buildstatus() and then
into dirstate.status() and dirstate.walk(), which will the intersect
it again with the narrow matcher. That's functionally fine, but
wasteful. I hope to later remove the dirstate wrapping that adds the
second layer of matcher intersection.
Differential Revision: https://phab.mercurial-scm.org/D4897
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Sep 2018 12:29:21 -0700] rev 40083
localrepo: allow narrowmatch() to accept matcher to intersect with
It's pretty common that we need to intersect a matcher we already have
(usually from the user) with the narrow matcher. Let's make
repo.narrowmatch() take an optional matcher to intersect with.
Differential Revision: https://phab.mercurial-scm.org/D4896
Zharaskhan Aman <aman.zharaskhan@gmail.com> [Fri, 05 Oct 2018 01:55:51 +0300] rev 40082
obsolete: fix ValueError when stored note contains ':' char (issue5783)
The newer version of `amend -n 'Some some'` accepts containing ':' char.
The information contained in this note 'Testing::Obstore' gives ValueError,
because we are trying to store more than 2 values in key and value.
Differential Revision: https://phab.mercurial-scm.org/D4883
Differential Revision: https://phab.mercurial-scm.org/D4882
Martin von Zweigbergk <martinvonz@google.com> [Fri, 05 Oct 2018 16:06:51 -0700] rev 40081
narrow: update TODO.rst now that we share format with sparse
The narrowspec format was unified with the sparse format in
f64ebe7d2259 (narrowspec: use sparse.parseconfig() to parse narrowspec
file (BC), 2018-08-03).
Differential Revision: https://phab.mercurial-scm.org/D4904
Martin von Zweigbergk <martinvonz@google.com> [Fri, 05 Oct 2018 16:04:25 -0700] rev 40080
narrow: update TODO.rst now that we filter status in ctx
The comment referred to was addressed in e411774a2e0f (narrow: move
status-filtering to core and to ctx, 2018-08-02). I also think
84092edd5c88 (narrow: drop unnecessary overrides of patch, 2018-09-28)
suggests that it was the right thing to do.
Differential Revision: https://phab.mercurial-scm.org/D4903
Martin von Zweigbergk <martinvonz@google.com> [Fri, 05 Oct 2018 16:01:21 -0700] rev 40079
narrow: update TODO.rst now that the narrowspec is in .hg/store
We no longer have the unfortunate wrappostshare() and
unsharenarrowspec() since 576eef1ab43d (narrow: move .hg/narrowspec to
.hg/store/narrowspec (BC), 2018-08-02).
Differential Revision: https://phab.mercurial-scm.org/D4902
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 23:28:14 +0300] rev 40078
py3: add 8 new passing tests to whitelist found by buildbot
We are getting close!
Differential Revision: https://phab.mercurial-scm.org/D4893
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 23:31:51 +0300] rev 40077
py3: use '%f' for floats instead of '%s'
I remember Yuya saying we need to use bytestr() or '%r' because '%s' was clever.
Not sure it applies to this or not.
Differential Revision: https://phab.mercurial-scm.org/D4894
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 22:52:24 +0300] rev 40076
narrow: move adding of narrow server capabilities to core
We use the experimental.narrow config option introduced in one of the previous
patch and move the functionality of adding narrow server capabilities to core.
Differential Revision: https://phab.mercurial-scm.org/D4891
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 22:31:12 +0300] rev 40075
wireprotoserver: move narrow capabilities to wireprototypes.py
This is done because wireprotoserver import wireprotov1server, so you cannot
import wireprotoserver in wireprotov1server to use the capabilities constants.
Differential Revision: https://phab.mercurial-scm.org/D4890
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 22:19:19 +0300] rev 40074
narrow: introduce a config option to check if narrow is enabled or not
This patch introduces a new config option experimental.narrow which is set to
False by default and set to True by the narrow extension.
While moving narrow related logic into core, we need to know at places whether
narrow extension is enabled or not. Checking the list of extension enabled is
one solution but once narrow is inbuilt, we will definitely want a config option
to check whether narrow is turned on or not.
So this patch introduces a config option, which will evolve to the main point to
turn narrow capability on and off once all the narrow is in core.
Differential Revision: https://phab.mercurial-scm.org/D4889
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 20:24:07 +0300] rev 40073
narrow: move the code to generate a widening bundle2 to core
This is a part of moving more narrow related bits to core.
Differential Revision: https://phab.mercurial-scm.org/D4888
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 02 Oct 2018 17:09:56 +0300] rev 40072
narrow: start returning bundle2 from widen_bundle()
Differential Revision: https://phab.mercurial-scm.org/D4838
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 28 Sep 2018 23:42:31 +0300] rev 40071
narrow: the first version of narrow_widen wireprotocol command
This patch introduces a wireprotocol command narrow_widen() which will be used
to widen a narrow copy using `hg tracked` command provided by narrow extension.
The wireprotocol command takes the old and new includes and excludes, common
heads, changegroup version, known revs, and a boolean ellipses and generates a
bundle2 of the required data and send it. The clients receives the bundle2
and applies that.
A bundle2 instead of changegroup because in future we might want to add more
things to send while widening. Thanks for martinvonz for the suggestion.
I am not sure whether we need changegroup version as an argument to the command
as I *think* narrow needs changegroup3 already.
The tests shows that we don't exchange phase data now while widening which is
nice. Also we don't check for pushkeys, rbc-cache, bookmarks etc.
This does not support ellipses cases for now but will be supported in future
patches. Since we send bundle2, it won't be hard to plug the ellipses logic in
here.
The existing code for widening a non-ellipses case is also dropped in this
patch.
Differential Revision: https://phab.mercurial-scm.org/D4813
Yuya Nishihara <yuya@tcha.org> [Fri, 05 Oct 2018 21:43:57 +0900] rev 40070
remotenames: abort if literal revset pattern matches nothing
This is the convention of the other namespace revsets such as tag(). Let's
make the remote variants do the same.
Yuya Nishihara <yuya@tcha.org> [Fri, 05 Oct 2018 21:39:41 +0900] rev 40069
remotenames: remove unneeded sorted() from revset implementation
The order is constrained by the subset.
Yuya Nishihara <yuya@tcha.org> [Fri, 05 Oct 2018 21:36:48 +0900] rev 40068
remotenames: don't call a set of nodes as "revs"
Yuya Nishihara <yuya@tcha.org> [Fri, 05 Oct 2018 21:30:55 +0900] rev 40067
remotenames: use util.always instead of handcrafted lambda
Yuya Nishihara <yuya@tcha.org> [Fri, 05 Oct 2018 21:29:21 +0900] rev 40066
remotenames: inline _parseargs() into _revsetutil()
The _parseargs() function gets quite simple, and the 0/1 loop can be rewritten
as "if".
Martin von Zweigbergk <martinvonz@google.com> [Thu, 04 Oct 2018 16:27:40 -0700] rev 40065
repo: create changectx in a single place in localrepo.__getitem__
Differential Revision: https://phab.mercurial-scm.org/D4885
Martin von Zweigbergk <martinvonz@google.com> [Thu, 04 Oct 2018 16:06:36 -0700] rev 40064
repo: remove the last few "pass" statements in localrepo.__getitem__
In case of IndexError or LookupError, we used "pass" statements and
fell through to the end of localrepo.__getitem__. I find the pass
statements easy to miss. Consistently raising and catching exceptions
seems easier to follow.
Oh -- and I didn't plan this before I wrote the above -- that probably
also lets us reuse the "return context.changectx(self, rev, node)" in
a later patch.
Differential Revision: https://phab.mercurial-scm.org/D4884
Martin von Zweigbergk <martinvonz@google.com> [Thu, 04 Oct 2018 10:38:55 -0700] rev 40063
filectx: correct docstring about "changeid"
The changeid argument must be a revnum (basefile.rev() is defined as
"return self._changeid"), so fix the lie in the docstring. It seems to
have been incorrect for at least 10 years (I didn't check further
back).
Differential Revision: https://phab.mercurial-scm.org/D4881
Martin von Zweigbergk <martinvonz@google.com> [Thu, 04 Oct 2018 10:30:05 -0700] rev 40062
context: drop incorrect and superfluous docstring
It's been incorrect at least since 8b86acc7aa64 (context: drop support
for looking up context by ambiguous changeid (API), 2018-04-28).
Differential Revision: https://phab.mercurial-scm.org/D4880
Augie Fackler <raf@durin42.com> [Thu, 04 Oct 2018 21:35:12 -0400] rev 40061
remotenames: follow-up on D3639 to make revset funcs take only one arg
Per the review discussion on D3639, we want this to just take one
argument. That ended up simplifying the code, so I'm sharing this as a
follow-up to that revision rather than editing in-flight.
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Jul 2018 03:12:09 +0530] rev 40060
remotenames: add names argument to remotenames revset
This patch adds names argument to the revsets provided by the remotenames
extension. The revsets are remotenames(), remotebranches() and
remotebookmarks(). names can be a single names, list of names or can be empty too
which means it's an optional argument.
If names is/are passed, changesets which have those remotenames will be
returned.
If names are not passed, changesets from all the remotenames are shown.
Passing an invalid remotename does not throw error.
The name argument also supports pattern matching.
Tests are added for the argument in tests/test-logexchange.t
Differential Revision: https://phab.mercurial-scm.org/D3639
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:43:48 -0400] rev 40059
copies: add time information to the debug information
Boris Feld <boris.feld@octobus.net> [Fri, 07 Sep 2018 11:16:06 -0400] rev 40058
copies: add a devel debug mode to trace what copy tracing does
Mercurial can spend a lot of time finding renames between two commits. Having
more information about that process help to understand what makes it slow in
an individual instance. (eg: many files vs 1 file, etc...)
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 17:34:34 -0700] rev 40057
revlog: rewrite censoring logic
I was able to corrupt a revlog relatively easily with the existing
censoring code. The underlying problem is that the existing code
doesn't fully take delta chains into account. When copying revisions
that occur after the censored revision, the delta base can refer
to a censored revision. Then at read time, things blow up due to the
revision data not being a compressed delta.
This commit rewrites the revlog censoring code to take a higher-level
approach. We now create a new revlog instance pointing at temp files.
We iterate through each revision in the source revlog and insert
those revisions into the new revlog, replacing the censored revision's
data along the way.
The new implementation isn't as efficient as the old one. This is
because it will fully engage delta computation on insertion. But I
don't think it matters.
The new implementation is a bit hacky because it attempts to reload
the revlog instance with a new revlog index/data file. This is fragile.
But this is needed because the index (which could be backed by C) would
have a cached copy of the old, possibly changed data and that could
lead to problems accessing index or revision data later.
One benefit of the new approach is that we integrate with the
transaction. The old revlog is backed up and if the transaction is
rolled back, the original revlog is restored.
As part of this, we had to teach the transaction about the store
vfs. I'm not super keen about this. But this was the easiest way
to hook things up to the transaction. We /could/ just ignore the
transaction like we were doing before. But any file mutation should
be governed by transaction semantics, including undo during rollback.
Differential Revision: https://phab.mercurial-scm.org/D4869
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 17:28:54 -0700] rev 40056
revlog: move loading of index data into own method
This will allow us to "reload" a revlog instance from a rewritten
index file, which will be used in a subsequent commit.
Differential Revision: https://phab.mercurial-scm.org/D4868
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 10:57:35 -0700] rev 40055
revlog: clear revision cache on hash verification failure
The revision cache is populated after raw revision fulltext is
retrieved but before hash verification. If hash verification
fails, the revision cache will be populated and subsequent
operations to retrieve the invalid fulltext may return the cached
fulltext instead of raising.
This commit changes hash verification so it will invalidate the
revision cache if the cached node fails hash verification. The
side-effect is that subsequent operations to request the revision
text - even the raw revision text - will always fail.
The new behavior is consistent and is definitely less wrong. There
is an open question of whether revision(raw=True) should validate
hashes. But I'm going to punt on this problem. We can always change
behavior later. And to be honest, I'm not sure we should expose
raw=True on the storage interface at all. Another day...
Differential Revision: https://phab.mercurial-scm.org/D4867
Augie Fackler <augie@google.com> [Thu, 06 Sep 2018 02:36:25 -0400] rev 40054
fuzz: new fuzzer for cext/manifest.c
This is a bit messy, because lazymanifest is tightly coupled to the
cpython API for performance reasons. As a result, we have to build a
whole Python without pymalloc (so ASAN can help us out) and link
against that. Then we have to use an embedded Python interpreter. We
could manually drive the lazymanifest in C from that point, but
experimentally just using PyEval_EvalCode isn't really any slower so
we may as well do that and write the innermost guts of the fuzzer in
Python.
Leak detection is currently disabled for this fuzzer because there are
a few global-lifetime things in our extensions that we more or less
intentionally leak and I didn't want to take the detour to work around
that for now.
This should not be pushed to our repo until
https://github.com/google/oss-fuzz/pull/1853 is merged, as this
depends on having the Python tarball around.
Differential Revision: https://phab.mercurial-scm.org/D4879
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 10:32:21 -0700] rev 40053
revlog: rename _cache to _revisioncache
"cache" is generic and revlog instances have multiple caches. Let's
be descriptive about what this is a cache for.
Differential Revision: https://phab.mercurial-scm.org/D4866
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 10:56:48 -0700] rev 40052
testing: add file storage integration for bad hashes and censoring
In order to implement these tests, we need a backdoor to write data
into storage backends while bypassing normal checks. We invent a
callable to do that.
As part of writing the tests, I found a bug with censorrevision()
pretty quickly! After calling censorrevision(), attempting to
access revision data for an affected node raises a cryptic error
related to malformed compression. This appears to be due to the
revlog not adjusting delta chains as part of censoring.
I also found a bug with regards to hash verification and revision
fulltext caching. Essentially, we cache the fulltext before hash
verification. If we look up the fulltext after a failed hash
verification, we don't get a hash verification exception. Furthermore,
the behavior of revision(raw=True) can be inconsistent depending on
the order of operations.
I'll be fixing both these bugs in subsequent commits.
Differential Revision: https://phab.mercurial-scm.org/D4865
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 10:03:41 -0700] rev 40051
testing: add file storage tests for getstrippoint() and strip()
Differential Revision: https://phab.mercurial-scm.org/D4864
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 10:04:04 -0700] rev 40050
wireprotov2: always advertise raw repo requirements
I'm pretty sure my original thinking behind making it conditional
on stream clone support was that the behavior mirrored wire protocol
version 1.
I don't see a compelling reason for us to not advertise the server's
storage requirements. The proper way to advertise stream clone support
in wireprotov2 would be to not advertise the command(s) required to
perform stream clone or to advertise a separate capability denoting
stream clone support.
Stream clone isn't yet implemented on wireprotov2, so we can cross
this bridge later.
Differential Revision: https://phab.mercurial-scm.org/D4863
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 09:48:22 -0700] rev 40049
tests: don't be as verbose in wireprotov2 tests
I don't think that printing low-level I/O and frames is beneficial to
testing command-level functionality. Protocol-level testing, yes. But
command-level functionality shouldn't care about low-level details in
most cases. This output makes tests more verbose and harder to read.
It also makes them harder to maintain, as you need to glob over various
dynamic width fields.
Let's remove these low-level details from many of the wireprotov2
tests.
Differential Revision: https://phab.mercurial-scm.org/D4861
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 12:57:01 -0700] rev 40048
repository: define and use revision flag constants
Revlogs have a per-revision 2 byte field holding integer flags that
define how revision data should be interpreted. For historical reasons,
these integer values are sent verbatim on the wire protocol as part of
changegroup data.
From a semantic standpoint, the flags that go out over the wire are
different from the flags stored internally by revlogs. Failure to
establish this semantic distinction creates unwanted strong coupling
between revlog's internals and the wire protocol.
This commit establishes new constants on the repository module that
define the revision flags used by the wire protocol (and by some
internal storage APIs, sadly). The changegroups internals documentation
has been updated to document them explicitly. Various references
throughout the repo now use the repository constants instead of the
revlog constants. This is done to make it clear that we're operating
on generic revision data and this isn't tied to revlogs.
Differential Revision: https://phab.mercurial-scm.org/D4860
Boris Feld <boris.feld@octobus.net> [Thu, 04 Oct 2018 01:22:25 +0200] rev 40047
context: reverse conditional branch order in introrev
Positive logic will be simpler to follow. It will help to clarify coming
refactoring.
Boris Feld <boris.feld@octobus.net> [Thu, 04 Oct 2018 08:40:01 +0200] rev 40046
context: drop a redundant fast path in introrev
Now that _adjustlinkrev fast path this case itself, we no longer need an extra
conditional.
A nice side effect is that we are no longer calling `self.rev()`. In case
where `_descendantrev` is set, calling `self.rev` will trigger a potentially
expensive `_adjustlinkrev` call. So blindly calling `self.rev()` to avoid
another `_adjustlinkrev` call can be counterproductive.
Note that `_descendantrev` is currently never taken into account in `introrev`
so far which is wrong. We'll fix that in changeset later in this series.
Boris Feld <boris.feld@octobus.net> [Thu, 04 Oct 2018 08:34:59 +0200] rev 40045
context: fast path linkrev adjustement in trivial case
If the search starts from the linkrev, there is nothing to adjust.
Cédric Krier <ced@b2ck.com> [Thu, 04 Oct 2018 11:28:48 +0200] rev 40044
url: allow to configure timeout on http connection
By default, httplib.HTTPConnection opens connection with no timeout.
If the server is hanging, Mercurial will wait indefinitely. This may be an
issue for automated scripts.
Differential Revision: https://phab.mercurial-scm.org/D4878
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 23:50:14 +0200] rev 40043
obsolete: explicitly track folds inside the markers
We now record information to be able to recognize "fold" event from
obsolescence markers. To do so, we track the following pieces of information:
a) a fold ID. Unique to that fold (per successor),
b) the number of predecessors,
c) the index of the predecessor in that fold.
We will now be able to create an algorithm able to find "predecessorssets".
We now store this data in the generic "metadata" field of the markers.
Updating the format to have a more compact storage for this would be useful.
This way of tracking a fold through multiple markers could be applied to split
too. This would have two advantages:
1) We get a simpler format, since number of successors is limited to [0-1].
2) We can better deal with situations where only some of the split successors
are pushed to a remote repository.
We should look into the relevance of such a change before updating the on-disk
format.
note: unlike splits, folds do not have to deal with cases where only some of
the markers have been synchronized. As they all share the same successor
changesets, they are all relevant to the same nodes.
Boris Feld <boris.feld@octobus.net> [Wed, 03 Oct 2018 11:59:47 +0200] rev 40042
cleanupnodes: update comment to drop mention of filtering
Since changeset 1857f50a9643 drop the filtering, we should not longer mention it
in code comment.
spectral <spectral@google.com> [Wed, 26 Sep 2018 18:04:46 -0700] rev 40041
treemanifests: remove _loadalllazy when doing copies
'before' here is https://phab.mercurial-scm.org/D4845 (not the committed/rebased
version)
diff --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.329 s +- 0.011 s | 1.320 s +- 0.010 s | 99.3%
m-u | | x | 1.316 s +- 0.005 s | 1.334 s +- 0.018 s | 101.4%
m-u | x | | 1.330 s +- 0.021 s | 1.322 s +- 0.005 s | 99.4%
m-u | x | x | 87.2 ms +- 0.7 ms | 86.9 ms +- 1.5 ms | 99.7%
l-d-r | | | 203.3 ms +- 7.8 ms | 199.4 ms +- 1.8 ms | 98.1%
l-d-r | | x | 204.6 ms +- 2.8 ms | 201.7 ms +- 2.1 ms | 98.6%
l-d-r | x | | 90.5 ms +- 11.0 ms | 86.2 ms +- 1.0 ms | 95.2%
l-d-r | x | x | 66.3 ms +- 2.0 ms | 66.4 ms +- 0.9 ms | 100.2%
diff -c . --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 239.4 ms +- 2.0 ms | 241.7 ms +- 4.6 ms | 101.0%
m-u | | x | 128.9 ms +- 1.9 ms | 130.9 ms +- 7.7 ms | 101.6%
m-u | x | | 241.1 ms +- 1.6 ms | 240.1 ms +- 1.4 ms | 99.6%
m-u | x | x | 133.4 ms +- 1.5 ms | 133.4 ms +- 1.2 ms | 100.0%
l-d-r | | | 84.3 ms +- 1.5 ms | 83.5 ms +- 1.0 ms | 99.1%
l-d-r | | x | 200.9 ms +- 6.3 ms | 203.0 ms +- 4.4 ms | 101.0%
l-d-r | x | | 108.1 ms +- 1.4 ms | 108.7 ms +- 2.1 ms | 100.6%
l-d-r | x | x | 190.2 ms +- 4.8 ms | 191.6 ms +- 2.0 ms | 100.7%
rebase -r . --keep -d .^^:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 5.655 s +- 0.029 s | 5.640 s +- 0.036 s | 99.7%
m-u | | x | 5.813 s +- 0.038 s | 5.773 s +- 0.028 s | 99.3%
m-u | x | | 5.593 s +- 0.043 s | 5.589 s +- 0.028 s | 99.9%
m-u | x | x | 648.2 ms +- 19.2 ms | 637.3 ms +- 27.7 ms | 98.3%
l-d-r | | | 673.3 ms +- 8.0 ms | 673.2 ms +- 6.8 ms | 100.0%
l-d-r | | x | 6.583 s +- 0.030 s | 5.721 s +- 0.028 s | 86.9% <--
l-d-r | x | | 277.8 ms +- 6.7 ms | 276.0 ms +- 2.7 ms | 99.4%
l-d-r | x | x | 1.692 s +- 0.013 s | 720.9 ms +- 13.3 ms | 42.6% <--
status --change . --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 220.9 ms +- 1.6 ms | 219.9 ms +- 2.2 ms | 99.5%
m-u | | x | 109.2 ms +- 1.0 ms | 109.4 ms +- 0.8 ms | 100.2%
m-u | x | | 222.6 ms +- 1.7 ms | 221.4 ms +- 2.1 ms | 99.5%
m-u | x | x | 113.4 ms +- 0.5 ms | 113.1 ms +- 1.1 ms | 99.7%
l-d-r | | | 82.1 ms +- 1.7 ms | 82.1 ms +- 1.2 ms | 100.0%
l-d-r | | x | 199.8 ms +- 4.0 ms | 200.7 ms +- 3.6 ms | 100.5%
l-d-r | x | | 85.4 ms +- 1.5 ms | 85.2 ms +- 0.3 ms | 99.8%
l-d-r | x | x | 202.6 ms +- 4.4 ms | 208.0 ms +- 4.0 ms | 102.7%
status --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.941 s +- 0.014 s | 1.930 s +- 0.009 s | 99.4%
m-u | | x | 1.924 s +- 0.007 s | 1.950 s +- 0.010 s | 101.4%
m-u | x | | 1.959 s +- 0.085 s | 1.926 s +- 0.009 s | 98.3%
m-u | x | x | 96.2 ms +- 1.0 ms | 96.4 ms +- 0.7 ms | 100.2%
l-d-r | | | 604.4 ms +- 10.6 ms | 602.6 ms +- 7.1 ms | 99.7%
l-d-r | | x | 605.7 ms +- 4.1 ms | 607.4 ms +- 6.1 ms | 100.3%
l-d-r | x | | 182.4 ms +- 1.2 ms | 183.4 ms +- 1.2 ms | 100.5%
l-d-r | x | x | 150.8 ms +- 2.0 ms | 150.6 ms +- 1.0 ms | 99.9%
update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 3.185 s +- 0.027 s | 3.181 s +- 0.017 s | 99.9%
m-u | | x | 3.028 s +- 0.021 s | 2.954 s +- 0.010 s | 97.6%
m-u | x | | 3.168 s +- 0.010 s | 3.175 s +- 0.023 s | 100.2%
m-u | x | x | 317.5 ms +- 3.5 ms | 313.2 ms +- 2.9 ms | 98.6%
l-d-r | | | 456.2 ms +- 10.6 ms | 454.4 ms +- 5.8 ms | 99.6%
l-d-r | | x | 9.236 s +- 0.063 s | 757.9 ms +- 9.2 ms | 8.2% <--
l-d-r | x | | 257.6 ms +- 2.3 ms | 261.2 ms +- 1.7 ms | 101.4%
l-d-r | x | x | 1.614 s +- 0.013 s | 478.0 ms +- 14.3 ms | 29.6% <--
Differential Revision: https://phab.mercurial-scm.org/D4875
spectral <spectral@google.com> [Tue, 25 Sep 2018 19:25:41 -0700] rev 40040
treemanifests: store whether a lazydirs entry needs copied after materializing
Due to the way that things like manifestlog.get caches its values, without
making a copy (if necessary) after calling readsubtree(), we might end up
adjusting the state of the same object on different contexts, breaking things
like dirty state tracking (and probably other things).
Differential Revision: https://phab.mercurial-scm.org/D4874
spectral <spectral@google.com> [Tue, 02 Oct 2018 18:55:07 -0700] rev 40039
treemanifests: extract _loaddifflazy from _diff, use in _filesnotin
Differential Revision: https://phab.mercurial-scm.org/D4873
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 03 Oct 2018 18:07:49 -0400] rev 40038
identify: show remote bookmarks in `hg id url -Tjson -B`
I didn't display bookmarks when `default and not ui.quiet`: it seems
strange for templates to depend on --id or -q, and it would take more
code for `hg id url -T {node}` to not request remote bookmarks.
An alternative I thought of was providing lazy data to the formatter,
`fm.data(bookmarks=lambda: fm.formatlist(getbms(), name='bookmark'))`.
The plainformatter would naturally not compute it, the
templateformatter would compute only what it needs, and the other ones
would compute everything, but that's not supported (or I don't see
how), so I abandoned this idea.
Differential Revision: https://phab.mercurial-scm.org/D4872
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 16:03:16 -0400] rev 40037
showstack: also handle SIGALRM
This is looking *very* handy when debugging mysterious hangs in a
test: you can wrap a hanging invocation in
`perl -e 'alarm shift @ARGV; exec @ARGV' 1`
for example, a hanging `hg pull` becomes
`perl -e 'alarm shift @ARGV; exec @ARGV' 1 hg pull`
where the `1` is the timeout in seconds before the process will be hit
with SIGALRM. After making that edit to the test file, you can then
use --extra-config-opt on run-tests.py to globaly enable showstack
during the test run, so you'll get full stack traces as you force your
hg to exit.
I wonder (but only a little, not enough to take action just yet) if we
should wire up some scaffolding in run-tests itself to automatically
wrap all commands in alarm(3) somehow to avoid hangs in the future?
Differential Revision: https://phab.mercurial-scm.org/D4870
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 13:54:31 -0700] rev 40036
exchangev2: add progress bar around manifest scanning
This can take a long time on large repositories. Let's add a progress
bar so we don't have long periods where it isn't obvious what is
going on.
Differential Revision: https://phab.mercurial-scm.org/D4859
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 13:17:38 -0700] rev 40035
httppeer: report http statistics
Now that keepalive.py records HTTP request count and the
number of bytes sent and received as part of performing those
requests, we can easily print a report on the activity when
closing a peer instance!
Exact byte counts are globbed in tests because they are influenced
by non-deterministic things, such as hostnames and port numbers.
Plus, the exact byte count isn't too important anyway.
I feel obliged to note that printing the byte count could have
security implications. e.g. if sending a password via HTTP basic
auth, the length of that password will influence the byte count
and the reporting of the byte count could be a side-channel leak
of the password length. I /think/ this is beyond our threshold
for concern. But if we think it poses a problem, we can teach the
byte count logging code to e.g. ignore sensitive HTTP request
headers. We could also consider not reporting the byte count of
request headers altogether. But since the wire protocol uses HTTP
headers for sending command arguments, it is kind of important to
report their size.
Differential Revision: https://phab.mercurial-scm.org/D4858
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:30:32 -0700] rev 40034
keepalive: track number of bytes received from an HTTP response
We also bubble the byte count up to the HTTPConnection instance and its
parent opener at read time. Unlike sending, there isn't a clear
"end of response" signal we can intercept to defer updating the
accounting.
Differential Revision: https://phab.mercurial-scm.org/D4857
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:02:54 -0700] rev 40033
keepalive: track request count and bytes sent
I want wire protocol interactions to report the number of
requests made and bytes transferred.
This commit teaches the very low-level custom HTTPConnection class
to track the number of bytes sent to the socket. This may vary from
the number of bytes that go on the wire due to e.g. TLS. That's OK.
KeepAliveHandler is taught to track the total number of requests
and total number of bytes sent across all requests.
Differential Revision: https://phab.mercurial-scm.org/D4856
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 12:06:36 -0700] rev 40032
url: have httpsconnection inherit from our custom HTTPConnection
This will ensure that any customizations we perform to HTTPConnection
will be available to httpsconnection.
Differential Revision: https://phab.mercurial-scm.org/D4855
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 09:43:01 -0700] rev 40031
cborutil: change buffering strategy
Profiling revealed that we were spending a lot of time on the
line that was concatenating the old buffer with the incoming data
when attempting to decode long byte strings, such as manifest
revisions.
Essentially, we were feeding N chunks of size len(X) << len(Y) into
decode() and continuously allocating a new, larger buffer to hold
the undecoded input. This created substantial memory churn and
slowed down execution.
Changing the code to aggregate pending chunks in a list until we
have enough data to fully decode the next atom makes things much
more efficient.
I don't have exact data, but I recall the old code spending >1s
on manifest fulltexts from the mozilla-unified repo. The new code
doesn't significantly appear in profile output.
Differential Revision: https://phab.mercurial-scm.org/D4854
Martin von Zweigbergk <martinvonz@google.com> [Wed, 03 Oct 2018 10:27:44 -0700] rev 40030
cleanup: some Yoda conditions, this patch removes
It seems the factor 20 is less than the frequency of " < \d" compared
to " \d > ".
Differential Revision: https://phab.mercurial-scm.org/D4862
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:43:54 -0700] rev 40029
streamclone: don't support stream clone unless repo feature present
This change means custom repository types must opt in to enabling
stream clone. This seems reasonable, as stream clones are a very
low-level feature that has historically assumed the use of revlogs
and the layout of .hg/ that they entail.
Differential Revision: https://phab.mercurial-scm.org/D4853
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 12:40:39 -0700] rev 40028
localrepo: add repository feature when repo can be stream cloned
Right now, the wire protocol server assumes all repository objects can
be stream cloned (unless the stream clone feature is disabled via
config option).
But not all storage backends or repository objects may support stream
clone.
This commit defines a repository feature denoting whether stream clone
is supported. The feature is defined for revlog-based repositories,
which should currently be "all repositories."
Differential Revision: https://phab.mercurial-scm.org/D4852
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 18:08:08 -0700] rev 40027
wireprotov2: client support for following content redirects
And with the server actually sending content redirects, it is finally
time to implement client support for following them!
When a redirect response is seen, we wait until all data for that
request has been received (it should be nearly immediate since no
data is expected to follow the redirect message). Then we use
a URL opener to make a request. We stuff that response into the
client handler and construct a new response object to track it.
When readdata() is called for servicing requests, we attempt to
read data from the first redirected response. During data reading,
data is processed similarly to as if it came from a frame payload.
The existing test for the functionality demonstrates the client
transparently following the redirect and obtaining the command
response data from an alternate URL!
There is still plenty of work to do here, including shoring up
testing. I'm not convinced things will work in the presence of
multiple redirect responses. And we don't yet implement support
for integrity verification or configuring server certificates
to validate the connection. But it's a start. And it should enable
us to start experimenting with "real" caches.
Differential Revision: https://phab.mercurial-scm.org/D4778
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 18:07:55 -0700] rev 40026
wireprotov2: server support for sending content redirects
A "content redirect" can be sent in place of inline response content.
In terms of code, we model a content redirect as a special type of
response object holding the attributes describing that redirect.
Sending a content redirect thus becomes as simple as the object
emission layer sending an instance of that type. A cacher using
externally-addressable content storage could replace the outgoing
object stream with an object advertising its location.
The bulk of the code in this commit is teaching the output layer
which handles the object stream to recognize alternate location
objects. The rules are that if an alternate location object is
present, it must be the first and only object in the object stream.
Otherwise the server emits an error.
Differential Revision: https://phab.mercurial-scm.org/D4777
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 15:02:19 -0700] rev 40025
wireprotov2: client support for advertising redirect targets
With the server now able to emit a redirect target descriptor, we can
start to teach the client to recognize it.
This commit implements support for filtering the advertised
redirect targets against supported features and for advertising
compatible redirect targets as part of command requests. It also
adds the minimal boilerplate required to fail when a content
redirect is seen.
The server doesn't yet do anything with the advertised redirect
targets. And the client can't yet follow redirects if it did. But
at least we're putting bytes on the wire.
Differential Revision: https://phab.mercurial-scm.org/D4776
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 17:46:48 -0700] rev 40024
wireprotov2: advertise redirect targets in capabilities
This is pretty straightforward.
Redirect targets will require an extension to support. So we've added
a function that can be wrapped to define redirect targets.
To test this, we teach our simple cache test extension to read
redirect targets from a file. It's a bit hacky. But it gets the
job done.
Differential Revision: https://phab.mercurial-scm.org/D4775
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 18:02:06 -0700] rev 40023
wireprotov2: define semantics for content redirects
When I implemented the clonebundles feature and deployed it on
hg.mozilla.org using Amazon S3 as a content server, server-side CPU
and bandwidth usage dropped off a cliff and a ton of server scaling
headaches went away pretty much the instant clients with support for
clonebundles were rolled out to Firefox CI.
An obvious takeaway from that experience was that offloading server
load to scalable file servers - potentially backed by a CDN - is a
really good idea. Another takeaway was that Mercurial's wire protocol
wasn't in a good position to support data offload generally.
In wire protocol version 1, there isn't a mechanism in the protocol to
say "grab the data from over here instead." For HTTP, we could teach
the client to follow HTTP redirects. Or we could invent a media type
that encoded redirects inline. But for SSH, we were pretty much out of
luck because that protocol wasn't very flexible.
Wire protocol version 2 offers the opportunity to do something better.
The recent generic server-side content caching layer in the wire
protocol version 2 server demonstrated that it is possible to have
drop-in caching of responses to command requests. This by itself
adds tons of value and already makes the built-in server much more
scalable. But I don't want to stop there.
The existing server-side caching implementation has a big weakness:
it requires the server to send data to the client. This means that
the Mercurial server is potentially sending gigabytes of data to
thousands of clients. This is problematic because compared to scaling
static file servers, scaling dynamic servers is *hard*.
A solution to this is to "offload" serving of content to something
that isn't the Mercurial server. By offloading content serving, you
turn the Mercurial server from a centralized monolithic service to
a distributed mostly-indexing service. Assuming high rates of content
offload, this should drastically reduce the total work performed by
the Mercurial server, both in terms of CPU and data transfer. This
will make Mercurial servers vastly easier to scale.
This commit defines the semantics for "content redirects" in wire
protocol version 2. Essentially:
* Servers advertise the set of locations a response could be served
from.
* When making requests, clients advertise the set of locations they
are willing to fetch content from.
* Servers can then replace the inline response with one that says
"get the response from over here instead."
This feature - when fully implemented - will allow extending the
server-side caching layer to facilitate such things as integrating
your server-side cache with a scalable blob store (such as S3 or
a CDN) and offloading most data transfer to that external service.
This feature could also be leveraged for load balancing. e.g.
requests could come into a central server and then get redirected
to an available mirror depending on server availability or locality.
There's tons of potential :)
Differential Revision: https://phab.mercurial-scm.org/D4774
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 17:16:56 -0700] rev 40022
wireprotov2: support response caching
One of the things I've learned from managing VCS servers over the
years is that they are hard to scale. It is well known that some
companies have very beefy (read: very expensive) servers to power
their VCS needs. It is also known that specialized servers for
various VCS exist in order to facilitate scaling servers. (Mercurial
is in this boat.)
One of the aspects that make a VCS server hard to scale is the
high CPU load incurred by constant client clone/pull operations.
To alleviate the scaling pain associated with data retrieval
operations, I want to integrate caching into the Mercurial wire
protocol server as robustly as possible such that servers can
aggressively cache responses and defer as much server load as
possible.
This commit represents the initial implementation of a general
caching layer in wire protocol version 2.
We define a new interface and behavior for a wire protocol cacher
in repository.py. (This is probably where a reviewer should look
first to understand what is going on.)
The bulk of the added code is in wireprotov2server.py, where we
define how a command can opt in to being cached and integrate
caching into command dispatching.
From a very high-level:
* A command can declare itself as cacheable by providing a callable
that can be used to derive a cache key.
* At dispatch time, if a command is cacheable, we attempt to
construct a cacher and use it for serving the request and/or
caching the request.
* The dispatch layer handles the bulk of the business logic for
caching, making cachers mostly "dumb content stores."
* The mechanism for invalidating cached entries (one of the harder
parts about caching in general) is by varying the cache key when
state changes. As such, cachers don't need to be concerned with
cache invalidation.
Initially, we've hooked up support for caching "manifestdata" and
"filedata" commands. These are the simplest to cache, as they should
be immutable over time. Caching of commands related to changeset
data is a bit harder (because cache validation is impacted by
changes to bookmarks, phases, etc). This will be implemented later.
(Strictly speaking, censoring a file should invalidate caches. I've
added an inline TODO to track this edge case.)
To prove it works, this commit implements a test-only extension
providing in-memory caching backed by an lrucachedict. A new test
showing this extension behaving properly is added. FWIW, the
cacher is ~50 lines of code, demonstrating the relative ease with
which a cache can be added to a server.
While the test cacher is not suitable for production workloads, just
for kicks I performed a clone of just the changeset and manifest data
for the mozilla-unified repository. With a fully warmed cache (of just
the manifest data since changeset data is not cached), server-side
CPU usage dropped from ~73s to ~28s. That's pretty significant and
demonstrates the potential that response caching has on server
scalability!
Differential Revision: https://phab.mercurial-scm.org/D4773
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 17:16:27 -0700] rev 40021
wireprotov2: define type to represent pre-encoded object
An upcoming commit will introduce a caching layer to command
serving. This will require the ability to cache pre-encoded data.
This commit introduces a type to represent pre-encoded data and
teaches the output layer to not CBOR encode an instance of that
type.
Differential Revision: https://phab.mercurial-scm.org/D4772
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 15:53:49 -0700] rev 40020
wireprotov2: change name and behavior of readframe()
In the near future, we will want to support performing I/O from
other sources. Let's rename readframe() to readdata() and tweak
its logic to support future growth.
Differential Revision: https://phab.mercurial-scm.org/D4771
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 16:07:59 -0700] rev 40019
url: move _wraphttpresponse() from httpeer
This is a generally useful function. Having it on the url module
will make it more accessible outside of the HTTP peers.
Differential Revision: https://phab.mercurial-scm.org/D4770
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 14:54:15 -0700] rev 40018
debugcommands: print all CBOR objects
application/mercurial-cbor may contain multiple objects. Let's print
all of them.
Differential Revision: https://phab.mercurial-scm.org/D4769
Yuya Nishihara <yuya@tcha.org> [Wed, 03 Oct 2018 22:48:19 +0900] rev 40017
help: document about "export" template keywords
Yuya Nishihara <yuya@tcha.org> [Wed, 03 Oct 2018 22:43:57 +0900] rev 40016
help: document about "config" template keywords
Yuya Nishihara <yuya@tcha.org> [Wed, 03 Oct 2018 22:34:18 +0900] rev 40015
help: document about "cat" template keywords
Yuya Nishihara <yuya@tcha.org> [Wed, 03 Oct 2018 22:38:49 +0900] rev 40014
help: document about "branches" template keywords
Yuya Nishihara <yuya@tcha.org> [Wed, 03 Oct 2018 22:32:18 +0900] rev 40013
help: document about "bookmarks" template keywords
Yuya Nishihara <yuya@tcha.org> [Wed, 03 Oct 2018 22:27:45 +0900] rev 40012
help: document about "annotate" template keywords
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 16:34:53 -0700] rev 40011
storageutil: pass nodes into emitrevisions()
The main emitrevisions() uses nodes. So it makes sense to use
nodes for the helper API.
Not bothering with API since this function was introduced a few
commits ago.
Differential Revision: https://phab.mercurial-scm.org/D4805
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 16:16:09 -0700] rev 40010
storageutil: make all callables optional
Not all storage backends may implement these callables. That's part
of the reason these methods aren't exposed on the storage interface.
Differential Revision: https://phab.mercurial-scm.org/D4804
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 16:16:22 -0700] rev 40009
storageutil: extract most of emitrevisions() to standalone function
As part of implementing a storage backend, I found myself copying
most of revlog.emitrevisions(). This code is highly nuanced and it
bothered me greatly to be copying such low-level code.
This commit extracts the bulk of revlog.emitrevisions() into a
new standalone function. In order to make the function generally
usable, all "self" function calls that aren't exposed on the
ifilestorage interface are passed in via callable arguments.
No meaningful behavior should have changed as part of the port.
Upcoming commits will tweak behavior to make the code more
generically usable.
Differential Revision: https://phab.mercurial-scm.org/D4803
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 11:51:17 -0700] rev 40008
storageutil: invert logic of file data comparison
IMO things make more sense when the function is explicitly a test
for file data equivalence.
Not bothering with API since the function was introduced by the
previous commit.
Differential Revision: https://phab.mercurial-scm.org/D4802
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 11:47:53 -0700] rev 40007
storageutil: extract filelog.cmp() to a standalone function
As part of implementing an alternate storage backend, I found myself
reimplementing this code.
With a little massaging, we can extract filelog.cmp() to a standalone
function.
As part of this, the call to revlog.cmp() was inlined (it is just a
2-line function).
I also tweaked some variable names to improve readability. I'll
further tweak names in a subsequent commit.
Differential Revision: https://phab.mercurial-scm.org/D4801
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 11:37:49 -0700] rev 40006
storageutil: extract copy metadata retrieval out of filelog
As part of implementing an alternate storage backend, I found myself
reinventing this wheel.
Let's create a utility function for doing the work.
Differential Revision: https://phab.mercurial-scm.org/D4800
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 11:29:05 -0700] rev 40005
storageutil: extract functionality for resolving strip revisions
I found myself having to copy this method as part of implementing a new
storage backend. With a little tweaking, we can extract it to a
standalone function so it can be reused easily.
We'll likely want to implement a better method for removing revisions
on the storage interface someday. But until then, let's use what we
have.
Differential Revision: https://phab.mercurial-scm.org/D4799
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 11:16:44 -0700] rev 40004
storageutil: consistently raise LookupError (API)
The interface docs say this is supposed to raise LookupError on
failure. But for invalid revision number input, it could raise
IndexError because ifileindex.node() is documented to raise
IndexError.
lookup() for files isn't used that much (pretty much just in
basefilectx in core AFAICT). And callers should already be catching
LookupError. So I don't anticipate that much fallout from this
change.
Differential Revision: https://phab.mercurial-scm.org/D4798
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 11:03:17 -0700] rev 40003
storageutil: implement file identifier resolution method (BC)
revlog.lookup() has a number of advanced features, including partial
node matching.
These advanced features aren't needed for file id lookup because file
identifiers are almost always from internal sources. (An exception to
this is hgweb, which appears to have some code paths that attempt
to resolve a user-supplied string to a node.)
This commit implements a new function for resolving file identifiers
to nodes. It behaves like revlog.lookup() but without the
advanced features.
Tests reveal behavior changes:
* Partial hex nodes are no longer resolved to nodes.
* "-1" now returns nullid instead of raising LookupError.
* "0" on an empty store now raises LookupError instead of returning
nullid.
I'm not sure why "-1" wasn't recognized before. But it seems reasonable
to accept it as a value since integer -1 resolves to nullid.
These changes all seem reasonable to me. And with the exception of
partial hex node matching, we may want to consider changing
revlog.lookup() as well.
Differential Revision: https://phab.mercurial-scm.org/D4797
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 11:00:20 -0700] rev 40002
testing: add more testing for ifileindex.lookup()
The tests demonstrate some... questionable behavior of revlog.lookup().
Differential Revision: https://phab.mercurial-scm.org/D4796
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 10:20:37 -0700] rev 40001
dagop: extract DAG local heads functionality from revlog
As part of implementing an alternate storage backend, I found myself
having to reimplement this function.
The DAG traversal logic is generic and can be factored out into a
standalone function.
We could probably combine this with headrevs(). But I'll leave that
for another time.
Differential Revision: https://phab.mercurial-scm.org/D4795
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 10:03:32 -0700] rev 40000
dagop: extract descendants() from revlog module
This method needs to be implemented in other storage backends and is
generic if we parameterize the functions for retrieving revision
numbers and parent revision numbers.
Let's extract it to dagop so backends don't need to reinvent the
wheel.
I'm not too worried about performance overhead of the extra function
call because I don't expect anyone to be calling descendants() in a
tight loop.
Differential Revision: https://phab.mercurial-scm.org/D4794
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 09:33:05 -0700] rev 39999
filelog: remove checkhash() (API)
It is unused.
While a caller may want to ask the store to validate the hash of some
provided text, since there are no in-core consumers of this method,
let's drop it for now.
Differential Revision: https://phab.mercurial-scm.org/D4793
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 09:28:38 -0700] rev 39998
filelog: remove revdiff() (API)
This proxy method is no longer used.
While it might be useful to query a storage backend for the delta
between any 2 revisions because the store could have a delta cached
and could compute it more efficiently than the caller calling
revision() twice in order to compute a delta, since nothing in core
is using this API now, I feel comfortable nuking it.
Differential Revision: https://phab.mercurial-scm.org/D4792
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 28 Sep 2018 09:46:50 -0700] rev 39997
localrepo: define storage backend in creation options (API)
We add an experimental config option to define the storage backend
for new repositories. By default, it uses "revlogv1," which maps to
the current and only modern supported repository format.
We add a "backend" creation option to control which backend to
use. It defaults to using the value from the config option.
newreporequirements() will now barf if it sees a "backend" value
that isn't "revlogv1." This forces extensions to monkeypatch the
function to handle requirements derivation for custom backends.
In order for this to "just work," we factored out obtaining the
default creation options into its own function and made callers
of newreporequirements() responsible for passing in valid data.
Without this, direct callers of newreporequirements() wouldn't
get the proper results.
Differential Revision: https://phab.mercurial-scm.org/D4791
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 27 Sep 2018 09:23:17 -0700] rev 39996
wireprotov2: derive "required" from presence of default value
If we define a default value for all optional arguments, we no
longer need to explicitly declare whether the argument is required.
Instead, we can derive it from the presence of a default value
callable.
Differential Revision: https://phab.mercurial-scm.org/D4790
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 09:05:40 -0700] rev 39995
localrepo: capture repo interface factory functions as lambas
Previously, the list would hold a reference to the original function
and attempting to wrap the module function via extension wouldn't
result in the wrapped function being called, completely undermining
the expected behavior.
Differential Revision: https://phab.mercurial-scm.org/D4821
Joerg Sonnenberger <joerg@bec.de> [Mon, 14 May 2018 00:43:07 +0200] rev 39994
extensions: new closehead module for closing arbitrary heads
``hg close-head`` allows closing arbitrary heads. It is equivalent to
checking out the given revisions and commit an empty change with
``hg commit --close-branch``.
Differential Revision: https://phab.mercurial-scm.org/D3557
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 13:12:56 -0700] rev 39993
cext: use modern buffer protocol in mpatch_flist()
Differential Revision: https://phab.mercurial-scm.org/D4841
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 02 Oct 2018 13:13:03 -0700] rev 39992
cext: use modern buffer protocol in patches()
PyObject_AsCharBuffer() is part of the "Old Buffer Protocol," which
has been deprecated for years. Let's port away from it.
PyBuffer_GetBuffer() must be paired with PyBuffer_Release(), hence the
added "goto cleanup" in a failure case.
We don't bump the extension version because the API has not changed.
Differential Revision: https://phab.mercurial-scm.org/D4840
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Mon, 01 Oct 2018 14:44:27 -0400] rev 39991
identify: when using -T, avoid unnecessary remote bookmarks query
Differential Revision: https://phab.mercurial-scm.org/D4839
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Mon, 01 Oct 2018 09:58:42 -0400] rev 39990
identify: only query remote bookmarks if needed
Instead of all the time when operating on a remote repo. This
perf regression was introduced in 15a79ac823e8, in 4.3.
This datahint method returns nothing for -Tjson, -Tpickle, -Tdebug
--config ui.formatdebug=true and --config ui.formatjson, so the
bookmarks won't show up. I don't know what these formatters are for.
plainformatter and templateformatter work properly, and the few other
uses of datahint should have the same kind of problem.
There is further weirdness where "--template '{node}'" is not enough
to avoid querying the bookmarks, you also need to pass --id or -q.
Differential Revision: https://phab.mercurial-scm.org/D4819
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 03 Oct 2018 13:59:19 +0300] rev 39989
py3: whitelist another passing tests caught by buildbot
I this indygreg recent patches made this test pass on Python 3. So thanks to
him!
Differential Revision: https://phab.mercurial-scm.org/D4848
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 03 Oct 2018 13:55:51 +0300] rev 39988
manifest: remove an unused variable caught by pyflakes
Uses of todel were removed in D4843 (19103e68a698). This patch will make
buildbots green again.
Differential Revision: https://phab.mercurial-scm.org/D4847
Matt Harbison <matt_harbison@yahoo.com> [Tue, 02 Oct 2018 22:40:01 -0400] rev 39987
setup: ignore message about disabling 3rd party extensions because of version
I started getting into a bind recently when switching between py2 and py3
because switching requires a `make clean`, which kills __version__.py. But then
when running `make local`, it picks up the local hg.exe (MSYS seems to prefix
$PATH with '.'), which doesn't know its version. That causes it to emit a
warning about needing at least 4.3 to load evolve, which caused setup.py to fail
saying there is no working hg executable to figure out the version. If we can
ignore general extension import failures, we should be able to ignore this too.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 02 Oct 2018 09:11:18 -0700] rev 39986
narrow: avoid overwriting a variable
I don't like to overwrite variables, especially not with a different
type (because it makes it harder to explain what the variable is for,
and, therefore, to name it well).
Differential Revision: https://phab.mercurial-scm.org/D4846
spectral <spectral@google.com> [Thu, 27 Sep 2018 20:16:48 -0700] rev 39985
treemanifests: remove _loadalllazy in _diff()
The benchmarks below use a similar setup as in ee7ee0c516ca and my other recent
commits. Yes, in some cases this runs in literally 5% of the time it
previously took.
before = a6f8ab53
diff --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.301 s +- 0.004 s | 1.309 s +- 0.012 s | 100.6%
m-u | | x | 1.303 s +- 0.009 s | 1.302 s +- 0.006 s | 99.9%
m-u | x | | 1.308 s +- 0.006 s | 1.309 s +- 0.007 s | 100.1%
m-u | x | x | 85.7 ms +- 0.6 ms | 86.0 ms +- 0.3 ms | 100.4%
l-d-r | | | 197.5 ms +- 0.7 ms | 197.8 ms +- 2.2 ms | 100.2%
l-d-r | | x | 199.4 ms +- 0.6 ms | 199.3 ms +- 0.9 ms | 99.9%
l-d-r | x | | 86.1 ms +- 0.5 ms | 85.8 ms +- 0.9 ms | 99.7%
l-d-r | x | x | 64.4 ms +- 0.4 ms | 64.4 ms +- 0.3 ms | 100.0%
diff -c . --git:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 236.7 ms +- 1.1 ms | 236.5 ms +- 1.3 ms | 99.9%
m-u | | x | 158.7 ms +- 1.0 ms | 128.0 ms +- 1.0 ms | 80.7% <--
m-u | x | | 239.7 ms +- 1.8 ms | 238.1 ms +- 1.5 ms | 99.3%
m-u | x | x | 132.4 ms +- 0.9 ms | 132.3 ms +- 0.6 ms | 99.9%
l-d-r | | | 81.8 ms +- 0.4 ms | 81.8 ms +- 0.3 ms | 100.0%
l-d-r | | x | 3.894 s +- 0.017 s | 193.6 ms +- 0.4 ms | 5.0% <--
l-d-r | x | | 106.9 ms +- 0.4 ms | 106.6 ms +- 0.3 ms | 99.7%
l-d-r | x | x | 182.7 ms +- 0.8 ms | 183.3 ms +- 0.9 ms | 100.3%
rebase -r . --keep -d .^^:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 5.615 s +- 0.109 s | 5.562 s +- 0.015 s | 99.1%
m-u | | x | 5.701 s +- 0.027 s | 5.715 s +- 0.023 s | 100.2%
m-u | x | | 5.572 s +- 0.128 s | 5.613 s +- 0.182 s | 100.7%
m-u | x | x | 633.3 ms +- 28.7 ms | 636.2 ms +- 13.8 ms | 100.5%
l-d-r | | | 666.4 ms +- 17.0 ms | 658.5 ms +- 9.3 ms | 98.8%
l-d-r | | x | 6.520 s +- 0.070 s | 6.505 s +- 0.026 s | 99.8%
l-d-r | x | | 279.0 ms +- 13.0 ms | 276.5 ms +- 4.7 ms | 99.1%
l-d-r | x | x | 1.636 s +- 0.058 s | 1.657 s +- 0.014 s | 101.3%
status --change . --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 218.6 ms +- 1.4 ms | 217.9 ms +- 1.5 ms | 99.7%
m-u | | x | 138.5 ms +- 0.4 ms | 108.4 ms +- 0.2 ms | 78.3% <--
m-u | x | | 220.1 ms +- 1.3 ms | 219.7 ms +- 1.5 ms | 99.8%
m-u | x | x | 113.2 ms +- 0.4 ms | 112.4 ms +- 0.8 ms | 99.3%
l-d-r | | | 80.2 ms +- 0.3 ms | 80.6 ms +- 0.6 ms | 100.5%
l-d-r | | x | 3.899 s +- 0.020 s | 194.8 ms +- 4.0 ms | 5.0% <--
l-d-r | x | | 83.4 ms +- 0.8 ms | 83.2 ms +- 0.2 ms | 99.8%
l-d-r | x | x | 732.2 ms +- 4.3 ms | 194.9 ms +- 1.0 ms | 26.6% <--
status --copies:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 1.917 s +- 0.005 s | 1.914 s +- 0.004 s | 99.8%
m-u | | x | 1.909 s +- 0.012 s | 1.934 s +- 0.004 s | 101.3%
m-u | x | | 1.915 s +- 0.005 s | 1.904 s +- 0.004 s | 99.4%
m-u | x | x | 94.8 ms +- 0.3 ms | 94.7 ms +- 0.2 ms | 99.9%
l-d-r | | | 593.9 ms +- 1.2 ms | 594.6 ms +- 9.4 ms | 100.1%
l-d-r | | x | 595.2 ms +- 3.8 ms | 597.2 ms +- 2.6 ms | 100.3%
l-d-r | x | | 182.5 ms +- 1.6 ms | 182.1 ms +- 0.6 ms | 99.8%
l-d-r | x | x | 149.6 ms +- 0.9 ms | 149.1 ms +- 0.8 ms | 99.7%
update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u | | | 3.121 s +- 0.007 s | 3.129 s +- 0.012 s | 100.3%
m-u | | x | 2.972 s +- 0.011 s | 2.981 s +- 0.012 s | 100.3%
m-u | x | | 3.144 s +- 0.014 s | 3.141 s +- 0.011 s | 99.9%
m-u | x | x | 312.2 ms +- 2.4 ms | 312.3 ms +- 2.1 ms | 100.0%
l-d-r | | | 444.4 ms +- 4.3 ms | 446.9 ms +- 5.3 ms | 100.6%
l-d-r | | x | 9.159 s +- 0.069 s | 9.182 s +- 0.040 s | 100.3%
l-d-r | x | | 254.6 ms +- 1.6 ms | 255.2 ms +- 1.6 ms | 100.2%
l-d-r | x | x | 1.525 s +- 0.007 s | 1.577 s +- 0.007 s | 103.4% <--?
Differential Revision: https://phab.mercurial-scm.org/D4845
spectral <spectral@google.com> [Tue, 02 Oct 2018 13:41:00 -0700] rev 39984
treemanifests: skip extraneous check for item before calling _loadlazy
Differential Revision: https://phab.mercurial-scm.org/D4844
spectral <spectral@google.com> [Tue, 02 Oct 2018 13:38:26 -0700] rev 39983
treemanifests: make _loadchildrensetlazy just call _loadlazy
This was suggested on the original review for _loadchildrensetlazy (D4370), it's
taken me a while to get to it though.
Differential Revision: https://phab.mercurial-scm.org/D4843
spectral <spectral@google.com> [Tue, 02 Oct 2018 13:37:12 -0700] rev 39982
treemanifests: make _loadlazy tolerate item not on _lazydirs
I'd like to clean up a few cases where we check for an item in _lazydirs before
calling _loadlazy - this will remove an extraneous dict lookup and make it
slightly more versatile.
Differential Revision: https://phab.mercurial-scm.org/D4842
Martin von Zweigbergk <martinvonz@google.com> [Wed, 08 Aug 2018 23:17:16 -0700] rev 39981
debugcommands: add a debugindexstats command
Someone went through the trouble of recording these stats, so let's
make them accessible.
Differential Revision: https://phab.mercurial-scm.org/D4832
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 19:23:50 +0900] rev 39980
rust-chg: add main program
It can at least connect to a *running* command server, and execute Mercurial
command. No signal handling nor daemon management is implemented yet.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 19:06:30 +0900] rev 39979
rust-chg: add interface to chdir the server
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 18:57:54 +0900] rev 39978
rust-chg: add Client extensions to run cHg-specific requests
This just provides a nicer way to issue command-server requests.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 18:33:46 +0900] rev 39977
rust-chg: port basic socket path handling from cHg of C
This is basically modeled after setcmdserveropts() of chg.c.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 18:21:10 +0900] rev 39976
rust-chg: add state machine to handle "runcommand" request with cHg extension
This is modeled after tokio-hglib's RunCommand state to support the "S"
channel message.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 18:18:35 +0900] rev 39975
rust-chg: add callback to handle pager and shell command requests
This could be inlined into the ChgRunCommand state to be introduced by
the next patch, but it seemed good to separate any user interactions from
the IPC code.
Yuya Nishihara <yuya@tcha.org> [Sat, 29 Sep 2018 21:59:07 +0900] rev 39974
rust-chg: add low-level function to set pager fd blocking
This is necessary because the server expects stdout/stderr to be blocking,
whereas we'll use async library to spawn pager, which makes pipes unblocking.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 16:59:12 +0900] rev 39973
rust-chg: add future that handles "attachio" request
This is the sequence to send client-side stdio and pager stdin to the
server.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 16:33:24 +0900] rev 39972
rust-chg: add parser for request messages sent to "S" channel
The data structure is documented at chgserver.py:channeledsystem().
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 16:22:03 +0900] rev 39971
rust-chg: add wrapper around C function
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 16:14:35 +0900] rev 39970
rust-chg: add function to send fds via domain socket
As a beginning, I wrote some C.
It's extracted from attachio() of contrib/chg/hgclient.c. Maybe it could
be rewritten in Rust by using the libc (and/or nix) crates, but doing that
wouldn't be trivial as the code depends on CMSG_*() macros. IMO, using C
is better here.
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 15:57:28 +0900] rev 39969
rust-chg: update dependencies
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Sep 2018 15:54:18 +0900] rev 39968
rust-chg: add project skeleton
This directory will host the reimplementation of cHg in Rust. It will use
Tokio [1], and tokio-hglib [2] which I wrote for an oxidized CHg, no idea
if there's such carbon bonding in nature btw.
[1]: https://tokio.rs/
[2]: https://bitbucket.org/yuja/tokio-hglib/
The reasoning for depending on Tokio is that it will allow us to handle Unix
signals in a safer way. Well, I believed that until I found a weird function,
handlestopsignal(), in cHg codebase. It resends the same signal to the same
process by temporarily masking the handler, which can't be inherently async.
So the signal handlers will stay in C, which means there isn't actually much
reason to write async code right now, other than I've already done most of
the async stuff, and slightly easier pager handling.
The reasoning for the rewrite is that it will eventually be possible to port
server-side config validation back to the client side, which will reduce
the complexity of the current daemon management. It will also encourage us
to write frontend library (e.g. command line and config parsers) in Rust.
The license is GPL2+ because it's likely to include derived work from the
cHg of C. The rust/chg crate is excluded from the root workspace as it's
unclear how the whole rust packages should be laid out. That can be revisited
later.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Sep 2018 12:56:57 -0700] rev 39967
narrow: move copies overrides to core
The copies overrides seems to have been a little complicated just by
not being in core. When moved to core, it becomes trivial (at least I
think these overrides have the same effect).
Differential Revision: https://phab.mercurial-scm.org/D4825
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 30 Sep 2018 18:45:16 +0300] rev 39966
narrow: pass old includes and excludes to _widen()
In future patches we will need to pass them in the widen wireprotocol command
which we are building.
Differential Revision: https://phab.mercurial-scm.org/D4812
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 28 Sep 2018 23:53:09 +0300] rev 39965
narrow: check for servers' narrow support before doing anything (BC)
Recently we introduced narrow capabilities for the server. So we can check
whether a server has narrow clone support or not before doing anything. This is
BC because new clients won't be able to extend from old narrow-enabled servers.
I *think* narrow is not used much (maybe just inside Google), also it's
experimental so I think we can change this. We will need to this someday anyway.
The "doesn't" in error is changed to "does not" because I think that's we do in
core. I also changed one more instance of the error message to use 'does not'
for consistency.
Differential Revision: https://phab.mercurial-scm.org/D4789
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 30 Sep 2018 18:59:27 +0300] rev 39964
narrow: don't do the dirstate dance if ellipses is not enabled
I believe we set dirstate parents to nullid before widening pull because in
ellipses cases, the parent might be stripped off with a new changeset. However
the second ds.setparents() call invalidate my assumption. I am not sure why we
do this. So here is a patch.
This patch also adds tests showing we break nothing in non-ellipses cases.
Differential Revision: https://phab.mercurial-scm.org/D4788
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 28 Sep 2018 19:21:24 +0300] rev 39963
narrow: pass 'narrow_widen' as source while generating changegroup
Extensions inspect the `source` parameter and add some custom logic on the basis
of value of that parameter. The parameter is also passed to hooks. So let's pass
the value as 'narrow_widen' to differentiate widening from pull.
Differential Revision: https://phab.mercurial-scm.org/D4787
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 28 Sep 2018 19:18:17 +0300] rev 39962
narrow: factor out logic to create cg while widening into separate fn
This patch takes out the logic which generates a changegroup for widening a
narrow clone when ellipses are disabled. This is done because future patches
will introduce a narrow_widen() wireprotocol command which will send a
bundle2 with changegroup and will use this function.
The new function for now returns just the changegroup for compatibility with
existing code, but in future patches, when we establish a wireprotocol command
and call this function from there, this will return the required bundle2.
Differential Revision: https://phab.mercurial-scm.org/D4786
Martin von Zweigbergk <martinvonz@google.com> [Mon, 01 Oct 2018 15:29:31 -0700] rev 39961
narrow: avoid looking up dirstate again when editing dirstate
The narrow extension overrides the dirstate editing functions to
restrict paths outside the narrowspec. These overrides access the
dirstate from repo.dirstate instead of using the "self" reference
passed to the overridden functions. I don't see a reason for this and
it caused me problems with a later patch (it caused infinite recursion
when I modified localrepo.dirstate()), so let's change it.
Differential Revision: https://phab.mercurial-scm.org/D4829
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 23:09:28 -0700] rev 39960
repo: move unfiltered-repo optimization to workingctx
localrepo.__getitem__ special-cased lookup of the working copy parent
to avoid looking up obsmarkers. I think the reason for that code
(which I once wrote myself) was to make `hg commit` not load
obsmarkers, which it would otherwise do via ctx.p1() in
localrepo.commitctx().
That had the somewhat unfortunate consequence of making lookup of an
unrelated binary nodeid load the dirstate. Now that changectx's
constructor is dumb, we can let workingctx._parents() have the
opmtimization instead.
This affects two tests, because they no longer end up loading the
dirstate and their "warning: ignoring unknown working parent ..."
messages therefore go way.
Differential Revision: https://phab.mercurial-scm.org/D4828
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 22:53:14 -0700] rev 39959
context: move logic from changectx.__init__ to localrepo.__getitem__ (API)
My motivation for this change was to make repo[node] not load the
dirstate (more about that in the next patch), but I think it makes
more sense this way too. For example, raising RepoLookupError seems to
belong better in the repo lookup function
(i.e. localrepo.__getitem__).
This makes the changectx constructor very simple -- it just assigns
the given repo, revnum, and nodeid to properties.
Differential Revision: https://phab.mercurial-scm.org/D4827
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 22:44:51 -0700] rev 39958
context: reduce dependence of changectx constructor
I want to change the constructor's signature and letting all creation
of changectx instances go through the repo simplifies that.
Differential Revision: https://phab.mercurial-scm.org/D4826
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 23:11:07 -0700] rev 39957
py3: use system strings for HTTP response header comparison
res.headers is holding system strings. Attempting to compare against
bytes always fails and we fail to print decoded CBOR responses on
Python 3.
After this change, various test-wireproto* tests are now properly
printing CBOR response objects.
Differential Revision: https://phab.mercurial-scm.org/D4835
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 23:08:04 -0700] rev 39956
py3: convert HTTP request headers to str
The low-level request object ideally takes system strings for
HTTP request headers and values. If we send in bytes, it works. But
a duplicate header check fails and various tests emit duplicate
user-agent headers.
Differential Revision: https://phab.mercurial-scm.org/D4834
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 23:12:42 -0700] rev 39955
py3: use system strings in HTTP server code
Previously the source transformer was converting some string literals
to bytes and we were comparing a system native string to bytes
and this was leading to sending the wrong HTTP response headers on
Python 3.
After this change, we now properly send Transfer-Encoding and
Connection response headers on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4833
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 01 Oct 2018 23:39:49 -0700] rev 39954
py3: byteify test-storage.py
The test now passed on Python 3.
# skip-blame just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4836
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 02 Oct 2018 14:29:56 +0300] rev 39953
narrow: remove narrowpatch instead of narrowcopies
84092edd5c88 removed narrowpatch.py and wanted to drop narrowpatch from
__init__.py but mistakenly removed narrowcopies.
This will make buildbots green again.
Differential Revision: https://phab.mercurial-scm.org/D4837
Matt Harbison <matt_harbison@yahoo.com> [Mon, 01 Oct 2018 23:26:24 -0400] rev 39952
py3: suppress the output from .write() calls in more tests
I missed these in 803b7569c9ea because I forgot about `...` lines.
Matt Harbison <matt_harbison@yahoo.com> [Mon, 01 Oct 2018 23:07:19 -0400] rev 39951
py3: byteify test-lfs.t
Matt Harbison <matt_harbison@yahoo.com> [Mon, 01 Oct 2018 23:04:58 -0400] rev 39950
lfs: explicitly name a key when sorting blob pointers
This is needed to keep py3 happy. The other two instances of sorting already
did this.
Matt Harbison <matt_harbison@yahoo.com> [Mon, 01 Oct 2018 21:54:00 -0400] rev 39949
py3: byteify test-lock.py
There's still a stacktrace about not being able to serialize _io.TextIOWrapper.
Matt Harbison <matt_harbison@yahoo.com> [Mon, 01 Oct 2018 21:48:45 -0400] rev 39948
py3: byteify contrib/dumprevlog
Matt Harbison <matt_harbison@yahoo.com> [Mon, 01 Oct 2018 19:39:05 -0400] rev 39947
py3: quote $PYTHON in test-merge-symlinks.t
Martin von Zweigbergk <martinvonz@google.com> [Thu, 21 Jun 2018 22:07:34 -0700] rev 39946
copies: inline a variable that's used only once
By inlining it, we also avoid calculating the value when "if of in
seen" is false (probably not significant).
Differential Revision: https://phab.mercurial-scm.org/D4831
Martin von Zweigbergk <martinvonz@google.com> [Sat, 15 Sep 2018 22:56:57 -0700] rev 39945
bundlerepo: remove a variable alias
"parentrepo" and "repo" were the same thing and I don't see much
reason for it (unionrepo has similar structure and a similar alias but
there are two repos there so at least it makes a little more sense
there).
Differential Revision: https://phab.mercurial-scm.org/D4830
Augie Fackler <augie@google.com> [Mon, 01 Oct 2018 16:11:48 -0400] rev 39944
merge with stable
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Sep 2018 12:05:48 -0700] rev 39943
narrow: move support for `hg verify` into core
Differential Revision: https://phab.mercurial-scm.org/D4824
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Sep 2018 13:55:17 -0700] rev 39942
narrow: drop unnecessary overrides of patch
The patch overrides became unnecessary when I made context.status()
filter by the narrowspec in e411774a2e0f (narrow: move
status-filtering to core and to ctx, 2018-08-02).
Differential Revision: https://phab.mercurial-scm.org/D4823
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Sep 2018 00:00:36 -0700] rev 39941
tests: add test showing that paths outside narrowspec are not added
We had a test that shows that trying to explicitly add a file outside
the narrowspec results in an error, but we didn't have a test that
shows that paths from subdirectories outside the narrowspec are not
added (on e.g. `hg add .`) but silently ignored.
Differential Revision: https://phab.mercurial-scm.org/D4822
Augie Fackler <augie@google.com> [Mon, 01 Oct 2018 09:05:41 -0400] rev 39940
catapipe: fix bug introduced when switching to the timer api
Differential Revision: https://phab.mercurial-scm.org/D4818
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 30 Sep 2018 03:45:51 +0530] rev 39939
narrow: move the ellipses server capability to core
This will be used in core logic to determining whether a server is ellipses
enabled or not. And also this will ease moving narrow related things to core.
Differential Revision: https://phab.mercurial-scm.org/D4809
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 30 Sep 2018 03:42:35 +0530] rev 39938
narrow: move the wireprotocol narrow capability name to core
We are trying to integrate the whole of narrow logic into core and it will be
helpful for upcoming patches to have these capability names in core. The next
patch will move the ellipses capability to core also.
The exact motivation is to know whether we are cloning a ellipses repo or not
and adding an ellipses repo requirement.
Differential Revision: https://phab.mercurial-scm.org/D4808
Yuya Nishihara <yuya@tcha.org> [Sun, 30 Sep 2018 16:11:35 +0900] rev 39937
help: document about "files" template keywords
Yuya Nishihara <yuya@tcha.org> [Sun, 30 Sep 2018 15:53:36 +0900] rev 39936
formatter: remove experimental marker from -T option
I think it's good enough to stabilize the feature, at least for the core
and non-experimental commands.
Yuya Nishihara <yuya@tcha.org> [Sun, 30 Sep 2018 15:37:09 +0900] rev 39935
grep: rename {line_number} to {lineno} as well (BC)
Yuya Nishihara <yuya@tcha.org> [Sun, 30 Sep 2018 15:35:17 +0900] rev 39934
annotate: rename {line_number} to {lineno} (BC)
I think {lineno} looks more like a common template keyword. It isn't called
a {line} to avoid conflicts with the element name of {lines} and the
{_|splitlines} filter.
https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary
Yuya Nishihara <yuya@tcha.org> [Sun, 30 Sep 2018 15:15:27 +0900] rev 39933
resolve: rename {status} to {mergestatus} to not shadow change status (BC)
This is a part of the name unification. I think it's somewhat useful to
provide a combined view of change/merge statuses.
https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary
Augie Fackler <augie@google.com> [Mon, 01 Oct 2018 09:06:01 -0400] rev 39932
py3: whitelist one additional test
Differential Revision: https://phab.mercurial-scm.org/D4817
Boris Feld <boris.feld@octobus.net> [Fri, 03 Nov 2017 21:11:07 +0100] rev 39931
logtoprocess: connect all fds to /dev/null to avoid bad interaction with pager
We detected that pager is waiting for log-to-process script to finish, which
is annoying when adding a script on commandfinish that does an HTTP push.
There seems to be no workaround on the script side and it will make the
behavior on Linux/MacOS closer to the Windows behavior.
The drawback is that it makes the related tests more flaky as log-to-process
outputs are now really asynchronous.
If it's considered a BC change, another option would be to add a config option
for this new behavior. I personally think that the different behavior between
Windows and Linux is confusing and that it's a bug I would be fine with a new
config option.
Differential Revision: https://phab.mercurial-scm.org/D4816
Boris Feld <boris.feld@octobus.net> [Fri, 03 Nov 2017 21:35:36 +0100] rev 39930
logtoprocess: add a test to show pager and ltp bad interaction
The next changeset will fix the bad interaction between the pager and log-to-
process. Add a test first to make the next changeset easier to understand.
Differential Revision: https://phab.mercurial-scm.org/D4815
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 30 Sep 2018 21:13:16 +0300] rev 39929
py3: add one more passing test to whitelist caught by buildbot
Thanks to Matt Harbison who fixed the remaining failures of this test.
Differential Revision: https://phab.mercurial-scm.org/D4814
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 12:06:32 +0200] rev 39928
cleanupnodes: pass multiple predecessors to `createmarkers` directly
Boris Feld <boris.feld@octobus.net> [Sat, 22 Sep 2018 14:40:33 +0200] rev 39927
obsolete: allow multiple predecessors in createmarkers
The logic for this change is similar to the change to `cleanupnodes` that we
did earlier. Now that the rebase code is trying to record a fold, we need to
actually record it in the markers. The first step is to have the markers
creation API able to receive such fold data.
To keep things sane, we restrict fold to on successors.
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 21:03:09 +0200] rev 39926
obsolete: preindent code in createmarkers
This will help to make the next change clearer.
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 23:37:43 +0200] rev 39925
obsolete: explicitly pass relation items to effectflag computation
To track folds, we are about to change the content of `rel`. To simplify this
change, we update the `geteffectflag` function beforehand.
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 22:45:44 +0200] rev 39924
rebase: explicitly track collapses as fold
A collapse is a large fold. It is now explicitly tracked (at the cleanupnodes
API level).
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 23:51:11 +0200] rev 39923
rebase: use tuple as `replacement` keys
Now that `cleanupnodes` support tuples as key, we update the rebase code to use
them. No changes in the replacement tracked are introduced yet.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 00:15:21 +0200] rev 39922
rebase: expand a long "one-liner"
When a one-liner gets 3 lines longs, it lose its expressivity benefits. We
expand it into a simple for loop. This makes future changes of the code in
that area clearer.
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 21:28:21 +0200] rev 39921
cleanupnodes: drop special casing around prune markers (API)
The `cleanupnodes` has logic to skip the creation of "prune" markers if the
changeset is already obsolete. This feels strange and gets in the way of code
changes to tracks folds. Now that callers no longer request such prune, we can
drop this logic.
In many cases, pruning through cleanupnodes should be replaced by internal
phase usage.
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 22:05:28 +0200] rev 39920
rebase: don't try to prune obsolete changeset already in the destination
With similar motivations to the previous changesets, we stop marking changeset
from pruning when it is not the command intention. In this case, we still need
to distinguish between the strip and the obsolete case.
Boris Feld <boris.feld@octobus.net> [Wed, 26 Sep 2018 21:51:29 +0200] rev 39919
histedit: don't cleanup nodes already disposed of
If something else took care of these temporary nodes, we don't need to do
anything about it. This less liberal usage of pruning through cleanup nodes
will help us further cleanup on the road to explicitly tracks folds.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 30 Sep 2018 01:15:46 -0400] rev 39918
py3: avoid b'' output in test-hgweb-non-interactive.t
I couldn't figure out how to get the list to print without b'' with
pycompat.write(), without converted each element to str.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 30 Sep 2018 01:10:41 -0400] rev 39917
py3: convert print to a function call in a few tests
I *think* this is the last of them.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 29 Sep 2018 23:52:00 -0400] rev 39916
py3: use util.forcebytestr() to convert push lock error to bytes
Matt Harbison <matt_harbison@yahoo.com> [Sat, 29 Sep 2018 20:57:49 -0400] rev 39915
py3: byteify windows.shelltocmdexe()
This makes test-doctest.py happy on Windows.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 29 Sep 2018 20:33:52 -0400] rev 39914
run-tests: flush output stream before prompting to accept changes
With py3 on Windows, the prompt to accept changes in an --interactive run wasn't
showing up until after the choice was read from stdin.
Pulkit Goyal <pulkit@yandex-team.ru> [Sun, 30 Sep 2018 05:52:42 +0530] rev 39913
py3: use util.forcebytestr() to convert error messages to bytes
This makes the python 3 buildbot green again.
Differential Revision: https://phab.mercurial-scm.org/D4811
Martin von Zweigbergk <martinvonz@google.com> [Thu, 27 Sep 2018 14:26:02 -0700] rev 39912
context: stop catching TypeError when converting hex nodeid to binary
It has been a programming error to pass a 40-character string that is
not a hex nodeid since 8b86acc7aa64 (context: drop support for looking
up context by ambiguous changeid (API), 2018-04-28), so we can just
let it raise a TypeError.
Differential Revision: https://phab.mercurial-scm.org/D4807
Martin von Zweigbergk <martinvonz@google.com> [Thu, 27 Sep 2018 09:56:13 -0700] rev 39911
context: stop catching and re-raising FilteredRepoLookupError
FilteredRepoLookupError is only raised by changectx's constructor and
the higher-level scmutil.revsymbol(), so there's no need to catch it
in changectx's constructor.
Differential Revision: https://phab.mercurial-scm.org/D4806
Matt Harbison <matt_harbison@yahoo.com> [Fri, 28 Sep 2018 22:18:45 -0400] rev 39910
util: use a context manager in readlock()
Matt Harbison <matt_harbison@yahoo.com> [Tue, 25 Sep 2018 21:16:12 -0400] rev 39909
py3: convert os.readlink() path to native strings on Windows
Windows insisted that it needs to be str. I skipped the stuff in the posix
module, and left `tests/f` and `run-tests.py` alone for now.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 29 Sep 2018 02:02:35 -0400] rev 39908
py3: suppress the output from .write() calls in the remaining tests
Matt Harbison <matt_harbison@yahoo.com> [Fri, 28 Sep 2018 23:17:06 -0400] rev 39907
py3: conditionalize json float precision difference in test-debugcommands.t
Matt Harbison <matt_harbison@yahoo.com> [Fri, 28 Sep 2018 22:39:18 -0400] rev 39906
py3: byteify extension in test-debugcommands.t
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 16:55:06 +0200] rev 39905
pullreport: rev duplicated and extinct into account
If we already have some obsolete and hidden nodes locally and the server send
them again to you, it seems useful to point it out instead of being silent about
it.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 16:52:25 +0200] rev 39904
pullreport: issue a message about "extinct" pulled changesets
Changeset pulled from a remote repository while already obsolete locally can
end up hidden after the pull. Hiding obsolete changesets is a good behavior
but silently "skipping" some of the pulled content can get confusing.
We now detect this situation and emit a message about it. The message is
simple and the wording could be improved, however, we focus on the detection
here. Evolution is still an experimental feature, so the output is open to
changes.
In particular, we could point out at the latest successors of the obsolete
changesets, however, it can get tricky is there are many of them. So we delay
these improvements to another adventure.
Another easy improvement would be to merge this message with the previous line
about the new nodes and their phases.
This is a good example of cases where we can only transmit a limited amount of
data to users by default. We need some sort of "transaction journal" we could
point the user to.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 17:00:00 +0200] rev 39903
pullreport: skip or rework some early return
We are about to add more logic in this report. Before that, we need it to not
quit so early.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 16:35:10 +0200] rev 39902
pullreport: skip filtered revs instead of obsolete ones
Obsolescence is closely related to visibility but still a distinct concept. We
can receive changesets that are obsolete but visible (eg: when pulling
orphans). Such changeset should be reported too. In addition, the filtering
level can be anything, we should respect it.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 18:06:13 +0200] rev 39901
pullreport: add a test to show misreporting of visible changeset
The current code ignores all obsolete changesets including the visible one. We
add a test showing this behavior before fixing the behavior.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 10:38:37 -0700] rev 39900
repo: don't look up context for tip node if it's not needed
We were doing repo['tip'].node() or similar in a few places where
repo.changelog.tip() would be enough.
Differential Revision: https://phab.mercurial-scm.org/D4781
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 22:17:34 -0700] rev 39899
repo: look up nullrev context by revnum, not symbolic name
I think lookup of the 'null' symbol should be done via
scmutil.revsymbol() and repo['null'] is only supported for historical
reasons. However, repo[nullrev] is fine, so we can switch to that
instead of switching to scmutil.revsymbol('null').
Differential Revision: https://phab.mercurial-scm.org/D4780
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 22:17:26 -0700] rev 39898
bundle: consistently put revnums in "base" collection
The "base" collection contains revnums, except that it can also
contain the "null" symbol. That's a little weird. Let's be consistent.
Differential Revision: https://phab.mercurial-scm.org/D4779
Matt Harbison <matt_harbison@yahoo.com> [Thu, 27 Sep 2018 22:07:12 -0400] rev 39897
py3: ensure printing to stdout uses str in test-hgweb-no-request-uri.t
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 13:57:50 -0700] rev 39896
scmutil: accept multiple predecessors in 'replacements' (API)
This changeset makes 'cleanupnodes' accepts multiple predecessors as
`replacements` keys. The same as it accepts multiple successors as
`replacements` values. To avoid breaking all callers, the old and new ways are
currently valid at the same time. We'll deprecate and drop the old way later.
This change is the first step toward a better tracking of "fold" event in the
evolution history. While working on the "rewind" command (in the evolve
extension), we realized that first class tracking of folds are necessary.
We already have good tracking of splits. When walking the evolution history
from predecessors to successors, that makes for a clear distinction between
having multiple successors because of the actual splitting of a changeset or
content-divergences.
The "rewind" command allows restoring older evolution of a stack of
changesets. One of its mode walks the evolution history to automatically find
appropriate predecessors. This means walking from successors to predecessors.
In this case, we need to be able to make the same distinction between an
actual fold and other cases. So we will have to track folds explicitly.
This changesets only focus on making it possible to express fold at the
`cleanupnodes` API level. The actual tracking will be implemented later.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Sep 2018 13:54:37 -0700] rev 39895
scmutil: expand long "one-liner"
When a one-liner gets 3 lines longs, it loses its expressivity benefits. We
expand it into a simple for loop. This makes future changes of the code in
that area clearer.
Boris Feld <boris.feld@octobus.net> [Thu, 20 Sep 2018 17:47:05 +0200] rev 39894
shelve: find shelvedctx from bundle even if they are already in the repo
We use the new "duplicates" node tracking to find the tip of the bundle even if
it already exists in the repository.
Such logic is not supposed to be needed in theory. If the shelve was made using
internal-phase, we already know its node. Otherwise, the bundle content should
have been stripped. However, handling it makes the shelve code more robust and
provide a good example of "revduplicates" usage.
Boris Feld <boris.feld@octobus.net> [Wed, 19 Sep 2018 12:19:28 +0200] rev 39893
shelve: return the shelved node as part of bundle application
It make sense to have the function in charge of unbundling the shelved revision
also return the node of that revision (when the data is in the bundle).
This will help us to handle unnatural state where the unshelved change already
exists in the repository.
Boris Feld <boris.feld@octobus.net> [Thu, 20 Sep 2018 11:18:28 +0200] rev 39892
changelog: keep track of duplicated node in the transaction adding them
The transaction is already tracking the new nodes. We now tracks the
"duplicates" in the same location.
Boris Feld <boris.feld@octobus.net> [Wed, 19 Sep 2018 21:02:47 +0200] rev 39891
revlog: add a callback "tracking" duplicate node addition
If a changegroup contains node already added to the repository, they will be
skipped. Skipping them is the right behavior (we don't need to store things
twice), but it can hide some information to the code doing the unbundle (eg:
shelve looking for the tip of the bundle).
The first step to improve this situation is to add a low level callback. We do
not need this tracking on all revlog, so actual tracking will be added in the
next changeset.
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 26 Sep 2018 18:30:19 -0400] rev 39890
logtoprocess: define $HG for children processes
So they can compute the hg version for instance.
Differential Revision: https://phab.mercurial-scm.org/D4768
Matt Harbison <matt_harbison@yahoo.com> [Wed, 26 Sep 2018 22:21:25 -0400] rev 39889
py3: mask out None type when printing in `debuglocks`
Apparently, %b doesn't allow None.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 26 Sep 2018 21:25:18 -0400] rev 39888
py3: ensure standard exceptions use `str` type strings in windows.py
See also edaa40dc5fe5.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 26 Sep 2018 20:49:28 -0400] rev 39887
py3: replace a StandardError reference
This doesn't exist on py3, and the standard way of handling this seems to be to
catch both exceptions.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 15:19:52 -0700] rev 39886
storageutil: extract revision number iteration
This code is a bit quirky (and possibly buggy). It will likely be used
by multiple storage backends. Let's extract it so it is reusable.
Differential Revision: https://phab.mercurial-scm.org/D4757
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:54:28 -0700] rev 39885
storageutil: new function for extracting metadata-less content from text
Other storage backends will want to do this.
I'm not concerned about Python function call overhead because I
expect self.revision() to dwarf the function call overhead time,
since self.revision() requires multiple function calls and may
involve decompression in the common case.
Differential Revision: https://phab.mercurial-scm.org/D4756
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:33:45 -0700] rev 39884
storageutil: move _censoredtext() from revlog
This seems like generic functionality we'll want to use from
non-revlog storage backends.
Differential Revision: https://phab.mercurial-scm.org/D4755
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:31:31 -0700] rev 39883
storageutil: move metadata parsing and packing from revlog (API)
Parsing and writing of revision text metadata is likely identical
across storage backends. Let's move the code out of revlog so we
don't need to import the revlog module in order to use it.
Differential Revision: https://phab.mercurial-scm.org/D4754
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 14:23:54 -0700] rev 39882
storageutil: new module for storage primitives (API)
There will exist common code between storage backends. It would
be nice to have a central place to put that code.
This commit attempts to create that place by creating the
"storageutil" module.
The first thing we move is revlog.hash(), which is the function for
computing the SHA-1 hash of revision fulltext and parents.
Differential Revision: https://phab.mercurial-scm.org/D4753
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 13:35:50 -0700] rev 39881
filelog: stop proxying deltaparent() (API)
deltaparent() obtains the revision number of the base revision a
delta in storage is stored against. It is highly revlog-centric and
may not apply to other storage backends. As a result, it doesn't
belong on the generic file storage interface.
This method/proxy is no longer used in core. The last consumer was
probably changegroup code and went away with the transition to
emitrevisions().
Differential Revision: https://phab.mercurial-scm.org/D4751
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 12:49:17 -0700] rev 39880
filelog: stop proxying rawsize() (API)
This method is no longer used by external consumers. The API is
quite low-level and is effectively len(revision(raw=True)). I don't
see a compelling reason to keep it around.
Let's drop the API and make the file storage interface simpler.
Differential Revision: https://phab.mercurial-scm.org/D4750
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 12:42:03 -0700] rev 39879
filelog: stop proxying "opener" (API)
The last consumer of it in upgrade code was removed as part of the
previous commit. This attribute is revlog specific (because it
assumes the existence of a vfs for performing I/O on tracked file
data) and therefore isn't appropriate for a generic storage interface.
So nuke it.
Differential Revision: https://phab.mercurial-scm.org/D4749
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 11:16:33 -0700] rev 39878
filelog: stop proxying flags() (API)
Per-revision storage flags are kinda a revlog-centric API. (Except for
the fact that changegroup uses the same integer flags as revlog does
and there's minimal verification that the server's flags map to the
client's storage flags - but that's another problem.)
The last user of flags() was in verify.py and that code was just moved
into revlog.py and is accessed behind the verifyintegrity() file
storage API.
Since there are no more consumers, let's drop the proxy and remove
the method from the file storage interface.
This commit only drops the dedicated API for reading a single
revision's storage flags: we still support reading and writing flags
through the bulk data retrieval and add revision APIs. And since
changegroups encode revlog integer flags over the wire, we'll always
need to support flags at some level. The removal of individual storage
flags may be too premature. But since flags() is now unused, I'd like
to see how far we can get without that dedicated API - especially
since it uses revision numbers instead of nodes.
Differential Revision: https://phab.mercurial-scm.org/D4746
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 11:27:47 -0700] rev 39877
revlog: move revision verification out of verify
File revision verification is performing low-level checks of file
storage, namely that flags are appropriate and revision data can
be resolved.
Since these checks are somewhat revlog-specific and may not
be appropriate for alternate storage backends, this commit moves
those checks from verify.py to revlog.py.
Because we're now emitting warnings/errors that apply to specific
revisions, we taught the iverifyproblem interface to expose the
problematic node and to report this node in verify output. This
was necessary to prevent unwanted test changes.
After this change, revlog.verifyintegrity() and file verify code
in verify.py both iterate over revisions and resolve their fulltext.
But they do so in separate loops. (verify.py needs to resolve
fulltexts as part of calling renamed() - at least when using revlogs.)
This should add overhead.
But on the mozilla-unified repo:
$ hg verify
before: time: real 700.640 secs (user 585.520+0.000 sys 23.480+0.000)
after: time: real 682.380 secs (user 570.370+0.000 sys 22.240+0.000)
I'm not sure what's going on. Maybe avoiding the filelog attribute
proxies shaved off enough time to offset the losses? Maybe fulltext
resolution has less overhead than I thought?
I've left a comment indicating the potential for optimization. But
because it doesn't produce a performance regression on a large
repository, I'm not going to worry about it.
Differential Revision: https://phab.mercurial-scm.org/D4745
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Sep 2018 12:06:44 -0700] rev 39876
tests: de-flake test-narrow-debugrebuilddirstate.t
If the dirstate gets written much later (usually 1-2 s, depending on
FS) than the working copy file (there's only one), then the `hg
debugdirstate` command will include a timestamp. There's nothing wrong
with that, so we should just allow it.
Differential Revision: https://phab.mercurial-scm.org/D4758
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 12:39:34 -0700] rev 39875
upgrade: use storageinfo() for obtaining storage metadata
Let's switch to our new API for obtaining information about storage.
This eliminates the last consumer of rawsize() and the opener proxy
from the file storage interface!
Differential Revision: https://phab.mercurial-scm.org/D4748
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 11:56:48 -0700] rev 39874
revlog: add method for obtaining storage info (API)
We currently have a handful of methods on the file and manifest
storage interfaces for obtaining metadata about storage. e.g.
files() is used to obtain the files backing storage. rawsize()
is to quickly compute the size of tracked revisions without resolving
their fulltext.
Code in upgrade and stream clone make heavy use of these methods.
The existing APIs are generic and don't necessarily have the
specialization that we need going forward. For example, files()
doesn't distinguish between exclusive storage and shared storage.
This makes stream clone difficult to implement when e.g. there may
be a single file backing storage for multiple tracked paths. It
also makes reporting difficult, as we don't know how many bytes are
actually used by storage since we can't easily identify shared files.
This commit implements a new method for obtaining storage metadata.
It is designed to accept arguments specifying what metadata to request
and to return a dict with those fields populated. We /could/ make
each of these attributes a separate method. But this is a specialized
API and I'm trying to avoid method bloat on the interfaces. There is
also the possibility that certain callers will want to obtain multiple
fields in different combinations and some backends may have performance
issues obtaining all that data via separate method calls.
Simple storage integration tests have been added. For now, we assume
fields can't be "None" (ignoring the interface documentation). We can
revisit this later.
Differential Revision: https://phab.mercurial-scm.org/D4747
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 11:27:41 -0700] rev 39873
lfs: drop unused import
A recent change dropped the last user of this module.
Differential Revision: https://phab.mercurial-scm.org/D4744
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 10:08:58 -0700] rev 39872
filelog: drop _generaldelta attribute (API)
With changegroup moving to emitrevisions(), this revlog-specific
attribute is no longer used and can be deleted. Good riddance.
Differential Revision: https://phab.mercurial-scm.org/D4727
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 09:59:19 -0700] rev 39871
revlog: drop emitrevisiondeltas() and associated functionality (API)
emitrevisions() is the future!
Differential Revision: https://phab.mercurial-scm.org/D4726
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 21 Sep 2018 18:47:04 -0700] rev 39870
changegroup: port to emitrevisions() (issue5976)
We now have a unified API for emitting revision data from a storage
backend. It handles sorting nodes and the complicated delta versus
revision decisions for us.
This commit ports changegroup to that API.
There should be no behavior changes for changegroups not using
ellipsis. And lack of test changes seems to confirm that.
There are some changes for ellipsis mode, however.
Before, when sending an ellipsis revision, we would always send a
fulltext revision (as opposed to a delta). There was a TODO tracking
this open item.
One of the things the emitrevisions() API does for us is figure out
whether we can safely emit a delta. So, it is now possible for
ellipsis revisions to be sent as deltas! (It does this by not
assuming parent/ancestor revisions are available and tracking which
revisions have been sent out.)
Because we eliminated the list of revision delta request objects,
performance has improved substantially:
$ hg perfchangegroupchangelog
before: ! wall 24.348077 comb 24.330000 user 24.140000 sys 0.190000 (best of 3)
after: ! wall 18.245911 comb 18.240000 user 18.100000 sys 0.140000 (best of 3)
That's a lot of overhead for creating a few hundred thousand Python
objects!
This is still a little slower than 4.7. Probably due to 23d582ca
introducing a type for the revision/delta results. There is
potentially room to optimize. But at some point we need to abstract
storage in order to support alternate storage backends. Unfortunately
that means using a Python data structure to represent results. And
unfortunately there is overhead with every new Python object created.
Differential Revision: https://phab.mercurial-scm.org/D4725
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 24 Sep 2018 09:48:02 -0700] rev 39869
wireprotov2server: port to emitrevisions()
We now have a proper storage API to request data on multiple
revisions. We can drop it into wire protocol version 2 with
minimal effort.
The new API handles pretty much everything we were doing manually to
build up the delta request. So we were able to delete a lot of code.
As a bonus, wireprotov2 code is no longer accessing some low-level
storage APIs. This includes the assumption that a node has an
associated numeric revision number! This should make it drastically
simpler to implement a server that doesn't have the concept of
revision numbers.
Differential Revision: https://phab.mercurial-scm.org/D4724
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 21 Sep 2018 14:54:59 -0700] rev 39868
tests: use more complex file storage test
The previous test was attempting to to test delta storage behavior.
It didn't do a very good job at it because there was a good chance
a delta wasn't being used in storage.
Let's switch the test to yield a delta in storage so an upcoming
change to delegate delta logic to storage has the desired effect.
Differential Revision: https://phab.mercurial-scm.org/D4723
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 21 Sep 2018 14:28:21 -0700] rev 39867
revlog: new API to emit revision data
I recently refactored changegroup generation code to make it more
storage agnostic. I made significant progress. But there is still
a bit of work to be done. Specifically:
* Changegroup code is looking at low-level storage attributes to
influence sorting. Sorting should be done at the storage layer.
* The linknode lookup and sorting code for ellipsis is very
complicated.
* Linknodes are just generally wonky because e.g. file storage doesn't
know how to translate a linkrev to a changelog node.
* We regressed performance when introducing the request-response
objects.
Having thought about this problem a bit, I think I've come up with
a better interface for emitting revision deltas.
This commit defines and implements that interface. See the docstring
in repository.py for more info.
This API adds 3 notable features over the previous one.
First, it defers node ordering to the storage implementation in
the common case but allows overriding as necessary. We have a
facility for requesting an exact ordering (used in ellipsis
mode). We have another facility for storage order (used for changelog).
Second, we have an argument specifying assumptions about parents
revisions. This can be used to force a fulltext revision when we
don't know the receiver has a parent revision to delta against.
Third, we can control whether revision data is emitted. This makes
the API suitable as a generic "index data retrieval" API as well
as for producing revision deltas - possibly in the same operation!
The new API is much simpler: we no longer need a complicated "request"
object to encapsulate the delta generation request. I'm optimistic
this will restore performance loss associated with
emitrevisiondeltas().
Storage unit tests for the new API have been implemented.
Future commits will port existing consumers of emitrevisiondeltas()
to the new API then remove emitrevisiondeltas().
Differential Revision: https://phab.mercurial-scm.org/D4722