Yuya Nishihara <yuya@tcha.org> [Sat, 03 Mar 2018 06:08:22 -0500] rev 36648
py3: bulk-replace bytes format specifier passed to Py_BuildValue()
On Python 3, "s" means a utf-8 string. We have to use "y" for bytes, sigh.
https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue
Substituted using the following pattern with some manual fixes:
'\b(Py_BuildValue)\((\s*)"([^"]+)"'
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Mar 2018 05:58:41 -0500] rev 36647
py3: add PY23() macro to switch string literal depending on python version
I have no better idea to work around the bytes-unicode divergence of
Py_BuildValue(). Maybe we can write a code transformer for C extensions? :)
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Mar 2018 05:50:45 -0500] rev 36646
py3: don't try to mangle C extension blob by code transformer
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 18:47:27 -0500] rev 36645
tests: add missing b prefixes in test-atomictempfile.py
# skip-blame just some b prefixes
Differential Revision: https://phab.mercurial-scm.org/D2570
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 16:31:44 -0800] rev 36644
wireproto: only expose "between" to version 1 of wire protocols
We recently marked other legacy commands as only available to version 1
of the wire protocols. We held off marking "between" because it is
used as part of the SSH handshake.
Since SSH servers assume they are version 1 by default and the
"between" request that is issued as part of the version 2 handshake
is swallowed and not operated on, everything should "just work" if
"between" is not available to version 2.
Differential Revision: https://phab.mercurial-scm.org/D2513
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 02 Mar 2018 18:55:18 -0500] rev 36643
tests: add more tests around hook output and getbundle
The previous tests around hook output only tested Python hooks. Let's
add some shell hooks in for additional test coverage.
Differential Revision: https://phab.mercurial-scm.org/D2550
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 16:24:02 -0800] rev 36642
wireproto: add transport specific capabilities in the transport
We add a method to the protocol handler interface that allows
specific implementations to add their own capabilities.
The SSH implementation is a no-op.
The HTTP implementation adds the HTTP-specific capabilities.
The end result is transport specific capabilities now live in the
transport code instead of in some generic function in the wireproto
module.
Differential Revision: https://phab.mercurial-scm.org/D2512
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 15:23:04 -0800] rev 36641
wireproto: don't expose changegroupsubset capability if not available
We just marked the changegroupsubset command as only available to
version 1 of the wire transports. There is a capability of the same
name of the command that indicates if the command is supported.
This commit teaches the capabilities code to conditionally emit that
capability depending on whether the command is available for the
current transport.
Most test output is reordering of capabilities. But the limited tests
for version 2 of the SSH protocol do show the capability disappearing.
Differential Revision: https://phab.mercurial-scm.org/D2486
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 15:06:10 -0800] rev 36640
wireproto: don't expose legacy commands to version 2 of wire protocol
Now that we have the ability to control which transports a wire
protocol command is exposed on, let's put it to use.
We flag the "branches," "changegroup," and "changegroupsubset"
commands as only available on version 1. "branches" was used by the
legacy discovery mechanism and was replaced by the "known" and
"heads" commands. "changegroup" and "changegroupsubset" were
replaced by "getbundle."
"between" is also legacy. However, since it is used by the SSH
handshake protocol, marking it as legacy is a bit more complicated
and will be done in a later commit.
Another nuanced issue with this change is that the server-advertised
capabilities still list "changegroupsubset" despite the command not
being available. This will be addressed in a subsequent commit.
Differential Revision: https://phab.mercurial-scm.org/D2485
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 14:56:03 -0800] rev 36639
wireprotoserver: identify requests via version 2 of SSH protocol as such
The protocol handler needs to advertise itself as version 2 in order
for a future feature to work.
Differential Revision: https://phab.mercurial-scm.org/D2484
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 02 Mar 2018 09:47:37 -0500] rev 36638
wireproto: allow wire protocol commands to declare transport support
Currently, wire protocol commands are exposed on all transports.
Some wire protocol commands are only supported or sensical on some
transports. In the future, new wire protocol commands may only be
available on new transports and legacy wire protocol commands may
not be available to newer transports.
This commit introduces a mechanism to allow @wireprotocommand to
declare transports for which they should not be available. The
mechanism for determining if a wire protocol command is available
for a given transport instance has been taught to take this knowledge
into account.
To help implement this feature, we add a dict to wireprototypes
declaring all wire transports and their metadata. There's probably
room to refactor the constants used to identify the wire protocols.
But that can be in another commit.
Differential Revision: https://phab.mercurial-scm.org/D2483
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 02 Mar 2018 18:50:49 -0500] rev 36637
sshpeer: don't read from stderr when that behavior is disabled
We previously prevented the creation of doublepipe instances when
we're not supposed to automatically read from stderr. However,
there were other automatic calls to read from stderr that were
undermining this effort.
This commit prevents all automatic reads from stderr from occurring
when they are supposed to be disabled.
Because stderr is no longer being read, we need to call "readavailable"
from tests so stderr is read from.
Test output changes because stderr is now always (manually) read after
stdout. And, since sshpeer no longer automatically tends to stderr,
no "remote: " messages are printed. This should fix non-deterministic
test output.
FWIW, doublepipe automatically reads from stderr when reading from
stdout, so I'm not sure some of these calls to self._readerr() are
even needed.
Differential Revision: https://phab.mercurial-scm.org/D2571
Boris Feld <boris.feld@octobus.net> [Thu, 15 Feb 2018 17:18:26 +0100] rev 36636
util: extract all date-related utils in utils/dateutil module
With this commit, util.py lose 262 lines
Note for extensions author, if this commit breaks your extension, you can pull
the step-by-step split here to help you more easily pinpoint the renaming that
broke your extension:
hg pull https://bitbucket.org/octobus/mercurial-devel/ -r ac1f6453010d
Differential Revision: https://phab.mercurial-scm.org/D2282
Sangeet Kumar Mishra <mail2sangeetmishra@gmail.com> [Thu, 08 Feb 2018 23:27:24 +0530] rev 36635
clone: updates the help text for hg clone -{r,b} (issue5654)
Differential Revision: https://phab.mercurial-scm.org/D2095
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 15:48:31 -0500] rev 36634
py3: whitelist more passing tests
Differential Revision: https://phab.mercurial-scm.org/D2569
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:07:42 -0500] rev 36633
lfs: convert hexdigest to bytes using sysbytes
Differential Revision: https://phab.mercurial-scm.org/D2568
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:07:25 -0500] rev 36632
lfs: use %d to encode int, not str()
Differential Revision: https://phab.mercurial-scm.org/D2567
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:07:07 -0500] rev 36631
lfs: use byteskwargs() on some **kwargs for python 3 compat
Differential Revision: https://phab.mercurial-scm.org/D2566
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:06:37 -0500] rev 36630
lfs: add some bytestring wrappers in blobstore.py
Differential Revision: https://phab.mercurial-scm.org/D2565
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:05:53 -0500] rev 36629
lfs: add missing b prefixes on some regular expressions
# skip-blame just some b prefixes
Differential Revision: https://phab.mercurial-scm.org/D2564
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 19:34:35 +0900] rev 36628
templatekw: deprecate showdict() and showlist() (API)
.. api::
templatekw.showdict() and showlist() are deprecated in favor of new
(context, mapping) API. Switch the keyword function to new API and use
templatekw.compatdict() and compatlist() instead.
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 19:25:14 +0900] rev 36627
templatekw: switch remainder of _showlist template keywords to new API
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 20:55:53 +0900] rev 36626
templatekw: switch manifest template keyword to new API
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 19:23:06 +0900] rev 36625
templatekw: switch latesttags template keywords to new API
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 19:08:02 +0900] rev 36624
templatekw: switch revset() to new API
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 19:05:57 +0900] rev 36623
templatekw: switch obsfate-related template keywords to new API
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 18:52:51 +0900] rev 36622
templatekw: switch namespace template keywords to new API
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 18:56:06 +0900] rev 36621
namespace: use registrar to add template keyword
Prepares for switching to the new API.
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 16:45:44 +0900] rev 36620
templatekw: switch most of showlist template keywords to new API (issue5779)
Non-trivial changes will follow.
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Feb 2018 16:22:55 +0900] rev 36619
templatekw: switch showdict template keywords to new API
Kevin Bullock <kbullock+mercurial@ringworld.org> [Fri, 02 Mar 2018 15:37:57 -0500] rev 36618
py3: bail on ratcheting tests forward on 3.6.0 and 3.6.1
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 18:22:36 -0500] rev 36617
py3: silence "bad escape" warning emitted by re.sub()
Since we pass user strings directly to re.sub(), we can't avoid this warning
without a BC.
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 14:12:17 -0500] rev 36616
debugcommands: add some strkwargs love to some **args calls
Differential Revision: https://phab.mercurial-scm.org/D2563
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 14:10:34 -0500] rev 36615
debugcommands: add an r prefix to make file mode for fdopen a sysstr
# skip-blame just an r prefix
Differential Revision: https://phab.mercurial-scm.org/D2562
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 14:09:50 -0500] rev 36614
util: work around Python 3 returning None at EOF instead of ''
Differential Revision: https://phab.mercurial-scm.org/D2561
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 14:09:20 -0500] rev 36613
util: add missing r prefix on some __setattr__ calls
# skip-blame just a pair of r prefixes
Differential Revision: https://phab.mercurial-scm.org/D2560
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 13:47:49 -0500] rev 36612
tests: add some re and globs for test-revset on python3
At this point we're down to two deprecation warnings (which I suspect
are showing a bug in the test?) and one weird-looking failure. Progress!
Differential Revision: https://phab.mercurial-scm.org/D2559
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:38:16 -0500] rev 36611
tests: add missing b prefixes and fix a %s to %d in test-revset.t
# skip-blame just b prefixes and a %d instead of %s
Differential Revision: https://phab.mercurial-scm.org/D2558
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:37:41 -0500] rev 36610
revset: use %d to turn an int into a bytestr
Differential Revision: https://phab.mercurial-scm.org/D2557
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 11:37:21 -0500] rev 36609
revset: use {force,}bytestr to fix some %r formatting issues
Differential Revision: https://phab.mercurial-scm.org/D2556
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 01:17:42 -0500] rev 36608
py3: sixteen more passing tests
Differential Revision: https://phab.mercurial-scm.org/D2552
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 00:37:33 -0500] rev 36607
debugcommands: fix repr in debugignore print with pycompat.bytestr
Differential Revision: https://phab.mercurial-scm.org/D2549
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 23:58:28 -0500] rev 36606
verify: fix exception formatting bug in Python 3
Differential Revision: https://phab.mercurial-scm.org/D2541
Yuya Nishihara <yuya@tcha.org> [Fri, 02 Mar 2018 13:50:31 -0500] rev 36605
get-with-headers: use bytes stdout thoroughly
On Python 3, sys.stdout.buffer is backed by a separate buffer from sys.stdout.
We should choose one.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 01 Mar 2018 17:16:36 -0800] rev 36604
exchange: remove dead assignment or forcebundle1
Differential Revision: https://phab.mercurial-scm.org/D2548
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 09:09:38 -0500] rev 36603
templatekw: fix dict construction in _showlist to not mix bytes and strs
What we had was fine on Python 2, but was slightly wrong on Python
3. This works on both.
Differential Revision: https://phab.mercurial-scm.org/D2554
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 09:08:11 -0500] rev 36602
templatefilters: avoid infinite recursion bug in stringify
This doesn't ever happen on Python 2, but it's been a persistent pain
on Python 3. Adding this helped produce some of my upcoming Python 3
fixes.
Differential Revision: https://phab.mercurial-scm.org/D2553
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 00:37:55 -0500] rev 36601
match: some minimal pycompat fixes guided by test-hgignore.t
Differential Revision: https://phab.mercurial-scm.org/D2551
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 00:37:07 -0500] rev 36600
purge: apply byteskwargs to opts, fixing all python3 issues here
Differential Revision: https://phab.mercurial-scm.org/D2547
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 00:19:51 -0500] rev 36599
tests: port test-bookmarks.t extension to Python 3
# skip-blame it's just b prefixes
Differential Revision: https://phab.mercurial-scm.org/D2546
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 00:19:30 -0500] rev 36598
scmutil: fix a repr in an error message on Python 3
Differential Revision: https://phab.mercurial-scm.org/D2545
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 00:19:16 -0500] rev 36597
bookmarks: fix a repr in a message on Python 3
Differential Revision: https://phab.mercurial-scm.org/D2544
Augie Fackler <augie@google.com> [Fri, 02 Mar 2018 00:11:19 -0500] rev 36596
py3: add missing b prefixes in test-debugextensions.t
# skip-blame just some b prefixes
Differential Revision: https://phab.mercurial-scm.org/D2543
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 23:59:44 -0500] rev 36595
tests: help dummysmtpd work on python 3
Differential Revision: https://phab.mercurial-scm.org/D2542
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 23:58:04 -0500] rev 36594
tests: port helper script revlog-formatv0.py to python 3
Differential Revision: https://phab.mercurial-scm.org/D2540
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 21:17:58 -0500] rev 36593
tests: add missing b prefix in test python in test-issue2137.t
# skip-blame just a b prefix
Differential Revision: https://phab.mercurial-scm.org/D2539
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 20:44:38 -0500] rev 36592
templatefilters: convert arguments to sysstrs for unicode() ctor
Differential Revision: https://phab.mercurial-scm.org/D2538
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 02 Mar 2018 07:14:59 +0530] rev 36591
py3: use util.forcebytestr() to convert IOErrors to bytes
Differential Revision: https://phab.mercurial-scm.org/D2536
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 18:20:49 -0500] rev 36590
py3: whitelist three more passing tests
Differential Revision: https://phab.mercurial-scm.org/D2530
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 18:13:50 -0500] rev 36589
convert: fix two %r output formats with pycompat.bytestr() wrapping
Differential Revision: https://phab.mercurial-scm.org/D2529
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 17:48:06 -0500] rev 36588
convert: use our shlex wrapper in filemap to avoid Python 3 tracebacks
Differential Revision: https://phab.mercurial-scm.org/D2527
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 17:47:49 -0500] rev 36587
convcmd: use our shlex wrapper to avoid Python 3 tracebacks
Differential Revision: https://phab.mercurial-scm.org/D2526
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 17:47:35 -0500] rev 36586
convert: add some utility code for working with shlex on Python 3
This could have gone in pycompat, but it's only needed in convert, so
I figured it made more sense here. It's got py3 in the name and checks
pycompat.ispy3, so we'll find it whenever we decide to drop Python 2
support in 20x6.
# no-check-commit because of required foo_bar naming on the proxy class
Differential Revision: https://phab.mercurial-scm.org/D2525
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 17:46:34 -0500] rev 36585
pycompat: add support for encoding argument to our wrapper
This only works on Python 3, but I'm about to need it for a
regrettable hack in the convert code.
Differential Revision: https://phab.mercurial-scm.org/D2524
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 02 Mar 2018 04:49:08 +0530] rev 36584
py3: port tests/test-wireproto.py to Python 3
Differential Revision: https://phab.mercurial-scm.org/D2534
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 02 Mar 2018 04:47:56 +0530] rev 36583
py3: use pycompat.bytestr() to convert None to bytes
Differential Revision: https://phab.mercurial-scm.org/D2533
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 18:39:21 -0500] rev 36582
webcommands: use explicit integer division for Python 3 compat
Differential Revision: https://phab.mercurial-scm.org/D2532
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 02 Mar 2018 04:59:27 +0530] rev 36581
templatefilters: stop using str as a variable name
str() is an inbuilt function.
Differential Revision: https://phab.mercurial-scm.org/D2531
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 18:15:58 -0500] rev 36580
py3: fix string slicing in util.parsetimezone()
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 18:06:25 -0500] rev 36579
py3: replace type 'str' by 'bytes' in templater.py
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 17:56:32 -0500] rev 36578
py3: don't crash when re-raising encoding error
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 17:43:25 -0500] rev 36577
py3: mark all string literals in test-command-template.t as bytes
# skip-blame because just b'' prefixes
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 17:03:40 -0500] rev 36576
py3: byte-stringify ValueError of unescapestr() to reraise as ParseError
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 16:56:38 -0500] rev 36575
py3: fix type of string literals in templater.tokenize()
# skip-blame because just b'' prefixes
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 16:52:17 -0500] rev 36574
py3: drop b'' from error message generated by templater.runmember()
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 16:42:24 -0500] rev 36573
py3: fix join(), min(), and max() template functions over string
It's silly to split a string into characters and concatenate them, but that
should work and test-command-template.t has one. min() and max() had the
same issue on Python 3, so fixed too.
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 16:32:45 -0500] rev 36572
py3: use startswith() to check existence of trailing '\n' in .hgtags file
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Mar 2018 15:57:27 -0500] rev 36571
py3: silence return value of file.write() in test-command-template.t
Augie Fackler <augie@google.com> [Thu, 01 Mar 2018 18:05:47 -0500] rev 36570
error: fix isinstnace check to use bytes instead of str
Differential Revision: https://phab.mercurial-scm.org/D2528
Augie Fackler <augie@google.com> [Sun, 25 Feb 2018 22:43:50 -0500] rev 36569
py3: whitelist another nine passing tests
I now see 347 tests passing[0] and 336 failing, so we're past halfway
there. Hooray!
0: A few tests appear to regress in small ways (doctest output
changes, for example) on Python 3.7.0a1, which is what I'm now
using to test. That said, I'm pleased to report no major regressions.
Differential Revision: https://phab.mercurial-scm.org/D2439
Augie Fackler <augie@google.com> [Sun, 25 Feb 2018 22:31:13 -0500] rev 36568
util: use pycompat.bytestr() on repr() in date parse abort
Avoids the b'' prefix.
Differential Revision: https://phab.mercurial-scm.org/D2438
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 02 Mar 2018 03:06:43 +0530] rev 36567
py3: whitelist 14 new tests passing
12 new tests passed because of parent changeset and 2 were passing before that
too.
Differential Revision: https://phab.mercurial-scm.org/D2523
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 02 Mar 2018 02:44:49 +0530] rev 36566
py3: slice over bytes to prevent getting ascii values
This fixed reading of mergestate files and fixes 14 tests on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D2522
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 14:26:00 -0800] rev 36565
wireprotoserver: move SSHV1 and SSHV2 constants to wireprototypes
To avoid a cycle between modules in an upcoming commit.
Differential Revision: https://phab.mercurial-scm.org/D2482
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 14:21:29 -0800] rev 36564
wireproto: use named arguments for commandentry
We'll be adding more arguments in upcoming commits. Using named
arguments will make the code easier to read.
Differential Revision: https://phab.mercurial-scm.org/D2481
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Feb 2018 18:01:13 -0800] rev 36563
debugcommands: support for triggering push protocol
The mechanism for pushing to a remote is a bit more complicated
than other commands. On SSH, we wait for a positive reply from
the server before we start sending the bundle payload.
This commit adds a mechanism to the "command" action in
`hg debugwireproto` to trigger the "push protocol" and to
specify a file whose contents should be submitted as the command
payload.
With this new feature, we implement a handful of tests for the
"unbundle" command. We try to cover various server failures and
hook/output scenarios so protocol behavior is as comprehensively
tested as possible. Even with so much test output, we only cover
bundle1 with Python hooks. There's still a lot of test coverage
that needs to be implemented. But this is certainly a good start.
Because there are so many new tests, we split these tests into their
own test file.
In order to make output deterministic, we need to disable the
doublepipe primitive. We add an option to `hg debugwireproto`
to do that. Because something in the bowels of the peer does a
read of stderr, we still capture read I/O from stderr. So there
is test coverage of what the server emits.
The tests around I/O capture some wonkiness. For example,
interleaved ui.write() and ui.write_err() calls are emitted in
order. However, (presumably due to buffering), print() to
sys.stdout and sys.stderr aren't in order.
We currently only test bundle1 because bundle2 is substantially
harder to test because it is more complicated (the server responds
with a stream containing a bundle2 instead of a frame).
Differential Revision: https://phab.mercurial-scm.org/D2471
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Feb 2018 13:12:03 -0800] rev 36562
sshpeer: support not reading and forwarding stderr
The "doublepipe" primitive as used by sshpeer will automatically read
from stderr and forward output to the local ui.
This poses problems for deterministic testing because reads may not
be consistent. For example, the server may not be done sending all
output to stderr and the client will perform different numbers of
read operations or will read from stderr and stdout at different times.
To make tests deterministic, we'll need to disable the "doublepipe"
primitive and perform stderr I/O explicitly. We add an argument to the
sshpeer constructor to disable the use of the doublepipe.
Differential Revision: https://phab.mercurial-scm.org/D2467
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 23 Feb 2018 16:03:27 -0800] rev 36561
tests: add wire protocol tests for pushkey
Let's get the wire format of some pushkey requests in test-ssh-proto.t.
Differential Revision: https://phab.mercurial-scm.org/D2466
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 23 Feb 2018 12:50:59 -0800] rev 36560
debugcommands: support for sending "batch" requests
Let's teach `hg debugwireproto` to send "batch" requests.
The easiest way to implement this was as a pair of instructions to
begin and end a batched operation. Otherwise, we would have to reinvent
the parsing wheel or factor out the parsing code.
To prove it works, we add a batched request to test-ssh-proto.t.
Differential Revision: https://phab.mercurial-scm.org/D2408
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 01 Mar 2018 08:27:30 -0800] rev 36559
debugcommands: allow sending of simple commands with debugwireproto
Previously, we only had support for low-level "raw" operations.
A goal of `hg debugwireproto` is to allow easily performing
higher-level primitives, such as sending a wire protocol command
and reading its response.
We implement a "command" action that does just this.
Currently, we only support simple commands (those without payloads).
We have basic support for sending command arguments. We don't yet
support sending dictionary arguments. This will be implemented later.
To prove it works, we add tests to test-ssh-proto.t that send some
"listkeys" commands.
Note: we don't observe/report os.read() events because these may not be
deterministic. We instead observe/report the read() and readline()
operations on the bufferedinputpipe. These *should* be deterministic.
Differential Revision: https://phab.mercurial-scm.org/D2406
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 23 Feb 2018 09:40:12 -0800] rev 36558
wireproto: sort response to listkeys
The listkeys protocol is defined to produce a dictionary.
pushkey.decodekeys() uses a plain dict to hold the decoded results
of the wire protocol response. So order should not matter.
Upcoming tests will verify low-level output of wire protocol
commands and the non-deterministic emitting of listkeys was causing
intermittent failures.
So we make the output of listkeys deterministic.
Differential Revision: https://phab.mercurial-scm.org/D2405
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 01 Mar 2018 08:24:54 -0800] rev 36557
debugcommands: add debugwireproto command
We currently don't have a low-level mechanism for sending
arbitrary wire protocol commands. Having a generic and robust
mechanism for sending wire protocol commands, examining wire
data, etc would make it vastly easier to test the wire protocol
and debug server operation. This is a problem I've wanted a
solution for numerous times, especially recently as I've been
hacking on a new version of the wire protocol.
This commit establishes a `hg debugwireproto` command for sending
data to a peer.
The command invents a mini language for specifying actions to take.
This will enable a lot of flexibility for issuing commands and testing
variations for how commands are sent.
Right now, we only support low-level raw sends and receives. These
are probably the least valuable commands to intended users of this
command. But they are the most useful commands to implement to
bootstrap the feature (I've chosen to reimplement test-ssh-proto.t
using this command to prove its usefulness).
My eventual goal of `hg debugwireproto` is to allow calling wire
protocol commands with a human-friendly interface. Essentially,
people can type in a command name and arguments and
`hg debugwireproto` will figure out how to send that on the wire.
I'd love to eventually be able to save the server's raw response
to a file. This would allow us to e.g. call "getbundle" wire
protocol commands easily.
test-ssh-proto.t has been updated to use the new command in lieu
of piping directly to a server process. As part of the transition,
test behavior improved. Before, we piped all request data to the
server at once. Now, we have explicit control over the ordering of
operations. e.g. we can send one command, receive its response,
then send another command. This will allow us to more robustly
test race conditions, buffering behavior, etc.
There were some subtle changes in test behavior. For example,
previous behavior would often send trailing newlines to the server.
The new mechanism doesn't treat literal newlines specially and
requires newlines be escaped in the payload.
Because the new logging code is very low level, it is easy to
introduce race conditions in tests. For example, the number of bytes
returned by a read() may vary depending on load. This is why tests
make heavy use of "readline" for consuming data: the result of
that operation should be deterministic and not subject to race
conditions. There are still some uses of "readavailable." However,
those are only for reading from stderr. I was able to reproduce
timing issues with my system under load when using "readavailable"
globally. But if I "readline" to grab stdout, "readavailable"
appears to work deterministically for stderr. I think this is
because the server writes to stderr first. As long as the OS
delivers writes to pipes in the same order they were made, this
should work. If there are timing issues, we can introduce a
mechanism to readline from stderr.
Differential Revision: https://phab.mercurial-scm.org/D2392
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 27 Feb 2018 15:47:44 -0800] rev 36556
debugcommands: add debugserve command
`hg serve --stdio` requires the exact command argument form
`hg -R <path> serve --stdio` for security reasons. An upcoming
commit will need to start an SSH protocol server process with
custom settings.
This commit creates a `hg debugserve` command for starting servers
with custom options. There are no security restrictions and we can
add options here that aren't appropriate for built-in commands.
We currently only support starting an SSH protocol server using
the process's stdio file descriptors. The server supports logging
its I/O activity to a file descriptor number passed as a command
argument.
Differential Revision: https://phab.mercurial-scm.org/D2464
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 25 Feb 2018 11:16:09 -0800] rev 36555
wireprotoserver: support logging SSH server I/O to a file descriptor
We will soon introduce a debug command and tests for low-level I/O
behavior of the SSH wire protocol.
To facilitate this, we need to instrument the SSH server so it
can log its I/O as events occur.
We teach the SSH server to convert its stdout and stderr file objects
into file object proxies. We configure these proxies to log to a
file descriptor whose file number is specified via a config option.
The idea is to have a future debug command start the SSH server
process with access to an extra file descriptor that can be used
by the server process to log I/O. Monitoring only the write I/O
will be more robust than monitoring both writes and reads from the
client process because read operations are not deterministic. This
will matter for tests that capture raw I/O activity.
Differential Revision: https://phab.mercurial-scm.org/D2463
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Feb 2018 12:24:03 -0800] rev 36554
util: enable observing of util.bufferedinputpipe
Our file object proxy is useful. But it doesn't capture all I/O.
The "os" module offers low-level interfaces to various system calls.
For example, os.read() exposes read(2) to read from a file
descriptor.
bufferedinputpipe is special in a few ways. First, it acts as a
proxy of sorts around our [potentially proxied] file object. In
addition, it uses os.read() to satisfy all I/O. This means that
our observer doesn't see notifications for reads on this type.
This is preventing us from properly instrumenting reads on ssh
peers.
This commit teaches bufferedinputpipe to be aware of our
observed file objects. We do this by introducing a class variation
that notifies our observer of os.read() events. Since read()
and readline() bypass os.read(), we also teach this instance
to notify the observer for buffered variations of these reads as
well. We don't report them as actual read() and readline() calls
because these methods are never called on the actual file object
but rather a buffered version of it.
We introduce bufferedinputpipe.__new__ to swap in the new class
if the passed file object is a fileobjectproxy. This makes hooking
up the observer automatic. And it is a zero cost abstraction for
I/O operations on non-proxied file objects.
Differential Revision: https://phab.mercurial-scm.org/D2404
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Feb 2018 12:22:20 -0800] rev 36553
util: add a file object proxy that can notify observers
There are various places in Mercurial where we may want to
instrument low-level I/O. The use cases I can think of all
involve development-type activities like monitoring the raw
bytes passing through a file (for testing and debugging),
counting the number of I/O function calls (for performance
monitoring), and changing the behavior of I/O function calls
(e.g. simulating a failure) (to facilitate testing).
This commit invents a mechanism to wrap a file object so we
can observe activity on it. We have similar functionality in
badserverext.py. But that's a test-only extension and is pretty
specific to the HTTP server. I would like a mechanism in core
that is sufficiently generic so it can be used by multiple
consumers, including `hg debug*` commands.
The added code consists of a proxy type for file objects.
It is bound to an "observer," which receives callbacks whenever
I/O methods are called.
We also add an implementation of an observer that logs specific
I/O events. This observer will be used in an upcoming commit
to record low-level wire protocol activity.
A helper function to convert a file object into an observed
file object has also been implemented.
I don't anticipate any critical functionality in core using
these types. So I don't think explicit test coverage is
worth implementing.
Differential Revision: https://phab.mercurial-scm.org/D2462