Gregory Szorc <gregory.szorc@gmail.com> [Wed, 30 Jan 2019 12:55:44 -0800] rev 41481
tests: conditionalize test-http-bad-server.t for Python 3.5
It appears that Python 3 introduced output buffering in the HTTP
response stack. And Python 3.6 switched from sock.makefile().write()
to sock.sendall().
So, we need to conditionalize test-http-bad-server.t to account
for the difference in behavior between Python 3.5 and 3.6.
Differential Revision: https://phab.mercurial-scm.org/D5754
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 30 Jan 2019 12:12:25 -0800] rev 41480
tests: log sendall() operations and port test-http-bad-server.t
Python 3's HTTP server layer buffers output and uses sendall()
instead of write(). In order to make test-http-bad-server.t pass
on Python 3, we needed to teach our socket proxy to log sendall()
events and to abort future sends if we reached our send limit.
The tests using `tail` were difficult to port with inline output
conditionals since the number of lines varied. So we now use
`#if py3` for these tests.
test-http-bad-server.t now passes on Python 3.6 and 3.7 on at
least Linux. However, it does not yet pass on Python 3.5 because
of low-level differences to how the HTTP server is implemented.
Differential Revision: https://phab.mercurial-scm.org/D5753
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 29 Jan 2019 14:06:46 -0800] rev 41479
tests: glob away readline(-1)
Most of these are readline(65537) on Python 3. I don't think it is
worth the readability hit to use (re), as it would require escaping
parenthesis.
Differential Revision: https://phab.mercurial-scm.org/D5752
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 30 Jan 2019 13:08:59 -0800] rev 41478
tests: change how sockets are closed
Python 3 uses a different type to represent a socket file object
than Python 2. We need to conditionalize how the socket is closed
accordingly.
While we're here, we switch to use socket.shutdown() to close the
socket. This is because socket.close() may not actually close
the socket until it is GCd. socket.shutdown() forces an immediate
shutdown.
I suspect Python 3 changed semantic behavior here, as I can't get
test-http-bad-server.t to work with socket.close(). socket.shutdown()
does appear to work, however.
Differential Revision: https://phab.mercurial-scm.org/D5751
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 30 Jan 2019 09:52:16 -0800] rev 41477
tests: add b'' prefixes to badserverext.py
This avoids a handful of failures due to missing str and bytes.
# skip-blame: just a bunch of b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5750
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 30 Jan 2019 11:44:34 -0800] rev 41476
hgweb: log error before attempting I/O
Previously, an uncaught exception during HTTP request serving would
attempt to send an error response then log the exception.
If an exception occurred during I/O, this exception would be
raised and the original exception wouldn't be logged.
This commit changes behavior so the original exception is logged
first, before we attempt to do anything else. This ensures the
exception is logged.
This change resulted in new tracebacks appearing in various tests.
Because tracebacks can vary between Python versions, we added a
simple script to filter the stack part of traceback lines. This
makes testing much simpler, as we don't need to glob over lines
and make lines conditional.
Differential Revision: https://phab.mercurial-scm.org/D5749
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 29 Jan 2019 11:51:19 -0800] rev 41475
tests: write commit message using file I/O
Python 2.7 will print() \x94\x5c\x0a whereas Python 3 will
print() \xc2\x94\x5c\x0a. Why, I'm not sure. It probably has to
do with print() being Unicode aware on Python 3 and Python
attempting some kind of encoding before emitting the output.
This difference results in a different bytes making it to the
commit message and the JSON output varying. We work around
this by writing bytes to a commit message file.
Differential Revision: https://phab.mercurial-scm.org/D5741
Kyle Lippincott <spectral@google.com> [Mon, 28 Jan 2019 18:00:14 -0800] rev 41474
patch: handle 0 context lines (diff.unified=0) when parsing patches
Previously, if there were no context lines, we would just keep updating the
ranges and the hunk, but not actually storing the hunk (just overwriting it each
time). Thus a diff like this:
$ hg diff --config diff.unified=0
diff --git a/bar b/bar
--- a/bar
+++ b/bar
@@ -1,0 +2,1 @@ 1
+change1
@@ -3,0 +5,1 @@ 3
+change2
would come out of the parser like this (change1 is lost):
bar:
@@ -3,0 +5,1 @@ 3
+change2
This had some really weird side effects for things like commit --interactive,
split, etc.
Differential Revision: https://phab.mercurial-scm.org/D5743
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 30 Jan 2019 03:56:35 +0530] rev 41473
py3: pass str into RuntimeError() to prevent b'' in output
# skip-blame as just r'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5740
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 30 Jan 2019 03:49:56 +0530] rev 41472
py3: use '%d' instead of '%s' for integers
Differential Revision: https://phab.mercurial-scm.org/D5738
Augie Fackler <augie@google.com> [Tue, 29 Jan 2019 13:26:18 -0500] rev 41471
tests: port test-hgweb-auth.py to Python 3
Differential Revision: https://phab.mercurial-scm.org/D5736
Augie Fackler <augie@google.com> [Tue, 29 Jan 2019 13:25:21 -0500] rev 41470
url: convert some variables back to bytes
Differential Revision: https://phab.mercurial-scm.org/D5735
Augie Fackler <augie@google.com> [Tue, 29 Jan 2019 13:24:20 -0500] rev 41469
url: add some defensive asserts on expected incoming types
Our type handling is a nightmare here, and we're loading passwords to
do network IO, so we can afford to be potentially-slow but pedantic
here.
Differential Revision: https://phab.mercurial-scm.org/D5734
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 29 Jan 2019 16:03:52 +0300] rev 41468
changegroup: don't try to build changelog chunks if not required
When we extend a narrow clone without ellipsis, we don't download changelog
information because that's already present with the client. However we still try
to build that chunk stream. Building that chunk stream involves calling a lookup
function and store.emitrevisions() API. The lookup function is called len(cl)
number of times.
On large repositories, where len(cl) is in millions, calling that lookup
function is not a good idea. Also it's not required to use the
store.emitrevisons() API because we already have nodes present which we can use.
This patch short-circuits state building logic if we are processing a
non-ellipsis case and changelog is not required.
This saves up ~20 seconds on our internal repo for a single extend call.
Differential Revision: https://phab.mercurial-scm.org/D5733
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 29 Jan 2019 15:43:02 +0300] rev 41467
changegroup: initialize the state variable a bit earlier
This will make the next patch much easier.
Differential Revision: https://phab.mercurial-scm.org/D5732
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 11:23:31 -0800] rev 41466
tests: conditionalize test output on Python 3.7
Python 3.7 changed behavior of urllib.parse.quote() from RFC 2396
to RFC 3986 and ~ is now in the set of reserved characters and
isn't escaped.
We conditioanlize test output accordingly.
Differential Revision: https://phab.mercurial-scm.org/D5717
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 13 Oct 2018 16:53:43 +0200] rev 41465
hghave: add pyXY features for Python version numbers
This will allow us to sniff for Python >= versions in tests.
Differential Revision: https://phab.mercurial-scm.org/D5088
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 29 Jan 2019 14:30:10 +0300] rev 41464
py3: whitelist couple more passing tests found by buildbot
Differential Revision: https://phab.mercurial-scm.org/D5731
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 13:52:39 -0800] rev 41463
keepalive: implement _close_conn() so closes are known
Keepalives were not working on Python 3 because
http.client.HTTPResponse was refactored to call _close_conn()
instead of close(). Our custom close() is what returns inactive
connections to the available state.
We better support Python 3 by implementing a _close_conn().
Differential Revision: https://phab.mercurial-scm.org/D5720
Matt Harbison <matt_harbison@yahoo.com> [Mon, 28 Jan 2019 21:35:06 -0500] rev 41462
lfs: explicitly add the Content-Length header when uploading blobs, for py3
This was the reason for test-lfs-test-server.t#git-server complaining about an
"invalid byte in chunk length". For some reason if this isn't explicitly added,
py3.7.1 is adding `transfer-encoding: chunked` as well as `Content-length: x`.
Wireshark flagged this as malformed. However, if this is set, it doesn't bother
with `transfer-encoding`.
Before this patch with py3:
PUT /objects/31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b HTTP/1.1
Accept-Encoding: identity
Content-length: 12
accept: application/vnd.git-lfs
content-type: application/octet-stream
host: localhost:20062
transfer-encoding: chunked
user-agent: git-lfs/2.3.4 (Mercurial 4.9rc0+149-7eb7637e34bf)
Before this patch with py27:
PUT /objects/31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b HTTP/1.1
Accept-Encoding: identity
accept: application/vnd.git-lfs
content-type: application/octet-stream
content-length: 12
host: localhost:20062
user-agent: git-lfs/2.3.4 (Mercurial 4.9rc0+149-7eb7637e34bf+20190128)
With this patch and py3, the content is the same as the py27 example. RFC2616
says to ignore `Content-Length` if `Transfer-Encoding` is present, so maybe
there's nothing to do in the hg-server side (though I'm not sure which it is
using if presented both).
Maybe chunked encoding is better to do? If someone knows how to suppress the
`Content-Length`, we can try that instead.
Boris Feld <boris.feld@octobus.net> [Mon, 28 Jan 2019 03:20:31 -0500] rev 41461
perf: support looking up multiple revisions
The nodemap code has optimisations around the number of lookup we actually made.
As a result, being able to specify multiple revisions to look up is important
when measuring performances. One can now specify full revspecs with the --rev
arguments.
Boris Feld <boris.feld@octobus.net> [Fri, 25 Jan 2019 18:43:48 -0500] rev 41460
perf: add a no-lookup variant to perfindex
It is useful to check how long it takes to create a index object without doing
anything with it. We add a new flag dedicated to that.
Boris Feld <boris.feld@octobus.net> [Mon, 28 Jan 2019 04:47:40 -0500] rev 41459
perf: add some documentation to perfindex
It seems useful to document how the arguments can affect the benchmark.
Boris Feld <boris.feld@octobus.net> [Fri, 25 Jan 2019 14:53:19 -0500] rev 41458
perf: move cache clearing in the `setup` step of `perfheads`
The cache clearing is pretty fast, but this seems more "correct".
Boris Feld <boris.feld@octobus.net> [Fri, 25 Jan 2019 18:22:02 -0500] rev 41457
revlog: document cext oddities in terms of object/caches
This clarify why we just call clearcaches on a single object.
Boris Feld <boris.feld@octobus.net> [Fri, 25 Jan 2019 14:52:31 -0500] rev 41456
perf: document perfheads
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 22:32:09 -0500] rev 41455
py3: stabilize the output of lfs commandserver tests
The print() statements were being output at the very end, so write to the same
stdout sink as runcommand, and explicitly flush.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 20:58:18 -0500] rev 41454
py3: conditionalize some LFS test output
I'm not sure why the one stackframe is py2 only, but that seems harmless. The
remaining failure is LfsCorruptionError printing the fully qualified name, as
well as b'' around its message.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 20:50:52 -0500] rev 41453
lfs: strip the response headers from the Batch API before printing
For reasons unknown, py3 is adding an extra '\n' before the headers print out.
This makes the output the same as py2.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 18:34:17 -0500] rev 41452
py3: force hgweb.server error log to internally write unicode
Otherwise, there's a lot of py2/py3 divergence in the LFS tests because of the
"HG error" lines picking up a b'' prefix. wsgicgi.py uses procutil.stderr, so I
assume the input was meant to be bytes.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 17:48:15 -0500] rev 41451
py3: byteify the decoded JSON responses upon receipt in the LFS blobstore
It got too confusing juggling r'' vs b'' across several functions.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 18:05:17 -0500] rev 41450
hgweb: ensure Content-Length and Content-Type are not promoted to HTTP_ on py3
In stabilizing test-lfs-serve-access.t for py3, the server started asserting on
blob upload:
Environment should not have the key: HTTP_CONTENT_LENGTH (use CONTENT_LENGTH instead)
It could be avoided by explicitly setting the Content-Length header on the
client side. I didn't go back to py2, but printing the original header here in
py37 revealed 'Content-length' when sent to the error log.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 15:42:55 -0500] rev 41449
py3: raw stringify various JSON and HTTP headers in the LFS blobstore module
This is (almost?) entirely from Augie's work. I'm a bit surprised that the JSON
data is being encoded with ASCII via `pycompat.bytesurl()`- I would have thought
UTF-8.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 15:19:28 -0500] rev 41448
py3: byteify the LFS blobstore module
This is almost entirely b'' prefixing, with a couple of exceptions forced to
bytes. Much of this is also borrowed from Augie's code. There's an
HTTPError.read() that I flagged that I assume needs to be converted to bytes,
but I can't find confirmation.
Handling the deserialized JSON object over several functions made r'' vs b''
accesses confusing, so this assumes that the JSON object will be converted to
bytes immediately. That will be done in the following commits, so it's not
buried in these trivial changes.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 00:50:39 -0500] rev 41447
py3: raw stringify various things in the LFS server module
Some of this is based on code written by Augie. I'm slightly unsure if these
are the correct pycompat bytes <-> str conversion methods.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 00:36:56 -0500] rev 41446
py3: add b'' prefixes to the LFS server module
There are a ton of these changes in the blobstore, as well as r'' prefixing. So
separating these out hopefully makes review easier.
# skip-blame for b'' prefixing
Matt Harbison <matt_harbison@yahoo.com> [Sun, 27 Jan 2019 00:26:17 -0500] rev 41445
py3: add b'' and r'' prefixes to LFS tests
Augie Fackler <augie@google.com> [Sun, 27 Jan 2019 20:22:07 -0500] rev 41444
py3: more tests from the ratchet
Differential Revision: https://phab.mercurial-scm.org/D5730
Yuya Nishihara <yuya@tcha.org> [Sat, 26 Jan 2019 17:53:03 +0900] rev 41443
dispatch: unify handler of IOError and OSError
These exceptions were merged in Python 3.
Yuya Nishihara <yuya@tcha.org> [Sat, 26 Jan 2019 17:51:55 +0900] rev 41442
dispatch: quote filename in IOError as well
It's explicitly added for OSError at 720308f741cb "dispatch: show empty
filename in OSError aborts". Let's do the same for IOError.
Yuya Nishihara <yuya@tcha.org> [Sat, 26 Jan 2019 17:44:07 +0900] rev 41441
dispatch: add inline comment about possible IOError subtypes
It's hard to tell which "if" would handle which exception.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 16:53:17 -0800] rev 41440
convert: print exception message directly
Otherwise the default repr() implementation will b'' prefix the
returned str on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5729
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 16:45:25 -0800] rev 41439
crecord: always return a str from uihunk.__repr__
Otherwise Python 3 complains about it returning bytes.
Differential Revision: https://phab.mercurial-scm.org/D5728
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 15:52:37 -0800] rev 41438
tests: handles bytes/str mismatch in test-commit-interface-curses.t
This fixes various test failures on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5727
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 14:16:34 -0800] rev 41437
githelp: format with %d if an integer
Python 3 doesn't allow us to format an int with %s like Python 2
did. So handle that.
Differential Revision: https://phab.mercurial-scm.org/D5726
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 14:14:44 -0800] rev 41436
githelp: make argument parsing more compatible with Python 3
There were various mixing of str and bytes in here. This change
fixes most of the failures in test-githelp.t.
Differential Revision: https://phab.mercurial-scm.org/D5725
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 14:08:35 -0800] rev 41435
tests: add b'' prefixes to flagprocessorext.py
Otherwise a part of test-flavprocessor.t fails due to not
setting the proper key.
Differential Revision: https://phab.mercurial-scm.org/D5724
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 14:06:07 -0800] rev 41434
tests: add b'' to config file name
Without this, things die in the bowels of the config system
due to mixing str and bytes.
# skip-blame: just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D5723
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 14:00:42 -0800] rev 41433
sslutil: ensure serverhostname is bytes when formatting
It will likely be a str on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5722
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 13:58:58 -0800] rev 41432
sslutil: use raw strings for exception reason compare
Otherwise we attempt to compare a bytes to a str on Python 3
and it always fails.
Differential Revision: https://phab.mercurial-scm.org/D5721
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 13:40:44 -0800] rev 41431
keepalive: track ready state with a bool
This code may have been written before Python had a bool type.
Differential Revision: https://phab.mercurial-scm.org/D5719
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 13:39:18 -0800] rev 41430
keepalive: use collections.defaultdict for host map
Cleaning up the code as part of debugging Python 3 issues.
Differential Revision: https://phab.mercurial-scm.org/D5718
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 10:57:17 -0800] rev 41429
statichttprepo: use str to appease Python 3
The URL fed into urllib and HTTP headers need to be str on
Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5716
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 10:53:10 -0800] rev 41428
statichttprepo: use URLError.reason directly
0b3f4be5c5bf changed str(inst) to inst.reason[0] all the way back
in 2006. URLError.reason is a str and we should have taken that
attribute in its entirety. I think the code was supposed to be
inst.args[1] for compatibility with ancient Python versions.
Python 2.7 always sets .reason, so it should be safe to use
directly.
Differential Revision: https://phab.mercurial-scm.org/D5715
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 10:40:37 -0800] rev 41427
mail: document behavior of Python 3
test-notify.t (and possibly other tests) are failing on Python 3
because email.message.Message is now aware of encodings and
attempts to roundtrip values with the specified message encoding.
Python 2 doesn't perform this roundtripping. We have tests with
non-ascii data being serialized to a message that claims to use
ascii encoding.
I /think/ Mercurial's behavior may be buggy here. But I'm not
sure.
I'm documenting the behavior so the next person who looks into
this doesn't start from scratch like I did.
Differential Revision: https://phab.mercurial-scm.org/D5714
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 10:22:09 -0800] rev 41426
notify: be more defensive aboute None values
encoding.strtolocal is the identity function on Python 2
but an actual string manipulation routine on Python 3.
In some cases, we were passing None, which caused Python 3
to barf.
Let's change the code to react properly when the value is
None.
Differential Revision: https://phab.mercurial-scm.org/D5713
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 26 Jan 2019 10:00:17 -0800] rev 41425
wireprotov2server: use our JSON encoder
Python's json module doesn't like to encode bytes instances.
This makes this code difficult to work with Python 3.
We simply swap in Mercurial's JSON encoder to work around it.
Differential Revision: https://phab.mercurial-scm.org/D5712
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 25 Jan 2019 17:11:49 -0800] rev 41424
tests: add optional setsockopt() lines for Python 3
Differential Revision: https://phab.mercurial-scm.org/D5711
Navaneeth Suresh <navaneeths1998@gmail.com> [Sat, 26 Jan 2019 13:14:21 +0530] rev 41423
diffstat: support filenames with whitespaces on renames
This is a follow-up patch to D5628. `line.split()` cannot get filenames with
whitespaces as mentioned by @yuja. This patch replaces `split()` method with
`slice`. Corresponding tests were also added.
Differential Revision: https://phab.mercurial-scm.org/D5709
Martin von Zweigbergk <martinvonz@google.com> [Mon, 21 Jan 2019 23:08:31 -0800] rev 41422
largefiles: avoid walking full manifest
When using treemanifest, diffs between two manifests can often be much
cheaper than iterating the full manifests (because common subtrees are
skipped).
Differential Revision: https://phab.mercurial-scm.org/D5644