Jun Wu <quark@fb.com> [Mon, 18 Jul 2016 18:55:06 +0100] rev 29602
chg: handle EOF reading data block
We recently discovered a case in production that chg uses 100% CPU and is
trying to read data forever:
recvfrom(4, "", 1814012019, 0, NULL, NULL) = 0
Using gdb, apparently readchannel() got wrong data. It was reading in an
infinite loop because rsize == 0 does not exit the loop, while the server
process had ended.
(gdb) bt
#0 ... in recv () at /lib64/libc.so.6
#1 ... in readchannel (...) at /usr/include/bits/socket2.h:45
#2 ... in readchannel (hgc=...) at hgclient.c:129
#3 ... in handleresponse (hgc=...) at hgclient.c:255
#4 ... in hgc_runcommand (hgc=..., args=<optimized>, argsize=<optimized>)
#5 ... in main (argc=...486922636, argv=..., envp=...) at chg.c:661
(gdb) frame 2
(gdb) p *hgc
$1 = {sockfd = 4, pid = 381152, ctx = {ch = 108 'l',
data = 0x7fb05164f010 "st):\nTraceback (most recent call last):\n"
"Traceback (most recent call last):\ne", maxdatasize = 1814065152,"
" datasize = 1814064225}, capflags = 16131}
This patch addresses the infinite loop issue by detecting continuously empty
responses and abort in that case.
Note that datasize can be translated to ['l', ' ', 'l', 'a']. Concatenate
datasize and data, it forms part of "Traceback (most recent call last):".
This may indicate a server-side channeledoutput issue. If it is a race
condition, we may want to use flock to protect the channels.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 18 Jul 2016 11:27:27 -0700] rev 29601
sslutil: more robustly detect protocol support
The Python ssl module conditionally sets the TLS 1.1 and TLS 1.2
constants depending on whether HAVE_TLSv1_2 is defined. Yes, these
are both tied to the same constant (I would think there would be
separate constants for each version). Perhaps support for TLS 1.1
and 1.2 were added at the same time and the assumption is that
OpenSSL either has neither or both. I don't know.
As part of developing this patch, it was discovered that Apple's
/usr/bin/python2.7 does not support TLS 1.1 and 1.2 (only TLS 1.0)!
On OS X 10.11, Apple Python has the modern ssl module including
SSLContext, but it doesn't appear to negotiate TLS 1.1+ nor does
it expose the constants related to TLS 1.1+. Since this code is
doing more robust feature detection (and not assuming modern ssl
implies TLS 1.1+ support), we now get TLS 1.0 warnings when running
on Apple Python. Hence the test changes.
I'm not super thrilled about shipping a Mercurial that always
whines about TLS 1.0 on OS X. We may want a follow-up patch to
suppress this warning.
Maciej Fijalkowski <fijall@gmail.com> [Mon, 11 Jul 2016 11:05:08 +0200] rev 29600
osutil: add darwin-only version of os.listdir using cffi
Yuya Nishihara <yuya@tcha.org> [Sun, 05 Jun 2016 12:29:08 +0900] rev 29599
url: drop support for proxying HTTP (not HTTPS) over CONNECT tunneling
It's been broken since cca59ef27e60, which made ui argument mandatory. I've
tried several combinations of HTTP/HTTPS proxying on old/new Python versions,
but I couldn't figure out how to reach this code path. Also, wrapping HTTP
connection by SSLSocket seems wrong. My understanding is that self.realhostport
is set by _generic_start_transaction() if HTTPS connection is tunneled.
This patch removes proxy tunneling from httpconnection.connect() assuming
that it was dead code from the beginning. Note that HTTPS over tunneling
should be handled by httpsconnection class.
Yuya Nishihara <yuya@tcha.org> [Sat, 21 May 2016 18:16:39 +0900] rev 29598
chgserver: rename private functions and variables of chgunixservicehandler
self.address has been reanmed to self._realaddress to clarify that it can
be different from the address argument.
Yuya Nishihara <yuya@tcha.org> [Sun, 22 May 2016 14:06:37 +0900] rev 29597
chgserver: refactor initialization of real/base addresses
Instead of overwriting self.address, calculate it from the address argument,
which is the base address.
Yuya Nishihara <yuya@tcha.org> [Sun, 22 May 2016 14:05:34 +0900] rev 29596
chgserver: reorder functions in chgunixservicehandler
This should make it slightly easier to follow the call path.
Yuya Nishihara <yuya@tcha.org> [Sat, 21 May 2016 18:15:20 +0900] rev 29595
chgserver: use ui.debug() to print server debug messages
commandserver.log() is noop at this time because no client connection is
established.
Yuya Nishihara <yuya@tcha.org> [Sun, 05 Jun 2016 12:18:20 +0900] rev 29594
ssl: remove special case of web.cacerts=! from remoteui()
It was introduced by b76d8c641746, which is no longer necessary thanks to
recent refactoring of sslutil including ef316c653b7f.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 15:13:51 -0700] rev 29593
bundle2: store changeset count when creating file bundles
The bundle2 changegroup part has an advisory param saying how many
changesets are in the part. Before this patch, we were setting
this part when generating bundle2 parts via the wire protocol but
not when generating local bundle2 files.
A side effect of not setting the changeset count part is that progress
bars don't work when applying changesets. As the tests show, this
impacted clone bundles, shelve, backup bundles, `hg unbundle`, and
anything touching bundle2 files.
This patch adds a backdoor to allow us to pass state from
changegroup generation into the unbundler. We store the number
of changesets in the changegroup in this state and use it to
populate the aforementioned advisory part parameter when generating
the bundle2 bundle.
I concede that I'm not thrilled by how state is being passed in
changegroup.py (it feels a bit hacky). I would love to overhaul the
rather confusing set of functions in changegroup.py with something that
passes rich objects around instead of e.g. low-level generators.
However, given the code freeze for 3.9 is imminent, I'd rather not
undertake this endeavor right now. This feels like the easiest way
to get the parameter added to the changegroup part.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 15:10:30 -0700] rev 29592
util: implement a deterministic __repr__ on sortdict
`hg debugbundle` is calling repr() on bundle2 part params, which are
now util.sortdict instances. Unfortunately, repr() doesn't appear
to be deterministic for util.sortdict. So, we implement one.
We include the type name because that's the common convention for
__repr__ implementations. Having the type name in `hg debugbundle`
is a bit ugly. But it's a debug command and I don't care enough to
fix it.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 14:51:00 -0700] rev 29591
bundle2: use a sorted dict for holding parameters
An upcoming change that introduces a 2nd part parameter to a part
reveals that `hg debugbundle` isn't deterministic because parameters
are stored on n plain, unsorted dict.
While we could change that command to sort before output, I think
the more important underlying issue is that bundle2 reading is taking
an ordered data structure and converting it to an unordered one.
Plugging in util.sortdict() fixes that problem while preserving API
compatibility.
This patch also appears to shine light on the fact that we don't
have tests verifying parts with multiple parameters roundtrip
correctly. That would be a good thing to test (and fuzz)... someday.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 15 Jul 2016 13:41:34 -0700] rev 29590
wireproto: extract repo filtering to standalone function
As part of teaching Mozilla's replication extension to better handle
repositories with obsolescence data, I encountered a few scenarios
where I wanted built-in wire protocol commands from replication clients
to operate on unfiltered repositories so they could have access to
obsolete changesets.
While the undocumented "web.view" config option provides a mechanism
to choose what filter/view hgweb operates on, this doesn't apply
to wire protocol commands because wireproto.dispatch() is always
operating on the "served" repo.
This patch extracts the line for obtaining the repo that
wireproto commands operate on to its own function so extensions
can monkeypatch it to e.g. return an unfiltered repo.
I stopped short of exposing a config option because I view the use
case for changing this as a niche feature, best left to the domain
of extensions.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 14 Jul 2016 19:16:46 -0700] rev 29589
url: add distribution and version to user-agent request header (BC)
As a server operator, I've always wanted to know what Mercurial
version clients are running so I can track version adoption and
make informed decisions about which versions of Mercurial to
support in extensions. Unfortunately, there is no easy way to discern
this today: the best you can do is look for high-level feature usage
(e.g. bundle2) or sniff capabilities from bundle2 commands. And these
things aren't changed frequently enough to tell you anything that
interesting.
Nearly every piece of software talking HTTP sends its version in
the user agent. This includes web browsers, curl, and even Git.
This patch adds the distribution name and version to the user-agent
HTTP request header. We choose "Mercurial" for the distribution
name because that seems appropriate. The version string comes
from __version__.
The value is inside parenthesis for a few reasons:
* The version *may* contain spaces
* Alternate forms like "Mercurial/<version>" imply structure and
since the user agent should not be used by servers for protocol
or feature negotiation/detection, we don't want to even give the
illusion that the value should be parsed. A free form field is
the most hostile to parsing.
Flagging the patch as BC so it shows up in release notes. This
change should be backwards compatible. But I wouldn't be surprised if
a server somewhere is filtering on the exact old user agent string. So
I want to make noise about this change.