Matt Mackall <mpm@selenic.com> [Mon, 18 Jul 2016 23:28:14 -0500] rev 29605
merge default into stable for 3.9 code freeze
Mads Kiilerich <madski@unity3d.com> [Mon, 18 Jul 2016 22:22:38 +0200] rev 29604
rbc: fix invalid rbc-revs entries caused by missing cache growth
It was in some cases possible to end up writing to the cache file without
growing it first. The range assignment in _setcachedata would append instead of
writing at the requested position and thus write the new record in the wrong
place.
To fix this, we avoid looking up in too small caches, and when growing the
cache, do it right before writing the new record to it so we know it has been
done correctly.
Mads Kiilerich <madski@unity3d.com> [Mon, 18 Jul 2016 22:21:42 +0200] rev 29603
rbc: test case for cache file not growing correctly, causing bad new entries
Jun Wu <quark@fb.com> [Mon, 18 Jul 2016 18:55:06 +0100] rev 29602
chg: handle EOF reading data block
We recently discovered a case in production that chg uses 100% CPU and is
trying to read data forever:
recvfrom(4, "", 1814012019, 0, NULL, NULL) = 0
Using gdb, apparently readchannel() got wrong data. It was reading in an
infinite loop because rsize == 0 does not exit the loop, while the server
process had ended.
(gdb) bt
#0 ... in recv () at /lib64/libc.so.6
#1 ... in readchannel (...) at /usr/include/bits/socket2.h:45
#2 ... in readchannel (hgc=...) at hgclient.c:129
#3 ... in handleresponse (hgc=...) at hgclient.c:255
#4 ... in hgc_runcommand (hgc=..., args=<optimized>, argsize=<optimized>)
#5 ... in main (argc=...486922636, argv=..., envp=...) at chg.c:661
(gdb) frame 2
(gdb) p *hgc
$1 = {sockfd = 4, pid = 381152, ctx = {ch = 108 'l',
data = 0x7fb05164f010 "st):\nTraceback (most recent call last):\n"
"Traceback (most recent call last):\ne", maxdatasize = 1814065152,"
" datasize = 1814064225}, capflags = 16131}
This patch addresses the infinite loop issue by detecting continuously empty
responses and abort in that case.
Note that datasize can be translated to ['l', ' ', 'l', 'a']. Concatenate
datasize and data, it forms part of "Traceback (most recent call last):".
This may indicate a server-side channeledoutput issue. If it is a race
condition, we may want to use flock to protect the channels.
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 18 Jul 2016 11:27:27 -0700] rev 29601
sslutil: more robustly detect protocol support
The Python ssl module conditionally sets the TLS 1.1 and TLS 1.2
constants depending on whether HAVE_TLSv1_2 is defined. Yes, these
are both tied to the same constant (I would think there would be
separate constants for each version). Perhaps support for TLS 1.1
and 1.2 were added at the same time and the assumption is that
OpenSSL either has neither or both. I don't know.
As part of developing this patch, it was discovered that Apple's
/usr/bin/python2.7 does not support TLS 1.1 and 1.2 (only TLS 1.0)!
On OS X 10.11, Apple Python has the modern ssl module including
SSLContext, but it doesn't appear to negotiate TLS 1.1+ nor does
it expose the constants related to TLS 1.1+. Since this code is
doing more robust feature detection (and not assuming modern ssl
implies TLS 1.1+ support), we now get TLS 1.0 warnings when running
on Apple Python. Hence the test changes.
I'm not super thrilled about shipping a Mercurial that always
whines about TLS 1.0 on OS X. We may want a follow-up patch to
suppress this warning.
Maciej Fijalkowski <fijall@gmail.com> [Mon, 11 Jul 2016 11:05:08 +0200] rev 29600
osutil: add darwin-only version of os.listdir using cffi
Yuya Nishihara <yuya@tcha.org> [Sun, 05 Jun 2016 12:29:08 +0900] rev 29599
url: drop support for proxying HTTP (not HTTPS) over CONNECT tunneling
It's been broken since cca59ef27e60, which made ui argument mandatory. I've
tried several combinations of HTTP/HTTPS proxying on old/new Python versions,
but I couldn't figure out how to reach this code path. Also, wrapping HTTP
connection by SSLSocket seems wrong. My understanding is that self.realhostport
is set by _generic_start_transaction() if HTTPS connection is tunneled.
This patch removes proxy tunneling from httpconnection.connect() assuming
that it was dead code from the beginning. Note that HTTPS over tunneling
should be handled by httpsconnection class.
Yuya Nishihara <yuya@tcha.org> [Sat, 21 May 2016 18:16:39 +0900] rev 29598
chgserver: rename private functions and variables of chgunixservicehandler
self.address has been reanmed to self._realaddress to clarify that it can
be different from the address argument.
Yuya Nishihara <yuya@tcha.org> [Sun, 22 May 2016 14:06:37 +0900] rev 29597
chgserver: refactor initialization of real/base addresses
Instead of overwriting self.address, calculate it from the address argument,
which is the base address.
Yuya Nishihara <yuya@tcha.org> [Sun, 22 May 2016 14:05:34 +0900] rev 29596
chgserver: reorder functions in chgunixservicehandler
This should make it slightly easier to follow the call path.
Yuya Nishihara <yuya@tcha.org> [Sat, 21 May 2016 18:15:20 +0900] rev 29595
chgserver: use ui.debug() to print server debug messages
commandserver.log() is noop at this time because no client connection is
established.
Yuya Nishihara <yuya@tcha.org> [Sun, 05 Jun 2016 12:18:20 +0900] rev 29594
ssl: remove special case of web.cacerts=! from remoteui()
It was introduced by b76d8c641746, which is no longer necessary thanks to
recent refactoring of sslutil including ef316c653b7f.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 15:13:51 -0700] rev 29593
bundle2: store changeset count when creating file bundles
The bundle2 changegroup part has an advisory param saying how many
changesets are in the part. Before this patch, we were setting
this part when generating bundle2 parts via the wire protocol but
not when generating local bundle2 files.
A side effect of not setting the changeset count part is that progress
bars don't work when applying changesets. As the tests show, this
impacted clone bundles, shelve, backup bundles, `hg unbundle`, and
anything touching bundle2 files.
This patch adds a backdoor to allow us to pass state from
changegroup generation into the unbundler. We store the number
of changesets in the changegroup in this state and use it to
populate the aforementioned advisory part parameter when generating
the bundle2 bundle.
I concede that I'm not thrilled by how state is being passed in
changegroup.py (it feels a bit hacky). I would love to overhaul the
rather confusing set of functions in changegroup.py with something that
passes rich objects around instead of e.g. low-level generators.
However, given the code freeze for 3.9 is imminent, I'd rather not
undertake this endeavor right now. This feels like the easiest way
to get the parameter added to the changegroup part.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 15:10:30 -0700] rev 29592
util: implement a deterministic __repr__ on sortdict
`hg debugbundle` is calling repr() on bundle2 part params, which are
now util.sortdict instances. Unfortunately, repr() doesn't appear
to be deterministic for util.sortdict. So, we implement one.
We include the type name because that's the common convention for
__repr__ implementations. Having the type name in `hg debugbundle`
is a bit ugly. But it's a debug command and I don't care enough to
fix it.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 14:51:00 -0700] rev 29591
bundle2: use a sorted dict for holding parameters
An upcoming change that introduces a 2nd part parameter to a part
reveals that `hg debugbundle` isn't deterministic because parameters
are stored on n plain, unsorted dict.
While we could change that command to sort before output, I think
the more important underlying issue is that bundle2 reading is taking
an ordered data structure and converting it to an unordered one.
Plugging in util.sortdict() fixes that problem while preserving API
compatibility.
This patch also appears to shine light on the fact that we don't
have tests verifying parts with multiple parameters roundtrip
correctly. That would be a good thing to test (and fuzz)... someday.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 15 Jul 2016 13:41:34 -0700] rev 29590
wireproto: extract repo filtering to standalone function
As part of teaching Mozilla's replication extension to better handle
repositories with obsolescence data, I encountered a few scenarios
where I wanted built-in wire protocol commands from replication clients
to operate on unfiltered repositories so they could have access to
obsolete changesets.
While the undocumented "web.view" config option provides a mechanism
to choose what filter/view hgweb operates on, this doesn't apply
to wire protocol commands because wireproto.dispatch() is always
operating on the "served" repo.
This patch extracts the line for obtaining the repo that
wireproto commands operate on to its own function so extensions
can monkeypatch it to e.g. return an unfiltered repo.
I stopped short of exposing a config option because I view the use
case for changing this as a niche feature, best left to the domain
of extensions.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 14 Jul 2016 19:16:46 -0700] rev 29589
url: add distribution and version to user-agent request header (BC)
As a server operator, I've always wanted to know what Mercurial
version clients are running so I can track version adoption and
make informed decisions about which versions of Mercurial to
support in extensions. Unfortunately, there is no easy way to discern
this today: the best you can do is look for high-level feature usage
(e.g. bundle2) or sniff capabilities from bundle2 commands. And these
things aren't changed frequently enough to tell you anything that
interesting.
Nearly every piece of software talking HTTP sends its version in
the user agent. This includes web browsers, curl, and even Git.
This patch adds the distribution name and version to the user-agent
HTTP request header. We choose "Mercurial" for the distribution
name because that seems appropriate. The version string comes
from __version__.
The value is inside parenthesis for a few reasons:
* The version *may* contain spaces
* Alternate forms like "Mercurial/<version>" imply structure and
since the user agent should not be used by servers for protocol
or feature negotiation/detection, we don't want to even give the
illusion that the value should be parsed. A free form field is
the most hostile to parsing.
Flagging the patch as BC so it shows up in release notes. This
change should be backwards compatible. But I wouldn't be surprised if
a server somewhere is filtering on the exact old user agent string. So
I want to make noise about this change.
Yuya Nishihara <yuya@tcha.org> [Sat, 16 Jul 2016 14:48:58 +0900] rev 29588
commandserver: use SOMAXCONN as queue size of pending connections
The old value 5 was arbitrary chosen. Since there's no practical reason to
limit the backlog, this patch simply uses SOMAXCONN as a value large enough.
Yuya Nishihara <yuya@tcha.org> [Sat, 16 Jul 2016 14:46:31 +0900] rev 29587
commandserver: rename _serveworker() to _runworker()
"run" sounds more natural as the function does never listen for new connection.
Yuya Nishihara <yuya@tcha.org> [Sun, 22 May 2016 13:53:32 +0900] rev 29586
commandserver: separate initialization and cleanup of forked process
Separated _initworkerprocess() and _serverequest() can be reused when
implementing a prefork service.
Yuya Nishihara <yuya@tcha.org> [Sat, 21 May 2016 18:14:13 +0900] rev 29585
commandserver: unindent superfluous "if True" blocks
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 17 Jul 2016 19:48:04 +0530] rev 29584
pycompat: make pycompat demandimport friendly
pycompat.py includes hack to import modules whose names are changed in Python 3.
We use try-except to load module according to the version of python. But this
method forces us to import the modules to raise an ImportError and hence making
it demandimport unfriendly.
This patch changes the try-except blocks to a single if-else block. To avoid
test-check-pyflakes.t complain about unused imports, pycompat.py is excluded
from the test.
Jun Wu <quark@fb.com> [Mon, 18 Jul 2016 08:55:30 +0100] rev 29583
run-tests: make --local set --with-chg if --chg is used
--local should work with chg as well.
Jun Wu <quark@fb.com> [Mon, 18 Jul 2016 08:45:46 +0100] rev 29582
run-tests: allow --local to set multiple attributes
This is to make the next patch easier to review. It does not change logic.
Jun Wu <quark@fb.com> [Sun, 17 Jul 2016 23:05:59 +0100] rev 29581
chg: add pgid to hgclient struct
The previous patch makes the server tell the client its pgid. This patch
stores it in hgclient_t and adds a function to get it.
Jun Wu <quark@fb.com> [Sun, 17 Jul 2016 22:56:05 +0100] rev 29580
commandserver: send pgid in hello message
See the next patches for why we need it.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 11:28:01 -0700] rev 29579
tests: update test certificate generation instructions
Suggestions from Anton Shestakov and Julien Cristau to use
-subj and faketime, respectively.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 11:03:08 -0700] rev 29578
sslutil: move comment about protocol constants
protocolsettings() is the appropriate place for this comment.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 10:59:32 -0700] rev 29577
sslutil: support defining cipher list
Python 2.7 supports specifying a custom cipher list to TLS sockets.
Advanced users may wish to specify a custom cipher list to increase
security. Or in some cases they may wish to prefer weaker ciphers
in order to increase performance (e.g. when doing stream clones
of very large repositories).
This patch introduces a [hostsecurity] config option for defining
the cipher list. The help documentation states that it is for
advanced users only.
Honestly, I'm a bit on the fence about providing this because
it is a footgun and can be used to decrease security. However,
there are legitimate use cases for it, so I think support should
be provided.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 17 Jul 2016 10:50:51 -0700] rev 29576
hghave: add test for Python 2.7+
Setting ciphers in the ssl module requires Python 2.7. Surprisingly,
we didn't have a test for running on Python 2.7.