url: add distribution and version to user-agent request header (BC) As a server operator, I've always wanted to know what Mercurial version clients are running so I can track version adoption and make informed decisions about which versions of Mercurial to support in extensions. Unfortunately, there is no easy way to discern this today: the best you can do is look for high-level feature usage (e.g. bundle2) or sniff capabilities from bundle2 commands. And these things aren't changed frequently enough to tell you anything that interesting. Nearly every piece of software talking HTTP sends its version in the user agent. This includes web browsers, curl, and even Git. This patch adds the distribution name and version to the user-agent HTTP request header. We choose "Mercurial" for the distribution name because that seems appropriate. The version string comes from __version__. The value is inside parenthesis for a few reasons: * The version *may* contain spaces * Alternate forms like "Mercurial/<version>" imply structure and since the user agent should not be used by servers for protocol or feature negotiation/detection, we don't want to even give the illusion that the value should be parsed. A free form field is the most hostile to parsing. Flagging the patch as BC so it shows up in release notes. This change should be backwards compatible. But I wouldn't be surprised if a server somewhere is filtering on the exact old user agent string. So I want to make noise about this change.

commandserver: use SOMAXCONN as queue size of pending connections The old value 5 was arbitrary chosen. Since there's no practical reason to limit the backlog, this patch simply uses SOMAXCONN as a value large enough.

commandserver: rename _serveworker() to _runworker() "run" sounds more natural as the function does never listen for new connection.

commandserver: separate initialization and cleanup of forked process Separated _initworkerprocess() and _serverequest() can be reused when implementing a prefork service.

commandserver: unindent superfluous "if True" blocks

pycompat: make pycompat demandimport friendly pycompat.py includes hack to import modules whose names are changed in Python 3. We use try-except to load module according to the version of python. But this method forces us to import the modules to raise an ImportError and hence making it demandimport unfriendly. This patch changes the try-except blocks to a single if-else block. To avoid test-check-pyflakes.t complain about unused imports, pycompat.py is excluded from the test.

run-tests: make --local set --with-chg if --chg is used --local should work with chg as well.

run-tests: allow --local to set multiple attributes This is to make the next patch easier to review. It does not change logic.

chg: add pgid to hgclient struct The previous patch makes the server tell the client its pgid. This patch stores it in hgclient_t and adds a function to get it.

commandserver: send pgid in hello message See the next patches for why we need it.

tests: update test certificate generation instructions Suggestions from Anton Shestakov and Julien Cristau to use -subj and faketime, respectively.

sslutil: move comment about protocol constants protocolsettings() is the appropriate place for this comment.

sslutil: support defining cipher list Python 2.7 supports specifying a custom cipher list to TLS sockets. Advanced users may wish to specify a custom cipher list to increase security. Or in some cases they may wish to prefer weaker ciphers in order to increase performance (e.g. when doing stream clones of very large repositories). This patch introduces a [hostsecurity] config option for defining the cipher list. The help documentation states that it is for advanced users only. Honestly, I'm a bit on the fence about providing this because it is a footgun and can be used to decrease security. However, there are legitimate use cases for it, so I think support should be provided.

hghave: add test for Python 2.7+ Setting ciphers in the ssl module requires Python 2.7. Surprisingly, we didn't have a test for running on Python 2.7.

spartan: make different blocks of annotated lines have different colors

monoblue: make different blocks of annotated lines have different colors

gitweb: make different blocks of annotated lines have different colors

paper: make different blocks of annotated lines have different colors

tests: check importing modules in perf.py for historical portability To check importing modules in perf.py for historical portability, this patch lists up files by "hg files" both for "1.2" and tip, and builds up "module whitelist" check from those files. This patch uses "1.2" as earlier side version of "module whitelist", because "mercurial.error" module is a blocker for loading perf.py with Mercurial earlier than 1.2, and just importing "mercurial.error" separately isn't enough.

tests: introduce check-perf-code.py to add extra checks on perf.py This patch introduces tests/check-perf-code.py as a preparation for adding extra checks on contrib/perf.py in subsequent patches (mainly, for historical portability). At this change, check-perf-code.py doesn't add any extra check, and is equal to check-code.py. This makes subsequent patch focus only on adding an extra check on perf.py check-perf-code.py. check-perf-code.py adds extra checks on perf.py by wrapping contrib/check-code.py, because "filtering" by check-code.py (e.g. normalize characters in string literal or comment line) is useful to simplify regexp for check, and avoid false positive matching.

check-code: move fixing up regexp into main procedure This patch makes an extra check pattern to be prepared by "_preparepats()" as similarly as existing patterns, if it is added to "checks" array before invocation of "main()" in check-code.py. This is a part of preparation for adding check-code.py extra checks by another python script in subsequent patch. This is also useful for SkeletonExtensionPlan. https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan

check-code: factor out boot procedure into main This is a part of preparation for adding check-code.py extra checks by another python script in subsequent patch. This is also useful for SkeletonExtensionPlan. https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan

perf: import newer modules separately for earlier Mercurial demandimport of early Mercurial loads an imported module immediately, if a module is imported absolutely by "from a import b" style. Recent perf.py satisfies this condition, because it does: - have "from __future__ import absolute_import" line - use "from a import b" style for modules in "mercurial" package Before this patch, importing modules below prevents perf.py from being loaded by earlier Mercurial, because these aren't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9. - branchmap 2.5 (or bcee63733aad) - repoview 2.5 (or 3a6ddacb7198) - obsolete 2.3 (or ad0d6c2b3279) - scmutil 1.9 (or 8b252e826c68) For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with Mercurial earlier than 1.8 (or 61c9bc3da402). After this patch, "mercurial.error" is the only blocker in "from mercurial import" statement for loading perf.py with Mercurial earlier than 1.2. This patch ignores it, because just importing it separately isn't enough.

py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import The BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer has been merged into http.server in python 3. All of them has been merged as util.httpserver to use in both python 2 and 3. This patch adds a regex to check-code to warn against the use of BaseHTTPServer. Moreover this patch also includes updates to lower part of test-check-py3-compat.t which used to remain unchanged.

py3: re-implement the BaseHTTPServer.test() function The function is changed in python 3. So the latest version of function is re-implemented. One can look at https://hg.python.org/cpython/file/3.5/Lib/http/server.py#l1184 and https://hg.python.org/cpython/file/2.7/Lib/BaseHTTPServer.py#l590 to see the change

test-http: use sed instead of fixed-with cut for reading access.log Some systems (like FreeBSD jails) use something other than 127.0.0.1 for localhost, and it's not safe to assume it'll always be the same width. Using sed with a replacement like this sidesteps the problem.

test-serve: add missing globs check-code missed this because of the closing ) in the "bound to" message.

tests: glob whitespace between path and OK in unzip(1) output FreeBSD's unzip(1) uses tabs instead of a run of spaces.

sslutil: print a warning when using TLS 1.0 on legacy Python Mercurial now requires TLS 1.1+ when TLS 1.1+ is supported by the client. Since we made the decision to require TLS 1.1+ when running with modern Python versions, it makes sense to do something for legacy Python versions that only support TLS 1.0. Feature parity would be to prevent TLS 1.0 connections out of the box and require a config option to enable them. However, this is extremely user hostile since Mercurial wouldn't talk to https:// by default in these installations! I can easily see how someone would do something foolish like use "--insecure" instead - and that would be worse than allowing TLS 1.0! This patch takes the compromise position of printing a warning when performing TLS 1.0 connections when running on old Python versions. While this warning is no more annoying than the CA certificate / fingerprint warnings in Mercurial 3.8, we provide a config option to disable the warning because to many people upgrading Python to make the warning go away is not an available recourse (unlike pinning fingerprints is for the CA warning). The warning appears as optional output in a lot of tests.

sslutil: require TLS 1.1+ when supported Currently, Mercurial will use TLS 1.0 or newer when connecting to remote servers, selecting the highest TLS version supported by both peers. On older Pythons, only TLS 1.0 is available. On newer Pythons, TLS 1.1 and 1.2 should be available. Security professionals recommend avoiding TLS 1.0 if possible. PCI DSS 3.1 "strongly encourages" the use of TLS 1.2. Known attacks like BEAST and POODLE exist against TLS 1.0 (although mitigations are available and properly configured servers aren't vulnerable). I asked Eric Rescorla - Mozilla's resident crypto expert - whether Mercurial should drop support for TLS 1.0. His response was "if you can get away with it." Essentially, a number of servers on the Internet don't support TLS 1.1+. This is why web browsers continue to support TLS 1.0 despite desires from security experts. This patch changes Mercurial's default behavior on modern Python versions to require TLS 1.1+, thus avoiding known security issues with TLS 1.0 and making Mercurial more secure by default. Rather than drop TLS 1.0 support wholesale, we still allow TLS 1.0 to be used if configured. This is a compromise solution - ideally we'd disallow TLS 1.0. However, since we're not sure how many Mercurial servers don't support TLS 1.1+ and we're not sure how much user inconvenience this change will bring, I think it is prudent to ship an escape hatch that still allows usage of TLS 1.0. In the default case our users get better security. In the worst case, they are no worse off than before this patch. This patch has no effect when running on Python versions that don't support TLS 1.1+. As the added test shows, connecting to a server that doesn't support TLS 1.1+ will display a warning message with a link to our wiki, where we can guide people to configure their client to allow less secure connections.