sslutil: require TLS 1.1+ when supported Currently, Mercurial will use TLS 1.0 or newer when connecting to remote servers, selecting the highest TLS version supported by both peers. On older Pythons, only TLS 1.0 is available. On newer Pythons, TLS 1.1 and 1.2 should be available. Security professionals recommend avoiding TLS 1.0 if possible. PCI DSS 3.1 "strongly encourages" the use of TLS 1.2. Known attacks like BEAST and POODLE exist against TLS 1.0 (although mitigations are available and properly configured servers aren't vulnerable). I asked Eric Rescorla - Mozilla's resident crypto expert - whether Mercurial should drop support for TLS 1.0. His response was "if you can get away with it." Essentially, a number of servers on the Internet don't support TLS 1.1+. This is why web browsers continue to support TLS 1.0 despite desires from security experts. This patch changes Mercurial's default behavior on modern Python versions to require TLS 1.1+, thus avoiding known security issues with TLS 1.0 and making Mercurial more secure by default. Rather than drop TLS 1.0 support wholesale, we still allow TLS 1.0 to be used if configured. This is a compromise solution - ideally we'd disallow TLS 1.0. However, since we're not sure how many Mercurial servers don't support TLS 1.1+ and we're not sure how much user inconvenience this change will bring, I think it is prudent to ship an escape hatch that still allows usage of TLS 1.0. In the default case our users get better security. In the worst case, they are no worse off than before this patch. This patch has no effect when running on Python versions that don't support TLS 1.1+. As the added test shows, connecting to a server that doesn't support TLS 1.1+ will display a warning message with a link to our wiki, where we can guide people to configure their client to allow less secure connections.

sslutil: config option to specify TLS protocol version Currently, Mercurial will use TLS 1.0 or newer when connecting to remote servers, selecting the highest TLS version supported by both peers. On older Pythons, only TLS 1.0 is available. On newer Pythons, TLS 1.1 and 1.2 should be available. Security-minded people may want to not take any risks running TLS 1.0 (or even TLS 1.1). This patch gives those people a config option to explicitly control which TLS versions Mercurial should use. By providing this option, one can require newer TLS versions before they are formally deprecated by Mercurial/Python/OpenSSL/etc and lower their security exposure. This option also provides an easy mechanism to change protocol policies in Mercurial. If there is a 0-day and TLS 1.0 is completely broken, we can act quickly without changing much code. Because setting the minimum TLS protocol is something you'll likely want to do globally, this patch introduces a global config option under [hostsecurity] for that purpose. wrapserversocket() has been taught a hidden config option to define the explicit protocol to use. This is queried in this function and not passed as an argument because I don't want to expose this dangerous option as part of the Python API. There is a risk someone could footgun themselves. But the config option is a devel option, has a warning comment, and I doubt most people are using `hg serve` to run a production HTTPS server (I would have something not Mercurial/Python handle TLS). If this is problematic, we can go back to using a custom extension in tests to coerce the server into bad behavior.

sslutil: prevent CRIME ssl.create_default_context() disables compression on the TLS channel in order to prevent CRIME. I think we should follow CPython's lead and attempt to disable channel compression in order to help prevent information leakage. Sadly, I don't think there is anything we can do on Python versions that don't have an SSLContext, as there is no way to set channel options with the limited ssl API.

sslutil: update comment about create_default_context() While ssl.create_default_context() creates a SSLContext with reasonable default options, we can't use it because it conflicts with our CA loading controls. So replace the comment with reality. (FWIW the comment was written before the existing CA loading code was in place.)

tests: use sslutil.wrapserversocket() Like the built-in HTTPS server, this code was using the ssl module directly and only using TLS 1.0. Like the built-in HTTPS server, we switch it to use sslutil.wrapserversocket() so it can follow better practices.

hgweb: use sslutil.wrapserversocket() This patch transitions the built-in HTTPS server to use sslutil for creating the server socket. As part of this transition, we implement developer-only config options to control CA loading and whether to require client certificates. This eliminates the need for the custom extension in test-https.t to define these. There is a slight change in behavior with regards to protocol selection. Before, we would always use the TLS 1.0 constant to define the protocol version. This would *only* use TLS 1.0. sslutil defaults to TLS 1.0+. So this patch improves the security of `hg serve` out of the box by allowing it to use TLS 1.1 and 1.2 (if available).

sslutil: implement wrapserversocket() wrapsocket() is heavily tailored towards client use. In preparation for converting the built-in server to use sslutil (as opposed to the ssl module directly), we add wrapserversocket() for wrapping a socket to be used on servers.

hgweb: pass ui into preparehttpserver Upcoming patches will need the built-in HTTPS server to be more configurable.

rebase: remove sortedstate-related confusion The following rebase implementation details are frustrating: - storing a list of sorted revision numbers in a field named sortedstate - having sortedstate be a field of the rebaseruntime class - using sortedstate[-1] as opposed to a more intuitive max(self.state) to compute the latest revision in the state This commit fixes those imperfections.

rebase: replace extrafn field with _makeextrafn invocations As per Yuya's advice, we would like to slightly reduce the amount of state which is stored in rebaseruntime class. In this case, we don't need to store extrafn field, as we can produce the necessary value by calling _makeextrafn and the perf overhead is negligible.

mercurial: implement a source transforming module loader on Python 3 The most painful part of ensuring Python code runs on both Python 2 and 3 is string encoding. Making this difficult is that string literals in Python 2 are bytes and string literals in Python 3 are unicode. So, to ensure consistent types are used, you have to use "from __future__ import unicode_literals" and/or prefix literals with their type (e.g. b'foo' or u'foo'). Nearly every string in Mercurial is bytes. So, to use the same source code on both Python 2 and 3 would require prefixing nearly every string literal with "b" to make it a byte literal. This is ugly and not something mpm is willing to do at this point in time. This patch implements a custom module loader on Python 3 that performs source transformation to convert string literals (unicode in Python 3) to byte literals. In effect, it changes Python 3's string literals to behave like Python 2's. In addition, the module loader recognizes well-known built-in functions (getattr, setattr, hasattr) and methods (encode and decode) that barf when bytes are used and prevents these from being rewritten. This prevents excessive source changes to accommodate this change (we would have to rewrite every occurrence of these functions passing string literals otherwise). The module loader is only used on Python packages belonging to Mercurial. The loader works by tokenizing the loaded source and replacing "string" tokens if necessary. The modified token stream is untokenized back to source and loaded like normal. This does add some overhead. However, this all occurs before caching: .pyc files will cache the transformed version. This means the transformation penalty is only paid on first load. As the extensive inline comments explain, the presence of a custom source transformer invalidates assumptions made by Python's built-in bytecode caching mechanism. So, we have to wrap bytecode loading and writing and add an additional header to bytecode files to facilitate additional cache validation when the source transformations change in the future. There are still a few things this code doesn't handle well, namely support for zip files as module sources and for extensions. Since Mercurial doesn't officially support Python 3 yet, I'm inclined to leave these as to-do items: getting a basic module loading mechanism in place to unblock further Python 3 porting effort is more important than comprehensive module importing support. check-py3-compat.py has been updated to ignore frames. This is necessary because CPython has built-in code to strip frames from the built-in importer. When our custom code is present, this doesn't work and the frames get all messed up. The new code is not perfect. It works for now. But once you start chasing import failures you find some edge cases where the files aren't being printed properly. This only burdens people doing future Python 3 porting work so I'm inclined to punt on the issue: the most important thing is for the source transforming module loader to land. There was a bit of churn in test-check-py3-compat.t because we now trip up on str/unicode/bytes failures as a result of source transformation. This is unfortunate but what are you going to do. It's worth noting that other approaches were investigated. We considered using a custom file encoding whose decode() would apply source transformations. This was rejected because it would require each source file to declare its custom Mercurial encoding. Furthermore, when changing the source transformation we'd need to version bump the encoding name otherwise the module caching layer wouldn't know the .pyc file was invalidated. This would mean mass updating every file when the source transformation changes. Yuck. We also considered transforming at the AST layer. However, Python's ast module is quite gnarly and doing AST transforms is quite complicated, even for trivial rewrites. There are whole Python packages that exist to make AST transformations usable. AST transforms would still require import machinery, so the choice was basically to perform source-level, token-level, or ast-level transforms. Token-level rewriting delivers the metadata we need to rewrite intelligently while being relatively easy to understand. So it won. General consensus seems to be that this approach is the best available to avoid bulk rewriting of '' to b''. However, we aren't confident that this approach will never be a future maintenance burden. This approach does unblock serious Python 3 porting efforts. So we can re-evaulate once more work is done to support Python 3.

compat: define ssize_t as int on 32bit Windows, silences C4142 warning It appears Python.h provides ssize_t, which is aliased to int. https://hg.python.org/cpython/file/v2.7.11/PC/pyconfig.h#l205

commandserver: drop old unixservice implementation It's been superseded by unixforkingservice.

chgserver: switch to new forking service Threading and complex classes are no longer necessary. _autoexitloop() has been replaced by polling cycle in the main thread.

chgserver: extract stub factory of service object The class inheritance will be replaced by composition. See the next patch for details.

chgserver: reorder service classes to make future patches readable Includes no functional change.

commandserver: add new forking server implemented without using SocketServer SocketServer.ForkingMixIn of Python 2.x has a couple of issues, such as: - race condition that leads to 100% CPU usage (Python 2.6) https://bugs.python.org/issue21491 - can't wait for children belonging to different process groups (Python 2.6) - leaves at least one zombie process (Python 2.6, 2.7) https://bugs.python.org/issue11109 The first two are critical because we do setpgid(0, 0) in child process to isolate terminal signals. The last one isn't, but ForkingMixIn seems to be doing silly. So there are two choices: a) backport and maintain SocketServer until we can drop support for Python 2.x b) replace SocketServer by simpler one and eliminate glue codes I chose (b) because it's great time for getting rid of utterly complicated SocketServer stuff, and preparing for future move towards prefork service. New unixforkingservice is implemented loosely based on chg 531f8ef64be6. It is monolithic but much simpler than SocketServer. unixservicehandler provides customizing points for chg, and it will be shared with future prefork service. Old unixservice class is still used by chgserver. It will be removed later. Thanks to Jun Wu for investigating these issues.

commandserver: extract function that serves for the current connection This will be used by new server implementation.

commandserver: manually create file objects from socket Prepares for moving away from SocketServer. See the subsequent patches for why.

bdiff: split bdiff into cpy-aware and cpy-agnostic part

bdiff: rename functions and structs to be amenable for later exporting

bdiff: use ssize_t in favor of Py_ssize_t in cpython-unaware locations This function and struct will be exposed via cffi, so we need to remove the cpython API dependency they currently have.

hgweb: enumerate lines in loop header, not before Doing this will allow access to the lines in arbitrary order (because the result of enumerate() is an iterator), and that will help calculating rowspan for annotate blocks.

sslutil: add assertion to prevent accidental CA usage on Windows Yuya suggested we add this check to ensure we don't accidentally try to load user-writable paths on Windows if we change the control flow of this function later.

shelve: make unshelve be able to abort in any case

osx: explicitly build hg with /usr/bin/python2.7 This should help avoid creating a package that depends on a custom Python, as happened when I built a package for 3.8.

osx: correct comment about ordering of welcome page

osx: jettison outdated build instructions

commandserver: extract _cleanup() hook to clarify chg is doing differently This makes it clear that chg needs its own way to unlink closed socket file. I made a mistake in draft patches without noting the difference.

chgserver: drop repo at chgunixservice.__init__() Since it isn't expensive operation, we don't have to delay it to init().

chgserver: extract utility to bind unix domain socket to long path This is common problem of using sockaddr_un.

chgserver: narrow scope of chdir() to socket.bind() This helps extracting a utility function.

annotate: handle empty files earlier Rather than looping on funcmap and then checking for non-zero `l` continue if the result of fctx.annotate is empty.

context: eliminate handling of linenumber being None in annotate I could not find any use of this parameter value. And it arguably makes understanding of the function more difficult. Setting the parameter default value to False.

tests: regenerate x509 test certificates The old x509 test certificates were using cryptographic settings that are ancient by today's standards, namely 512 bit RSA keys. To put things in perspective, browsers have been dropping support for 1024 bit RSA keys. I think it is important that tests match the realities of the times. And 2048 bit RSA keys with SHA-2 hashing are what the world is moving to. This patch replaces all the x509 certificates with new versions using modern best practices. In addition, the docs for generating the keys have been updated, as the existing docs left out a few steps, namely how to generate certs that were not active yet or expired.

hgweb: add a link on node id in annotate hover-box The link pointing the annotate view at this revision, just like the one in the left-column but accessible from anywhere.

hgweb: move author information from left-column to hover-box in annotate view And display the full author information since there is enough space there.

hgweb: add links to diff and changeset in hover-box on annotate view

hgweb: add link to parents of annotated revision in annotate view The link is embedded into a div with class="annotate-info" that only shows up upon hover of the annotate column. To avoid duplicate hover-overs (this new one and the one coming from link's title), drop "title" attribute from a element and put it in the annotate-info element.

compat: provide a declaration of ssize_t, for MS windows

check-code: enforce (glob) on output lines containing 127.0.0.1

tests: add (glob) annotations to output lines with 127.0.0.1

run-tests: add support for using 127.0.0.1 as a glob Some systems don't have a 127/8 address for localhost (I noticed this on a FreeBSD jail). In order to work around this, use 127.0.0.1 as a glob pattern. A future commit will update needed output lines and add a requirement to check-code.py.

check-code: only treat a # as a line in a t-test if it has a space before it Prior to this, check-code wouldn't notice things like (glob) annotations or similar in a test if they were after a # anywhere in the line. This resolves a defect in a future change, and also exposed a couple of small spots that needed some attention.

test-export: be more aggressive about quoting ^ An upcoming change to check-code will notice this isn't quoted enough. Presumably it's been fine by luck all this time.

test-check-shbang: work around check-code not wanting hardcoded paths I'm about to fix a bug in check-code that a # anywhere on a line treated the rest of the line as a comment, even if it was meaningful. This test is the one place we explicitly *do* want hardcoded paths referenced, but we can work around that by specifying bin as a regular expression.

tests: relax "Connection refused" match We already had the match relaxed on Windows, but on Google Compute Engine VMs I'm seeing "Network is unreachable" instead of "Connection refused". At this point, just give up and make sure we get an error back.

commandserver: backport handling of forking server from chgserver This is common between chg and vanilla forking server, so move it to commandserver and unify handle(). It would be debatable whether we really need gc.collect() or not, but that is beyond the scope of this series. Maybe we can remove gc.collect() once all resource deallocations are switched to context manager.

commandserver: promote .cleanup() hook from chgserver This allows us to unify _requesthandler.handle().

commandserver: extract method to create commandserver instance per request This is a step toward merging chgserver._requesthandler with commandserver's.

error: make hintable exceptions reject unknown keyword arguments (API) Previously they would accept any typos of the hint keyword.

error: make HintException a mix-in class not derived from BaseException (API) HintException is unrelated to the hierarchy of errors. It is an implementation detail whether a class inherits from HintException or not, a sort of "private inheritance" in C++. New Hint isn't an exception class, which prevents catching error by its type: try: dosomething() except error.Hint: pass Unfortunately, this passes on PyPy 5.3.1, but not on Python 2, and raises more detailed TypeError on Python 3.

sslutil: move context options flags to _hostsettings Again, moving configuration determination to a single location.

sslutil: move protocol determination to _hostsettings Most of the logic for configuring TLS is now in this function. Let's move protocol determination code there as well.

share: don't recreate the source repo each time Previously, every time you asked for the source repo of a shared working copy it would recreate the repo object, which required calling reposetup. With certain extension enabled, this can be quite expensive, and it can happen many times (for instance, share attaches a post transaction hook to update bookmarks that triggers this). The fix is to just cache the repo object instead of constantly recreating it.

setup: prepare for future cffi modules by adding placeholder in setup

journal: add support for seaching by pattern If a pattern is used, include the entry name in the output, to make it clear what name was matched.

journal: add share extension support Rather than put everything into one journal file, split entries up in *shared* and *local* entries. Working copy changes are local to a specific working copy, so should remain local only. Other entries are shared with the source if so configured when the share was created. When unsharing, any shared journale entries are copied across.

journal: add dirstate tracking Note that now the default action for `hg journal` is to list the working copy history, not all bookmarks. In its place is the `--all` switch which lists all name changes recorded, including the name for which the change was recorded on each line. Locking is switched to using a dedicated lock to avoid issues with the dirstate being written during wlock unlocking (you can't re-lock during that process).

merge with stable