gitweb: make different blocks of annotated lines have different colors

paper: make different blocks of annotated lines have different colors

tests: check importing modules in perf.py for historical portability To check importing modules in perf.py for historical portability, this patch lists up files by "hg files" both for "1.2" and tip, and builds up "module whitelist" check from those files. This patch uses "1.2" as earlier side version of "module whitelist", because "mercurial.error" module is a blocker for loading perf.py with Mercurial earlier than 1.2, and just importing "mercurial.error" separately isn't enough.

tests: introduce check-perf-code.py to add extra checks on perf.py This patch introduces tests/check-perf-code.py as a preparation for adding extra checks on contrib/perf.py in subsequent patches (mainly, for historical portability). At this change, check-perf-code.py doesn't add any extra check, and is equal to check-code.py. This makes subsequent patch focus only on adding an extra check on perf.py check-perf-code.py. check-perf-code.py adds extra checks on perf.py by wrapping contrib/check-code.py, because "filtering" by check-code.py (e.g. normalize characters in string literal or comment line) is useful to simplify regexp for check, and avoid false positive matching.

check-code: move fixing up regexp into main procedure This patch makes an extra check pattern to be prepared by "_preparepats()" as similarly as existing patterns, if it is added to "checks" array before invocation of "main()" in check-code.py. This is a part of preparation for adding check-code.py extra checks by another python script in subsequent patch. This is also useful for SkeletonExtensionPlan. https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan

check-code: factor out boot procedure into main This is a part of preparation for adding check-code.py extra checks by another python script in subsequent patch. This is also useful for SkeletonExtensionPlan. https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan

perf: import newer modules separately for earlier Mercurial demandimport of early Mercurial loads an imported module immediately, if a module is imported absolutely by "from a import b" style. Recent perf.py satisfies this condition, because it does: - have "from __future__ import absolute_import" line - use "from a import b" style for modules in "mercurial" package Before this patch, importing modules below prevents perf.py from being loaded by earlier Mercurial, because these aren't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9. - branchmap 2.5 (or bcee63733aad) - repoview 2.5 (or 3a6ddacb7198) - obsolete 2.3 (or ad0d6c2b3279) - scmutil 1.9 (or 8b252e826c68) For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with Mercurial earlier than 1.8 (or 61c9bc3da402). After this patch, "mercurial.error" is the only blocker in "from mercurial import" statement for loading perf.py with Mercurial earlier than 1.2. This patch ignores it, because just importing it separately isn't enough.

py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import The BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer has been merged into http.server in python 3. All of them has been merged as util.httpserver to use in both python 2 and 3. This patch adds a regex to check-code to warn against the use of BaseHTTPServer. Moreover this patch also includes updates to lower part of test-check-py3-compat.t which used to remain unchanged.

py3: re-implement the BaseHTTPServer.test() function The function is changed in python 3. So the latest version of function is re-implemented. One can look at https://hg.python.org/cpython/file/3.5/Lib/http/server.py#l1184 and https://hg.python.org/cpython/file/2.7/Lib/BaseHTTPServer.py#l590 to see the change

test-http: use sed instead of fixed-with cut for reading access.log Some systems (like FreeBSD jails) use something other than 127.0.0.1 for localhost, and it's not safe to assume it'll always be the same width. Using sed with a replacement like this sidesteps the problem.

test-serve: add missing globs check-code missed this because of the closing ) in the "bound to" message.

tests: glob whitespace between path and OK in unzip(1) output FreeBSD's unzip(1) uses tabs instead of a run of spaces.

sslutil: print a warning when using TLS 1.0 on legacy Python Mercurial now requires TLS 1.1+ when TLS 1.1+ is supported by the client. Since we made the decision to require TLS 1.1+ when running with modern Python versions, it makes sense to do something for legacy Python versions that only support TLS 1.0. Feature parity would be to prevent TLS 1.0 connections out of the box and require a config option to enable them. However, this is extremely user hostile since Mercurial wouldn't talk to https:// by default in these installations! I can easily see how someone would do something foolish like use "--insecure" instead - and that would be worse than allowing TLS 1.0! This patch takes the compromise position of printing a warning when performing TLS 1.0 connections when running on old Python versions. While this warning is no more annoying than the CA certificate / fingerprint warnings in Mercurial 3.8, we provide a config option to disable the warning because to many people upgrading Python to make the warning go away is not an available recourse (unlike pinning fingerprints is for the CA warning). The warning appears as optional output in a lot of tests.

sslutil: require TLS 1.1+ when supported Currently, Mercurial will use TLS 1.0 or newer when connecting to remote servers, selecting the highest TLS version supported by both peers. On older Pythons, only TLS 1.0 is available. On newer Pythons, TLS 1.1 and 1.2 should be available. Security professionals recommend avoiding TLS 1.0 if possible. PCI DSS 3.1 "strongly encourages" the use of TLS 1.2. Known attacks like BEAST and POODLE exist against TLS 1.0 (although mitigations are available and properly configured servers aren't vulnerable). I asked Eric Rescorla - Mozilla's resident crypto expert - whether Mercurial should drop support for TLS 1.0. His response was "if you can get away with it." Essentially, a number of servers on the Internet don't support TLS 1.1+. This is why web browsers continue to support TLS 1.0 despite desires from security experts. This patch changes Mercurial's default behavior on modern Python versions to require TLS 1.1+, thus avoiding known security issues with TLS 1.0 and making Mercurial more secure by default. Rather than drop TLS 1.0 support wholesale, we still allow TLS 1.0 to be used if configured. This is a compromise solution - ideally we'd disallow TLS 1.0. However, since we're not sure how many Mercurial servers don't support TLS 1.1+ and we're not sure how much user inconvenience this change will bring, I think it is prudent to ship an escape hatch that still allows usage of TLS 1.0. In the default case our users get better security. In the worst case, they are no worse off than before this patch. This patch has no effect when running on Python versions that don't support TLS 1.1+. As the added test shows, connecting to a server that doesn't support TLS 1.1+ will display a warning message with a link to our wiki, where we can guide people to configure their client to allow less secure connections.

sslutil: config option to specify TLS protocol version Currently, Mercurial will use TLS 1.0 or newer when connecting to remote servers, selecting the highest TLS version supported by both peers. On older Pythons, only TLS 1.0 is available. On newer Pythons, TLS 1.1 and 1.2 should be available. Security-minded people may want to not take any risks running TLS 1.0 (or even TLS 1.1). This patch gives those people a config option to explicitly control which TLS versions Mercurial should use. By providing this option, one can require newer TLS versions before they are formally deprecated by Mercurial/Python/OpenSSL/etc and lower their security exposure. This option also provides an easy mechanism to change protocol policies in Mercurial. If there is a 0-day and TLS 1.0 is completely broken, we can act quickly without changing much code. Because setting the minimum TLS protocol is something you'll likely want to do globally, this patch introduces a global config option under [hostsecurity] for that purpose. wrapserversocket() has been taught a hidden config option to define the explicit protocol to use. This is queried in this function and not passed as an argument because I don't want to expose this dangerous option as part of the Python API. There is a risk someone could footgun themselves. But the config option is a devel option, has a warning comment, and I doubt most people are using `hg serve` to run a production HTTPS server (I would have something not Mercurial/Python handle TLS). If this is problematic, we can go back to using a custom extension in tests to coerce the server into bad behavior.

sslutil: prevent CRIME ssl.create_default_context() disables compression on the TLS channel in order to prevent CRIME. I think we should follow CPython's lead and attempt to disable channel compression in order to help prevent information leakage. Sadly, I don't think there is anything we can do on Python versions that don't have an SSLContext, as there is no way to set channel options with the limited ssl API.

sslutil: update comment about create_default_context() While ssl.create_default_context() creates a SSLContext with reasonable default options, we can't use it because it conflicts with our CA loading controls. So replace the comment with reality. (FWIW the comment was written before the existing CA loading code was in place.)

tests: use sslutil.wrapserversocket() Like the built-in HTTPS server, this code was using the ssl module directly and only using TLS 1.0. Like the built-in HTTPS server, we switch it to use sslutil.wrapserversocket() so it can follow better practices.

hgweb: use sslutil.wrapserversocket() This patch transitions the built-in HTTPS server to use sslutil for creating the server socket. As part of this transition, we implement developer-only config options to control CA loading and whether to require client certificates. This eliminates the need for the custom extension in test-https.t to define these. There is a slight change in behavior with regards to protocol selection. Before, we would always use the TLS 1.0 constant to define the protocol version. This would *only* use TLS 1.0. sslutil defaults to TLS 1.0+. So this patch improves the security of `hg serve` out of the box by allowing it to use TLS 1.1 and 1.2 (if available).

sslutil: implement wrapserversocket() wrapsocket() is heavily tailored towards client use. In preparation for converting the built-in server to use sslutil (as opposed to the ssl module directly), we add wrapserversocket() for wrapping a socket to be used on servers.

hgweb: pass ui into preparehttpserver Upcoming patches will need the built-in HTTPS server to be more configurable.

rebase: remove sortedstate-related confusion The following rebase implementation details are frustrating: - storing a list of sorted revision numbers in a field named sortedstate - having sortedstate be a field of the rebaseruntime class - using sortedstate[-1] as opposed to a more intuitive max(self.state) to compute the latest revision in the state This commit fixes those imperfections.

rebase: replace extrafn field with _makeextrafn invocations As per Yuya's advice, we would like to slightly reduce the amount of state which is stored in rebaseruntime class. In this case, we don't need to store extrafn field, as we can produce the necessary value by calling _makeextrafn and the perf overhead is negligible.

mercurial: implement a source transforming module loader on Python 3 The most painful part of ensuring Python code runs on both Python 2 and 3 is string encoding. Making this difficult is that string literals in Python 2 are bytes and string literals in Python 3 are unicode. So, to ensure consistent types are used, you have to use "from __future__ import unicode_literals" and/or prefix literals with their type (e.g. b'foo' or u'foo'). Nearly every string in Mercurial is bytes. So, to use the same source code on both Python 2 and 3 would require prefixing nearly every string literal with "b" to make it a byte literal. This is ugly and not something mpm is willing to do at this point in time. This patch implements a custom module loader on Python 3 that performs source transformation to convert string literals (unicode in Python 3) to byte literals. In effect, it changes Python 3's string literals to behave like Python 2's. In addition, the module loader recognizes well-known built-in functions (getattr, setattr, hasattr) and methods (encode and decode) that barf when bytes are used and prevents these from being rewritten. This prevents excessive source changes to accommodate this change (we would have to rewrite every occurrence of these functions passing string literals otherwise). The module loader is only used on Python packages belonging to Mercurial. The loader works by tokenizing the loaded source and replacing "string" tokens if necessary. The modified token stream is untokenized back to source and loaded like normal. This does add some overhead. However, this all occurs before caching: .pyc files will cache the transformed version. This means the transformation penalty is only paid on first load. As the extensive inline comments explain, the presence of a custom source transformer invalidates assumptions made by Python's built-in bytecode caching mechanism. So, we have to wrap bytecode loading and writing and add an additional header to bytecode files to facilitate additional cache validation when the source transformations change in the future. There are still a few things this code doesn't handle well, namely support for zip files as module sources and for extensions. Since Mercurial doesn't officially support Python 3 yet, I'm inclined to leave these as to-do items: getting a basic module loading mechanism in place to unblock further Python 3 porting effort is more important than comprehensive module importing support. check-py3-compat.py has been updated to ignore frames. This is necessary because CPython has built-in code to strip frames from the built-in importer. When our custom code is present, this doesn't work and the frames get all messed up. The new code is not perfect. It works for now. But once you start chasing import failures you find some edge cases where the files aren't being printed properly. This only burdens people doing future Python 3 porting work so I'm inclined to punt on the issue: the most important thing is for the source transforming module loader to land. There was a bit of churn in test-check-py3-compat.t because we now trip up on str/unicode/bytes failures as a result of source transformation. This is unfortunate but what are you going to do. It's worth noting that other approaches were investigated. We considered using a custom file encoding whose decode() would apply source transformations. This was rejected because it would require each source file to declare its custom Mercurial encoding. Furthermore, when changing the source transformation we'd need to version bump the encoding name otherwise the module caching layer wouldn't know the .pyc file was invalidated. This would mean mass updating every file when the source transformation changes. Yuck. We also considered transforming at the AST layer. However, Python's ast module is quite gnarly and doing AST transforms is quite complicated, even for trivial rewrites. There are whole Python packages that exist to make AST transformations usable. AST transforms would still require import machinery, so the choice was basically to perform source-level, token-level, or ast-level transforms. Token-level rewriting delivers the metadata we need to rewrite intelligently while being relatively easy to understand. So it won. General consensus seems to be that this approach is the best available to avoid bulk rewriting of '' to b''. However, we aren't confident that this approach will never be a future maintenance burden. This approach does unblock serious Python 3 porting efforts. So we can re-evaulate once more work is done to support Python 3.

compat: define ssize_t as int on 32bit Windows, silences C4142 warning It appears Python.h provides ssize_t, which is aliased to int. https://hg.python.org/cpython/file/v2.7.11/PC/pyconfig.h#l205

commandserver: drop old unixservice implementation It's been superseded by unixforkingservice.

chgserver: switch to new forking service Threading and complex classes are no longer necessary. _autoexitloop() has been replaced by polling cycle in the main thread.

chgserver: extract stub factory of service object The class inheritance will be replaced by composition. See the next patch for details.

chgserver: reorder service classes to make future patches readable Includes no functional change.

commandserver: add new forking server implemented without using SocketServer SocketServer.ForkingMixIn of Python 2.x has a couple of issues, such as: - race condition that leads to 100% CPU usage (Python 2.6) https://bugs.python.org/issue21491 - can't wait for children belonging to different process groups (Python 2.6) - leaves at least one zombie process (Python 2.6, 2.7) https://bugs.python.org/issue11109 The first two are critical because we do setpgid(0, 0) in child process to isolate terminal signals. The last one isn't, but ForkingMixIn seems to be doing silly. So there are two choices: a) backport and maintain SocketServer until we can drop support for Python 2.x b) replace SocketServer by simpler one and eliminate glue codes I chose (b) because it's great time for getting rid of utterly complicated SocketServer stuff, and preparing for future move towards prefork service. New unixforkingservice is implemented loosely based on chg 531f8ef64be6. It is monolithic but much simpler than SocketServer. unixservicehandler provides customizing points for chg, and it will be shared with future prefork service. Old unixservice class is still used by chgserver. It will be removed later. Thanks to Jun Wu for investigating these issues.