commandserver: add new forking server implemented without using SocketServer SocketServer.ForkingMixIn of Python 2.x has a couple of issues, such as: - race condition that leads to 100% CPU usage (Python 2.6) https://bugs.python.org/issue21491 - can't wait for children belonging to different process groups (Python 2.6) - leaves at least one zombie process (Python 2.6, 2.7) https://bugs.python.org/issue11109 The first two are critical because we do setpgid(0, 0) in child process to isolate terminal signals. The last one isn't, but ForkingMixIn seems to be doing silly. So there are two choices: a) backport and maintain SocketServer until we can drop support for Python 2.x b) replace SocketServer by simpler one and eliminate glue codes I chose (b) because it's great time for getting rid of utterly complicated SocketServer stuff, and preparing for future move towards prefork service. New unixforkingservice is implemented loosely based on chg 531f8ef64be6. It is monolithic but much simpler than SocketServer. unixservicehandler provides customizing points for chg, and it will be shared with future prefork service. Old unixservice class is still used by chgserver. It will be removed later. Thanks to Jun Wu for investigating these issues.

commandserver: extract function that serves for the current connection This will be used by new server implementation.

commandserver: manually create file objects from socket Prepares for moving away from SocketServer. See the subsequent patches for why.

bdiff: split bdiff into cpy-aware and cpy-agnostic part

bdiff: rename functions and structs to be amenable for later exporting

bdiff: use ssize_t in favor of Py_ssize_t in cpython-unaware locations This function and struct will be exposed via cffi, so we need to remove the cpython API dependency they currently have.

hgweb: enumerate lines in loop header, not before Doing this will allow access to the lines in arbitrary order (because the result of enumerate() is an iterator), and that will help calculating rowspan for annotate blocks.

sslutil: add assertion to prevent accidental CA usage on Windows Yuya suggested we add this check to ensure we don't accidentally try to load user-writable paths on Windows if we change the control flow of this function later.

shelve: make unshelve be able to abort in any case

osx: explicitly build hg with /usr/bin/python2.7 This should help avoid creating a package that depends on a custom Python, as happened when I built a package for 3.8.

osx: correct comment about ordering of welcome page

osx: jettison outdated build instructions

commandserver: extract _cleanup() hook to clarify chg is doing differently This makes it clear that chg needs its own way to unlink closed socket file. I made a mistake in draft patches without noting the difference.

chgserver: drop repo at chgunixservice.__init__() Since it isn't expensive operation, we don't have to delay it to init().

chgserver: extract utility to bind unix domain socket to long path This is common problem of using sockaddr_un.

chgserver: narrow scope of chdir() to socket.bind() This helps extracting a utility function.

annotate: handle empty files earlier Rather than looping on funcmap and then checking for non-zero `l` continue if the result of fctx.annotate is empty.

context: eliminate handling of linenumber being None in annotate I could not find any use of this parameter value. And it arguably makes understanding of the function more difficult. Setting the parameter default value to False.

tests: regenerate x509 test certificates The old x509 test certificates were using cryptographic settings that are ancient by today's standards, namely 512 bit RSA keys. To put things in perspective, browsers have been dropping support for 1024 bit RSA keys. I think it is important that tests match the realities of the times. And 2048 bit RSA keys with SHA-2 hashing are what the world is moving to. This patch replaces all the x509 certificates with new versions using modern best practices. In addition, the docs for generating the keys have been updated, as the existing docs left out a few steps, namely how to generate certs that were not active yet or expired.

hgweb: add a link on node id in annotate hover-box The link pointing the annotate view at this revision, just like the one in the left-column but accessible from anywhere.

hgweb: move author information from left-column to hover-box in annotate view And display the full author information since there is enough space there.

hgweb: add links to diff and changeset in hover-box on annotate view

hgweb: add link to parents of annotated revision in annotate view The link is embedded into a div with class="annotate-info" that only shows up upon hover of the annotate column. To avoid duplicate hover-overs (this new one and the one coming from link's title), drop "title" attribute from a element and put it in the annotate-info element.

compat: provide a declaration of ssize_t, for MS windows

check-code: enforce (glob) on output lines containing 127.0.0.1

tests: add (glob) annotations to output lines with 127.0.0.1

run-tests: add support for using 127.0.0.1 as a glob Some systems don't have a 127/8 address for localhost (I noticed this on a FreeBSD jail). In order to work around this, use 127.0.0.1 as a glob pattern. A future commit will update needed output lines and add a requirement to check-code.py.

check-code: only treat a # as a line in a t-test if it has a space before it Prior to this, check-code wouldn't notice things like (glob) annotations or similar in a test if they were after a # anywhere in the line. This resolves a defect in a future change, and also exposed a couple of small spots that needed some attention.

test-export: be more aggressive about quoting ^ An upcoming change to check-code will notice this isn't quoted enough. Presumably it's been fine by luck all this time.

test-check-shbang: work around check-code not wanting hardcoded paths I'm about to fix a bug in check-code that a # anywhere on a line treated the rest of the line as a comment, even if it was meaningful. This test is the one place we explicitly *do* want hardcoded paths referenced, but we can work around that by specifying bin as a regular expression.

tests: relax "Connection refused" match We already had the match relaxed on Windows, but on Google Compute Engine VMs I'm seeing "Network is unreachable" instead of "Connection refused". At this point, just give up and make sure we get an error back.

commandserver: backport handling of forking server from chgserver This is common between chg and vanilla forking server, so move it to commandserver and unify handle(). It would be debatable whether we really need gc.collect() or not, but that is beyond the scope of this series. Maybe we can remove gc.collect() once all resource deallocations are switched to context manager.

commandserver: promote .cleanup() hook from chgserver This allows us to unify _requesthandler.handle().

commandserver: extract method to create commandserver instance per request This is a step toward merging chgserver._requesthandler with commandserver's.

error: make hintable exceptions reject unknown keyword arguments (API) Previously they would accept any typos of the hint keyword.

error: make HintException a mix-in class not derived from BaseException (API) HintException is unrelated to the hierarchy of errors. It is an implementation detail whether a class inherits from HintException or not, a sort of "private inheritance" in C++. New Hint isn't an exception class, which prevents catching error by its type: try: dosomething() except error.Hint: pass Unfortunately, this passes on PyPy 5.3.1, but not on Python 2, and raises more detailed TypeError on Python 3.

sslutil: move context options flags to _hostsettings Again, moving configuration determination to a single location.

sslutil: move protocol determination to _hostsettings Most of the logic for configuring TLS is now in this function. Let's move protocol determination code there as well.

share: don't recreate the source repo each time Previously, every time you asked for the source repo of a shared working copy it would recreate the repo object, which required calling reposetup. With certain extension enabled, this can be quite expensive, and it can happen many times (for instance, share attaches a post transaction hook to update bookmarks that triggers this). The fix is to just cache the repo object instead of constantly recreating it.

setup: prepare for future cffi modules by adding placeholder in setup

journal: add support for seaching by pattern If a pattern is used, include the entry name in the output, to make it clear what name was matched.

journal: add share extension support Rather than put everything into one journal file, split entries up in *shared* and *local* entries. Working copy changes are local to a specific working copy, so should remain local only. Other entries are shared with the source if so configured when the share was created. When unsharing, any shared journale entries are copied across.

journal: add dirstate tracking Note that now the default action for `hg journal` is to list the working copy history, not all bookmarks. In its place is the `--all` switch which lists all name changes recorded, including the name for which the change was recorded on each line. Locking is switched to using a dedicated lock to avoid issues with the dirstate being written during wlock unlocking (you can't re-lock during that process).

merge with stable

sslutil: try to find CA certficates in well-known locations Many Linux distros and other Nixen have CA certificates in well-defined locations. Rather than potentially fail to load any CA certificates at all (which will always result in a certificate verification failure), we scan for paths to known CA certificate files and load one if seen. Because a proper Mercurial install will have the path to the CA certificate file defined at install time, we print a warning that the install isn't proper and provide a URL with instructions to correct things. We only perform path-based fallback on Pythons that don't know how to call into OpenSSL to load the default verify locations. This is because we trust that Python/OpenSSL is properly configured and knows better than Mercurial. So this new code effectively only runs on Python <2.7.9 (technically Pythons without the modern ssl module).

sslutil: issue warning when unable to load certificates on OS X Previously, failure to load system certificates on OS X would lead to a certificate verify failure and that's it. We now print a warning message with a URL that will contain information on how to configure certificates on OS X. As the inline comment states, there is room to improve here. I think we could try harder to detect Homebrew and MacPorts installed certificate files, for example. It's worth noting that Homebrew's openssl package uses `security find-certificate -a -p` during package installation to export the system keychain root CAs to etc/openssl/cert.pem. This is something we could consider adding to setup.py. We could also encourage packagers to do this. For now, I'd just like to get this warning (which matches Windows behavior) landed. We should have time to improve things before release.

revert: don't backup if no files reverted in interactive mode (issue4793) When reverting interactively, we always backup files before prompting the user to find out if they actually want to revert them. This can create spurious *.orig files if a user enters an interactive revert session and then doesn't revert any files. Instead, we should only backup files that are actually being touched.

perf: define command annotation locally for Mercurial earlier than 3.1 Before this patch, using cmdutil.command() for "@command" annotation prevents perf.py from being loaded by Mercurial earlier than 1.9 (or 2daa5179e73f), because cmdutil.command() isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). In addition to it, "norepo" option of command annotation has been available since 3.1 (or 75a96326cecb), and this is another blocker for loading perf.py with earlier Mercurial. ============ ============ ====== command of hg version cmdutil norepo ============ ============ ====== 3.1 or later o o 1.9 or later o x earlier x x ============ ============ ====== This patch defines "command()" for annotation locally as below: - define wrapper of existing cmdutil.command(), if cmdutil.command() doesn't support "norepo" (for Mercurial earlier than 3.1) - define full command() locally with minimum function, if cmdutil.command() isn't available at runtime (for Mercurial earlier than 1.9) This patch also defines parsealiases() locally without examining whether it is available or not, because it is small enough to define locally.

perf: avoid using formatteropts for Mercurial earlier than 3.2 Before this patch, referring commands.formatteropts prevents perf.py from being loaded by Mercurial earlier than 3.2 (or 7a7eed5176a4), because it isn't available in such Mercurial, even though formatting itself has been available since 2.2 (or ae5f92e154d3). In addition to it, there are some code paths for Mercurial earlier than 3.2. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). This patch uses empty option list as formatteropts, if it isn't available in commands module at runtime. Disabling -T/--template option for earlier Mercurial should be reasonable, because: - since 427e80a18ef8, -T/--template for formatter has been available - since 7a7eed5176a4, commands.formatteropts has been available - the latter revision is direct child of the former

perf: use locally defined revlog option list for Mercurial earlier than 3.7 Before this patch, referring commands.debugrevlogopts prevents perf.py from being loaded by Mercurial earlier than 3.7 (or 5606f7d0d063), because it isn't available in such Mercurial, even though cmdutil.openrevlog(), a user of these options, has been available since 1.9 (or a79fea6b3e77). In addition to it, there are some code paths for Mercurial earlier than 3.7. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). But just "using locally defined revlog option list" might cause unexpected behavior at runtime. If --dir option is specified to cmdutil.openrevlog() of Mercurial earlier than 3.5 (or 49c583ca48c4), it is silently ignored without any warning or so. ============ ============ ===== =============== debugrevlogopts hg version openrevlog() --dir of commands ============ ============ ===== =============== 3.7 or later o o o 3.5 or later o o x 1.9 or later o x x earlier x x x ============ ============ ===== =============== Therefore, this patch does: - use locally defined option list, if commands.debugrevlogopts isn't available (for Mercurial earlier than 3.7) - wrap cmdutil.openrevlog(), if it is ambiguous whether cmdutil.openrevlog() can recognize --dir option correctly (for Mercurial earlier than 3.5) This wrapper function aborts execution, if: - --dir option is specified, and - localrepository doesn't have "dirlog" attribute, which indicates that localrepository has a function for '--dir' BTW, extensions.wrapfunction() has been available since 1.1 (or 0ab5f21c390b), and this seems old enough for "historical portability" of perf.py, which has been available since 1.1 (or eb240755386d).

perf: define util.safehasattr forcibly for Mercurial earlier than 1.9.3 Before this patch, using util.safehasattr() prevents perf.py from being loaded by Mercurial earlier than 1.9.3 (or 94b200a11cf7), because util.safehasattr() isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9.3. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with Mercurial earlier than 1.8 (or 61c9bc3da402). This patch is a preparation for using util.safehasattr() safely in subsequent patches. This patch defines util.safehasattr() forcibly without examining whether it is available or not, because: - examining existence of "safehasattr" safely itself needs similar logic - safehasattr() is small enough to define locally

perf: add historical portability policy for future reference

tests: check ETag format in test-hgweb-commands

hgweb: emit a valid, weak ETag Previously, ETag headers from hgweb weren't correctly formed, because rfc2616 (section 14, header definitions) requires double quotes around the content of the header. str(web.mtime) didn't do that. Additionally, strong ETags signify that the resource representations are byte-for-byte identical. That is, they can be reconstructed from byte ranges if client so wishes. Considering ETags for all hgweb pages is just mtime of 00changelog.i and doesn't consider of e.g. .hg/hgrc with description, contact and other fields, it's clearly shouldn't be strong. The W/ prefix marks it as weak, which still allows caching the whole served file/page, but doesn't allow byte-range requests.

policy: add cffi policy for PyPy This adds cffi policy in the case where we don't want to use C modules, but instead we're happy to rely on cffi (bundled with pypy)

sslutil: handle default CA certificate loading on Windows See the inline comment for what's going on here. There is magic built into the "ssl" module that ships with modern CPython that knows how to load the system CA certificates on Windows. Since we're not shipping a CA bundle with Mercurial, if we're running on legacy CPython there's nothing we can do to load CAs on Windows, so it makes sense to print a warning. I don't anticipate many people will see this warning because the official (presumed popular) Mercurial distributions on Windows bundle Python and should be distributing a modern Python capable of loading system CA certs.

sslutil: expand _defaultcacerts docstring to note calling assumptions We should document this so future message additions don't seem out of place.

sslutil: document the Apple OpenSSL cert trick This is sort of documented in _plainapplypython()'s docstring. But it helps to be explicit in security code.

sslutil: use certificates provided by certifi if available The "certifi" Python package provides a distribution of the Mozilla trusted CA certificates as a Python package. If it is present, we assume the user intends it to be used and we use it to provide the default CA certificates when certificates are otherwise not configured. It's worth noting that this behavior roughly matches the popular "requests" package, which also attempts to use "certifi" if present.

py3: make files use absolute_import and print_function This patch includes addition of absolute_import and print_function to the files where they are missing. The modern importing conventions are also followed.

sslutil: don't attempt to find default CA certs file when told not to Before, devel.disableloaddefaultcerts only impacted the loading of default certs via SSLContext. After this patch, the config option also prevents sslutil._defaultcacerts() from being called. This config option is meant to be used by tests to force no CA certs to be loaded. Future patches will enable _defaultcacerts() to have success more often. Without this change we can't reliably test the failure to load CA certs. (This patch also likely fixes test failures on some OS X configurations.)

sslutil: pass ui to _defaultcacerts We'll use this shortly.

sslutil: change comment and logged message for found ca cert file Future patches will change _defaultcacerts() to do something on platforms that aren't OS X. Change the comment and logged message to reflect the future.

tests: better testing of loaded certificates Tests were failing on systems like RHEL 7 where loading the system certificates results in CA certs being reported to Python. We add a feature that detects when we're able to load *and detect* the loading of system certificates. We update the tests to cover the 3 scenarios: 1) system CAs are loadable and detected 2) system CAs are loadable but not detected 3) system CAs aren't loadable