treemanifests: actually strip directory manifests Stripping has only partly worked since 7cbb3a01fa38 (repair: use cg3 for treemanifests, 2016-01-19): the bundle seems to have been created correctly, but revlog entries in subdirectory revlogs were not stripped. This meant that e.g. "hg verify" would fail after stripping in a tree manifest repo. To find the revisions to strip, we simply iterate over all directories in the repo (included in store.datafiles()). This is inefficient for stripping few commits, but efficient for stripping many commits. To optimize for stripping few commits, we could instead walk the tree from the root and find modified subdirectories, just like we do in the changegroup code. I'm leaving that for another day.

logtoprocess: do not leak the ui object in uisetup logtoprocess.log should use "self" passed in function arguments instead of the "ui" object from outside the function.

chgserver: document why we don't merge mtimehash and confighash People may get confused about chg's mtimehash and confighash design: why two hashes instead of just one. This patch adds text addressing the concern.

extensions: move uisetup and extsetup to standalone functions This is to make them wrap-able. chgserver wants to know if an extension accesses config or environment variables during uisetup and extsetup and include them in confighash accordingly.

sslutil: don't access message attribute in exception (issue5285) I should have ran the entire test suite on Python 2.6. Since the hostname matching tests are implemented in Python (not .t tests), it didn't uncover this warning. I'm not sure why - warnings should be printed regardless. This is possibly a bug in the test runner. But that's for another day...

merge with stable

hgweb: add absolute urls for archives in json-summary

tests: allow bz2 archives in test-hgweb-json.t Only testing a specific type because list items seem to be in arbitrary order.

keepalive: switch from thread to threading module The thread module in py3 is renamed to _thread, but we can use the high level threading module instead.

py3: conditionalize httplib import The httplib library is renamed to http.client in python 3. So the import is conditionalized and a test is added in check-code to warn to use util.httplib

Added signature for changeset 26a5d605b868

Added tag 3.8.4 for changeset 26a5d605b868

sslutil: synchronize hostname matching logic with CPython sslutil contains its own hostname matching logic. CPython has code for the same intent. However, it is only available to Python 2.7.9+ (or distributions that have backported 2.7.9's ssl module improvements). This patch effectively imports CPython's hostname matching code from its ssl.py into sslutil.py. The hostname matching code itself is pretty similar. However, the DNS name matching code is much more robust and spec conformant. As the test changes show, this changes some behavior around wildcard handling and IDNA matching. The new behavior allows wildcards in the middle of words (e.g. 'f*.com' matches 'foo.com') This is spec compliant according to RFC 6125 Section 6.5.3 item 3. There is one test where the matcher is more strict. Before, '*.a.com' matched '.a.com'. Now it doesn't match. Strictly speaking this is a security vulnerability.

tests: import CPython's hostname matching tests CPython has a more comprehensive test suite for it's built-in hostname matching functionality. This patch adds its tests so we can improve our hostname matching functionality. Many of the tests have different results from CPython. These will be addressed in a subsequent commit.

i18n-pt_BR: synchronized with dd9175ca81dc

sslutil: emit warning when no CA certificates loaded If no CA certificates are loaded, that is almost certainly a/the reason certificate verification fails when connecting to a server. The modern ssl module in Python 2.7.9+ provides an API to access the list of loaded CA certificates. This patch emits a warning on modern Python when certificate verification fails and there are no loaded CA certificates. There is no way to detect the number of loaded CA certificates unless the modern ssl module is present. Hence the differences in test output depending on whether modern ssl is available. It's worth noting that a test which specifies a CA file still renders this warning. That is because the certificate it is loading is a x509 client certificate and not a CA certificate. This test could be updated if anyone is so inclined.

tests: test case where default ca certs not available I'm not a fan of TLS tests not testing both branches of a possible configuration. While we have test coverage of the inability to validate a cert later in this file, I insist that we add this branch so our testing of security code is extra comprehensive.

sslutil: don't load default certificates when they aren't relevant Before, we would call SSLContext.load_default_certs() when certificate verification wasn't being used. Since SSLContext.verify_mode == ssl.CERT_NONE, this would ideally no-op. However, there is a slim chance the loading of system certs could cause a failure. Furthermore, this behavior interfered with a future patch that aims to provide a more helpful error message when we're unable to load CAs. The lack of test fallout is hopefully a sign that our security code and tests are in a relatively good state.

sslutil: display a better error message when CA file loading fails Before, sslcontext.load_verify_locations() would raise a ssl.SSLError which would be caught further up the stack and converted to a urlerror. By that time, we lost track of what actually errored. Trapping the error here gives users a slightly more actionable error message. The behavior between Python <2.7.9 and Python 2.7.9+ differs. This is because our fake SSLContext class installed on <2.7.9 doesn't actually do anything during load_verify_locations: it defers actions until wrap_socket() time. Unfortunately, a number of errors can occur at wrap_socket() time and we're unable to ascertain what the root cause is. But that shouldn't stop us from providing better error messages to people running a modern and secure Python version.

tests: add test for empty CA certs file smf reported that an environment with no loaded CA certs resulted in a weird error. I'd like to detect this a bit better so we can display an actionable error message. The actual error being globbed over in this patch is "unknown error" with a ssl.c line number. That isn't useful at all.

internals: move the bitmanipulation routines into its own file This is to allow more flexibility with the C sources -- now the bitmanipulation routines can be safely imported without importing Python.h

journal: new experimental extension Records bookmark locations and shows you where bookmarks were located in the past. This is the first in a planned series of locations to be recorded; a future patch will add working copy (dirstate) tracking, and remote bookmarks will be supported as well, so the journal storage format should be fairly generic to support those use-cases.

httpclient: update to 54868ef054d2 of httpplus As of that revision, httpplus fully supports Python 3, including mimicing all the subtle behavior changes around headers in Python 3's http.client.

revset: check invalid function syntax "func-name"() explicitly Before the error was caught at func() as an unknown identifier, and the optimizer failed to detect the syntax error. This patch introduces getsymbol() helper to ensure that a string is not allowed as a function name.

chg: silence warning of unused parameter 'sig'

hgweb: reindent atom/changelogentry.tmpl It was mixing tabs and spaces, and not in a good way. Indent style of other atom entries seems to be 1 space per level, so let's apply it here as well.

hgweb: remove unused/nonexistent filelogentry from atom/map filelog in atom style uses changelogentry to show commits, and atom/filelogentry.tmpl doesn't even exist.

hgweb: remove unnecessary CDATA markup in atom/branchentry.tmpl Since content is of type "text" (and is already escaped), using a CDATA section is not required. Looks like this was just an artifact of copying things from rss style in add2f9ddcfb5, because other entries in atom style don't use CDATA in such places.

hgweb: reindent rss/changelogentry.tmpl It was mixing tabs and spaces, and not in a good way. Indent style of other rss entries seems to be 4 spaces per level, so let's apply it here as well.

hgweb: remove a couple of superfluous spaces in rss style

py3: add tests in check-code to load modules from util.py The conditionalize imports are added in util.py and now we import modules from there. So adding tests so that someone in future can use that.

py3: conditionalize SocketServer import The SocketServer is renamed to socketserver in python 3

py3: conditionalize xmlrpclib import The xmlrpclib library is renamed to xmlrpc.client in python 3

py3: conditionalize the urlparse import The urlparse library is renamed to urllib.parse in python 3

py3: update tests/test-check-py3-compat.t The lower part of the test runs with python 3 and hence remain unchanged.

chg: send SIGPIPE to server immediately when pager exits (issue5278) If the user press 'q' to leave the 'less' pager, it is expected to end the hg process immediately. We currently rely on SIGPIPE for this behavior. But SIGPIPE won't arrive if we don't write anything (like doing heavy computation, reading from network etc). If that happens, the user will feel that the hg process just hangs. The patch address the issue by adding a SIGCHLD signal handler and sends SIGPIPE to the server as soon as the pager exits. This is also an issue with hg's pager implementation.

chgserver: do not ignore SIGPIPE if pager is used We rely on SIGPIPE to exit when the pager exits. And Python ignores SIGPIPE by default. Explicitly set SIGPIPE handler to SIG_DFL (terminate) just like pager.py.

debug: make debug{revlog,index,data} --dir not just a flag The directory argument (for tree manifests) should belong to to the --dir argument. I had mistakenly made --dir a flag. One effect of this was that I had meant for "-m" to be optional, but instead it changed the behavior of --dir, so with "hg debugdata -m --dir dir1 0", the -m took over and the "dir1" got treated as a revision in the root manifest log.

debugdata: disallow trailing option with -c/-m Before this change, "hg debugdata -c 0 foo" was allowed.

revset: get rid of redundant error checking from match() Actually there was no additional error checking. It should be caught by "not all(specs)".

share: move magic string to a constant

branchmap: remove unused exception variable

patch: allow copy information to be passed in When displaying patches from graphical tools where you can browse through individual files, with diff being called separately on each, recomputing the limits of file copy history can become rather expensive on large repositories. Instead, we can compute it once and pass it in for subsequent calls.

largefiles: check file in the repo store before checking remotely (issue5257) Problem was files to check were gathered in the repository where the verify was launched but verification was done on the remote store. It was observed when user committed in cloned repository and ran verify before pushing - committed files were marked as non existing. This commit fixes this by checking in the remote store only files that are not existing in the repository store where verify was launched. Solution is similiar to fd288d118074

largefiles: remove additional blank lines It does not conform to the coding style.

largefiles: fix misleading comments in lfutil instore and storepath Problem in both cases is cache in largefiles has assigned meaning - user cache which is additional place to get/put files. Those two function works on store - the main place to store largefiles in the repository - .hg/largefiles and using "cache" to describe it is misleading.

revset: implement match() in terms of matchany() match() is the special case of a single element list being passed to matchany() with the additional error checking that the revset spec is defined. Change the implementation to remove the redundant code and have match() call matchany().

scmutil: improve documentation of revset APIs I can never remember the differences between the various revset APIs. I can never remember that scmutil.revrange() is the one I want to use from user-facing commands. Add some documentation to clarify this. While we're here, the argument name for revrange() is changed to "specs" because that's what it actually is.

mdiff: remove use of __slots__ The use of __slots__ was added way back in 2006 in 4ec58b157265. __slots__ isn't necessary for this class.

i18n: use unicode literal Other parts of this expression are already using unicode literals. We need this to make Python 3 happy and to avoid an implicit conversion in Python 2.

pycompat: add HTTPPasswordMgrWithDefaultRealm to Python 3 block Looks like we missed this in 800ec7c048b0.

ui: path option to declare which revisions to push by default Now that we have a mechanism for declaring path sub-options, we can start to pile on features! Many power users have expressed frustration that bare `hg push` attempts to push all local revisions to the remote. This patch introduces the "pushrev" path sub-option to control which revisions are pushed when no "-r" argument is specified. The value of this sub-option is a revset, naturally. A future feature addition could potentially introduce a "pushnames" sub-options that declares the list of names (branches, bookmarks, topics, etc) to push by default. The entire "what to push by default" feature should probably be considered before this patch lands.

ui: don't fixup [paths] sub-options As part of developing a subsequent patch I discovered that sub-option values like "." were getting converted to paths. This is because the [paths] section is treated specially during config loading. This patch prevents post-processing sub-options from the [paths] section.

sslutil: abort when unable to verify peer connection (BC) Previously, when we connected to a server and were unable to verify its certificate against a trusted certificate authority we would issue a warning and continue to connect. This is obviously not great behavior because the x509 certificate model is based upon trust of specific CAs. Failure to enforce that trust erodes security. This behavior was defined several years ago when Python did not support loading the system trusted CA store (Python 2.7.9's backports of Python 3's improvements to the "ssl" module enabled this). This commit changes behavior when connecting to abort if the peer certificate can't be validated. With an empty/default Mercurial configuration, the peer certificate can be validated if Python is able to load the system trusted CA store. Environments able to load the system trusted CA store include: * Python 2.7.9+ on most platforms and installations * Python 2.7 distributions with a modern ssl module (e.g. RHEL7's patched 2.7.5 package) * Python shipped on OS X Environments unable to load the system trusted CA store include: * Python 2.6 * Python 2.7 on many existing Linux installs (because they don't ship 2.7.9+ or haven't backported modern ssl module) * Python 2.7.9+ on some installs where Python is unable to locate the system CA store (this is hopefully rare) Users of these Pythongs will need to configure Mercurial to load the system CA store using web.cacerts. This should ideally be performed by packagers (by setting web.cacerts in the global/system hgrc file). Where Mercurial packagers aren't setting this, the linked URL in the new abort message can contain instructions for users. In the future, we may want to add more code for finding the system CA store. For example, many Linux distributions have the CA store at well-known locations (such as /etc/ssl/certs/ca-certificates.crt in the case of Ubuntu). This will enable CA loading to "just work" on more Python configurations and will be best for our users since they won't have to change anything after upgrading to a Mercurial with this patch. We may also want to consider distributing a trusted CA store with Mercurial. Although we should think long and hard about that because most systems have a global CA store and Mercurial should almost certainly use the same store used by everything else on the system.

sslutil: remove out of place comment This comment likely got orphaned as a result of refactoring in this file. It isn't providing any useful value. So delete it.

largefiles: remove additional blank line between methods in localstore According to the coding style it should be a single blank line between functions.

revset: make head() honor order of subset The ordering of 'x & head()' was broken in 6a1a4c212d50 (revset: improve head revset performance, 2014-03-13). Presumably due to other optimizations since then, undoing that change to fix the order does not slow down the simple case of "hg log -r 'head()'" mentioned in that commit. I see a small slowdown from ~0.16s to about ~0.19s with 'not 0 & head()', but I'd say it's worth it for the correct output.

revsets: use itervalues() where only values are needed I don't think there will be a noticeable speedup, but it removes an unused variable.

revsets: passing a set to baseset() is not wrong Since 69c6e9623bdc (revset: force ascending order for baseset initialized from a set, 2016-04-04), it is safe to pass a revset to a baseset.

pyflakes: use pycompat.pickles to prevent error The pyflakes in my test box complain about pickle in pycompat. mercurial/pycompat.py:17: 'pickle' imported but unused