sslutil: display a better error message when CA file loading fails Before, sslcontext.load_verify_locations() would raise a ssl.SSLError which would be caught further up the stack and converted to a urlerror. By that time, we lost track of what actually errored. Trapping the error here gives users a slightly more actionable error message. The behavior between Python <2.7.9 and Python 2.7.9+ differs. This is because our fake SSLContext class installed on <2.7.9 doesn't actually do anything during load_verify_locations: it defers actions until wrap_socket() time. Unfortunately, a number of errors can occur at wrap_socket() time and we're unable to ascertain what the root cause is. But that shouldn't stop us from providing better error messages to people running a modern and secure Python version.

tests: add test for empty CA certs file smf reported that an environment with no loaded CA certs resulted in a weird error. I'd like to detect this a bit better so we can display an actionable error message. The actual error being globbed over in this patch is "unknown error" with a ssl.c line number. That isn't useful at all.

internals: move the bitmanipulation routines into its own file This is to allow more flexibility with the C sources -- now the bitmanipulation routines can be safely imported without importing Python.h

journal: new experimental extension Records bookmark locations and shows you where bookmarks were located in the past. This is the first in a planned series of locations to be recorded; a future patch will add working copy (dirstate) tracking, and remote bookmarks will be supported as well, so the journal storage format should be fairly generic to support those use-cases.

httpclient: update to 54868ef054d2 of httpplus As of that revision, httpplus fully supports Python 3, including mimicing all the subtle behavior changes around headers in Python 3's http.client.

revset: check invalid function syntax "func-name"() explicitly Before the error was caught at func() as an unknown identifier, and the optimizer failed to detect the syntax error. This patch introduces getsymbol() helper to ensure that a string is not allowed as a function name.

chg: silence warning of unused parameter 'sig'

hgweb: reindent atom/changelogentry.tmpl It was mixing tabs and spaces, and not in a good way. Indent style of other atom entries seems to be 1 space per level, so let's apply it here as well.

hgweb: remove unused/nonexistent filelogentry from atom/map filelog in atom style uses changelogentry to show commits, and atom/filelogentry.tmpl doesn't even exist.

hgweb: remove unnecessary CDATA markup in atom/branchentry.tmpl Since content is of type "text" (and is already escaped), using a CDATA section is not required. Looks like this was just an artifact of copying things from rss style in add2f9ddcfb5, because other entries in atom style don't use CDATA in such places.

hgweb: reindent rss/changelogentry.tmpl It was mixing tabs and spaces, and not in a good way. Indent style of other rss entries seems to be 4 spaces per level, so let's apply it here as well.

hgweb: remove a couple of superfluous spaces in rss style

py3: add tests in check-code to load modules from util.py The conditionalize imports are added in util.py and now we import modules from there. So adding tests so that someone in future can use that.

py3: conditionalize SocketServer import The SocketServer is renamed to socketserver in python 3

py3: conditionalize xmlrpclib import The xmlrpclib library is renamed to xmlrpc.client in python 3

py3: conditionalize the urlparse import The urlparse library is renamed to urllib.parse in python 3

py3: update tests/test-check-py3-compat.t The lower part of the test runs with python 3 and hence remain unchanged.

chg: send SIGPIPE to server immediately when pager exits (issue5278) If the user press 'q' to leave the 'less' pager, it is expected to end the hg process immediately. We currently rely on SIGPIPE for this behavior. But SIGPIPE won't arrive if we don't write anything (like doing heavy computation, reading from network etc). If that happens, the user will feel that the hg process just hangs. The patch address the issue by adding a SIGCHLD signal handler and sends SIGPIPE to the server as soon as the pager exits. This is also an issue with hg's pager implementation.

chgserver: do not ignore SIGPIPE if pager is used We rely on SIGPIPE to exit when the pager exits. And Python ignores SIGPIPE by default. Explicitly set SIGPIPE handler to SIG_DFL (terminate) just like pager.py.

debug: make debug{revlog,index,data} --dir not just a flag The directory argument (for tree manifests) should belong to to the --dir argument. I had mistakenly made --dir a flag. One effect of this was that I had meant for "-m" to be optional, but instead it changed the behavior of --dir, so with "hg debugdata -m --dir dir1 0", the -m took over and the "dir1" got treated as a revision in the root manifest log.

debugdata: disallow trailing option with -c/-m Before this change, "hg debugdata -c 0 foo" was allowed.

revset: get rid of redundant error checking from match() Actually there was no additional error checking. It should be caught by "not all(specs)".

share: move magic string to a constant

branchmap: remove unused exception variable

patch: allow copy information to be passed in When displaying patches from graphical tools where you can browse through individual files, with diff being called separately on each, recomputing the limits of file copy history can become rather expensive on large repositories. Instead, we can compute it once and pass it in for subsequent calls.

largefiles: check file in the repo store before checking remotely (issue5257) Problem was files to check were gathered in the repository where the verify was launched but verification was done on the remote store. It was observed when user committed in cloned repository and ran verify before pushing - committed files were marked as non existing. This commit fixes this by checking in the remote store only files that are not existing in the repository store where verify was launched. Solution is similiar to fd288d118074

largefiles: remove additional blank lines It does not conform to the coding style.

largefiles: fix misleading comments in lfutil instore and storepath Problem in both cases is cache in largefiles has assigned meaning - user cache which is additional place to get/put files. Those two function works on store - the main place to store largefiles in the repository - .hg/largefiles and using "cache" to describe it is misleading.

revset: implement match() in terms of matchany() match() is the special case of a single element list being passed to matchany() with the additional error checking that the revset spec is defined. Change the implementation to remove the redundant code and have match() call matchany().

scmutil: improve documentation of revset APIs I can never remember the differences between the various revset APIs. I can never remember that scmutil.revrange() is the one I want to use from user-facing commands. Add some documentation to clarify this. While we're here, the argument name for revrange() is changed to "specs" because that's what it actually is.

mdiff: remove use of __slots__ The use of __slots__ was added way back in 2006 in 4ec58b157265. __slots__ isn't necessary for this class.

i18n: use unicode literal Other parts of this expression are already using unicode literals. We need this to make Python 3 happy and to avoid an implicit conversion in Python 2.

pycompat: add HTTPPasswordMgrWithDefaultRealm to Python 3 block Looks like we missed this in 800ec7c048b0.

ui: path option to declare which revisions to push by default Now that we have a mechanism for declaring path sub-options, we can start to pile on features! Many power users have expressed frustration that bare `hg push` attempts to push all local revisions to the remote. This patch introduces the "pushrev" path sub-option to control which revisions are pushed when no "-r" argument is specified. The value of this sub-option is a revset, naturally. A future feature addition could potentially introduce a "pushnames" sub-options that declares the list of names (branches, bookmarks, topics, etc) to push by default. The entire "what to push by default" feature should probably be considered before this patch lands.

ui: don't fixup [paths] sub-options As part of developing a subsequent patch I discovered that sub-option values like "." were getting converted to paths. This is because the [paths] section is treated specially during config loading. This patch prevents post-processing sub-options from the [paths] section.

sslutil: abort when unable to verify peer connection (BC) Previously, when we connected to a server and were unable to verify its certificate against a trusted certificate authority we would issue a warning and continue to connect. This is obviously not great behavior because the x509 certificate model is based upon trust of specific CAs. Failure to enforce that trust erodes security. This behavior was defined several years ago when Python did not support loading the system trusted CA store (Python 2.7.9's backports of Python 3's improvements to the "ssl" module enabled this). This commit changes behavior when connecting to abort if the peer certificate can't be validated. With an empty/default Mercurial configuration, the peer certificate can be validated if Python is able to load the system trusted CA store. Environments able to load the system trusted CA store include: * Python 2.7.9+ on most platforms and installations * Python 2.7 distributions with a modern ssl module (e.g. RHEL7's patched 2.7.5 package) * Python shipped on OS X Environments unable to load the system trusted CA store include: * Python 2.6 * Python 2.7 on many existing Linux installs (because they don't ship 2.7.9+ or haven't backported modern ssl module) * Python 2.7.9+ on some installs where Python is unable to locate the system CA store (this is hopefully rare) Users of these Pythongs will need to configure Mercurial to load the system CA store using web.cacerts. This should ideally be performed by packagers (by setting web.cacerts in the global/system hgrc file). Where Mercurial packagers aren't setting this, the linked URL in the new abort message can contain instructions for users. In the future, we may want to add more code for finding the system CA store. For example, many Linux distributions have the CA store at well-known locations (such as /etc/ssl/certs/ca-certificates.crt in the case of Ubuntu). This will enable CA loading to "just work" on more Python configurations and will be best for our users since they won't have to change anything after upgrading to a Mercurial with this patch. We may also want to consider distributing a trusted CA store with Mercurial. Although we should think long and hard about that because most systems have a global CA store and Mercurial should almost certainly use the same store used by everything else on the system.

sslutil: remove out of place comment This comment likely got orphaned as a result of refactoring in this file. It isn't providing any useful value. So delete it.

largefiles: remove additional blank line between methods in localstore According to the coding style it should be a single blank line between functions.

revset: make head() honor order of subset The ordering of 'x & head()' was broken in 6a1a4c212d50 (revset: improve head revset performance, 2014-03-13). Presumably due to other optimizations since then, undoing that change to fix the order does not slow down the simple case of "hg log -r 'head()'" mentioned in that commit. I see a small slowdown from ~0.16s to about ~0.19s with 'not 0 & head()', but I'd say it's worth it for the correct output.

revsets: use itervalues() where only values are needed I don't think there will be a noticeable speedup, but it removes an unused variable.

revsets: passing a set to baseset() is not wrong Since 69c6e9623bdc (revset: force ascending order for baseset initialized from a set, 2016-04-04), it is safe to pass a revset to a baseset.

pyflakes: use pycompat.pickles to prevent error The pyflakes in my test box complain about pickle in pycompat. mercurial/pycompat.py:17: 'pickle' imported but unused

rebase: move local variable 'obsoletenotrebased' to the RR class

rebase: move restorestestatus function to be a method of the RR class

rebase: move local variables related to keeping things unchanged to the RR This commit moves the following variables, local to the rebase function to be fields of the rebaseruntime: -keepf -keepbranchesf -keepopen

rebase: move local variables 'date' and 'extrafns' to the RR class This commit moves the following variables, local to the rebase function to be fields of the rebaseruntime: -date -extrafns

rebase: move collapse-related local variables to the RR class This commit moves the following variables local to the 'rebase' function to be fields of the rebaseruntime class: -collapsef -collapsemsg

rebase: pass repo, ui and opts objects to the RR class constructor

check-code: build translation table for repquote in global for efficiency Rebuilding translation table (256 size) at each repquote() invocations is redundant. For example, this patch decreases user time of command invocation below from 18.297s to 13.445s (about -27%) on a Linux box. This command is main part of test-check-code.t. hg locate | xargs python contrib/check-code.py --warnings --per-file=0 This patch adds "_repquote" prefix to functions and variables factored out from repquote() to avoid conflict of name in the future.

check-code: detect "missing _() in ui message" more exactly Before this patch, "missing _() in ui message" rule overlooks translatable message, which starts with other than alphabet. To detect "missing _() in ui message" more exactly, this patch improves the regexp with assumptions below. - sequence consisting of below might precede "translatable message" in same string token - formatting string, which starts with '%' - escaped character, which starts with 'b' (as replacement of '\\'), or - characters other than '%', 'b' and 'x' (as replacement of alphabet) - any string tokens might precede a string token, which contains "translatable message" This patch builds an input file, which is used to examine "missing _() in ui message" detection, before '"$check_code" stringjoin.py' in test-contrib-check-code.t, because this reduces amount of change churn in subsequent patch. This patch also applies "()" instead of "_()" on messages below to hide false-positives: - messages for ui.debug() or debug commands/tools - contrib/debugshell.py - hgext/win32mbcs.py (ui.write() is used, though) - mercurial/commands.py - _debugchangegroup - debugindex - debuglocks - debugrevlog - debugrevspec - debugtemplate - untranslatable messages - doc/gendoc.py (ReST specific text) - hgext/hgk.py (permission string) - hgext/keyword.py (text written into configuration file) - mercurial/cmdutil.py (formatting strings for JSON)

revlog: add a fast path for "ambiguous identifier" Before fd1bb7c, if the C index.partialmatch raises RevlogError, the Python code raises "ambiguous identifier" error immediately, which is efficient. fd1bb7c took hidden revisions into consideration and forced the slow path enumerating the changelog to double-check hidden revisions. But it's not necessary if we know the revlog has no hidden revisions. This patch adds back the fast path for unfiltered revlogs.

import-checker: ensure cffi is always a system module I've had reports that this is not always happening, so whitelist it the way we whitelist other problem cases.

atomictempfile: add context manager support Close the file (moving it in place) on clean context exit, discard when there has been an exception.

atomictempfile: add read to the supported file operations

atomictempfile: remove test ordering These tests are independent and numbering only makes it harder to add more and logically group them.

atomictempfile: use a tempdir to keep the test environment clean Rather than pre-emptively delete a file, execute the test in a dedicated temporary directory that is removed after each test.

test-revset: show how inconsistent the ordering of compound expressions is This adds mostly broken tests that will be fixed by subsequent patches. We generally don't do that, but this patch series would be hard to review without a set of broken tests. Note that some tests pass thanks to the reordering problem in optimize(). For instance, '2:0 & _intlist(0 1 2)' doesn't fail because it is rewritten as '_intlist(0 1 2) & 2:0'.

i18n: translate abort messages I found a few places where message given to abort is not translated, I don't find any reason to not translate them.

hgweb: display blamed revision once per block in annotate view I.e. when a revision blames a block of source lines, only display the revision link on the first line of the block (this is identified by the "blockhead" key in annotate context). This addresses item "Visual grouping of changesets" of the blame improvements plan (https://www.mercurial-scm.org/wiki/BlamePlan) which states: "Typically there are block of lines all attributed to the same revision. Instead of rendering the revision/changeset for every line, we could only render it once per block."

hgweb: highlight data of the current revision in annotate view * Distinguish the /annotate/<revision>/<file>#<linenumber> link when it would lead to the current page (i.e. <revision> is the current revision) (style it gray and undecorated). This indicates more clearly that this is a "dead-end" in blame navigation. * Display lines changed in current revision in green.

bashcompletion: show available command-line switches for aliases When auto-completing hg commands, aliases are listed, but not the available switches for an alias, because `HGPLAIN=1` filters these out. Add a `HGPLAINEXCEPT=alias` exception to resolve this. We make heavy use of aliases that drive hg log with custom revsets, sorting and the -G switch, but want our users to be able to auto-complete any additional command-line switches.

py3: shift from __future__ import absolute import to beginning (issue5269)

pull: add help information about pulling active bookmark

templates: add support for search webcommand in json style

templates: add support for summary webcommand in json style Change summary webcommand to yield each element of the shortlog instead of the entire list. This makes generated json more readable since each entry can be formatted separately, instead of returning all the shortlog content in a single string.

templates: add support for filerevision webcommand in json style

templates: add support for filelog webcommand in json style Modify changelistentry structure to also deliver phase and branch data and use either 'parents' or 'allparents' depending on what is defined in the view, in order to reuse it in filelog structure.

largefiles: make cloning not ask two times about password (issue4883) Before this commit url.opener overwritten stored password for connection with given url/user even when new password for given connection was not filled. This commit makes opener overwrites saved authentication only when it contains password.

url: remember http password database in ui object This makes http password database stored in ui object. It allows reusing authentication information when we use this database for creating password manager for the new connection.

url: extract password database from password manager So far password manager was keeping authentication information so opening new connection and creating new password manager made all saved authentication information lost. This commit separates password manager and password database to make it possible to reuse saved authentication information. This commit violates code checker because it adds add_password method (name with underscore) to passwordmgr object to provide method required by urllib2.

bookmarks: add 'hg pull -B .' for pulling the active bookmark (issue5258)

demandimport: delay loading for "from a import b" with absolute_import Before this patch, "from a import b" doesn't delay loading module "b", if absolute_import is enabled, even though "from . import b" does. For example: - it is assumed that extension X has "from P import M" for module M under package P with absolute_import feature - if importing module M is already delayed before loading extension X, loading module M in extension X is delayed until actually referring util, cmdutil, scmutil or so of Mercurial itself should be imported by "from . import M" style before loading extension X - otherwise, module M is loaded immediately at loading extension X, even if extension X itself isn't used at that "hg" command invocation Some minor modules (e.g. filemerge or so) of Mercurial itself aren't imported by "from . import M" style before loading extension X. And of course, external libraries aren't, too. This might cause startup performance problem of hg command, because many bundled extensions already enable absolute_import feature. To delay loading module for "from a import b" with absolute_import feature, this patch does below in "from a (or .a) import b" with absolute_import case: 1. import root module of "name" by system built-in __import__ (referred as _origimport) 2. recurse down the module chain for hierarchical "name" This logic can be shared with non absolute_import case. Therefore, this patch also centralizes it into chainmodules(). 3. and fall through to process elements in "fromlist" for the leaf module of "name" Processing elements in "fromlist" is executed in the code path after "if _pypy: .... else: ..." clause. Therefore, this patch replaces "if _pypy:" with "elif _pypy:" to share it. At 4f1144c3c72b introducing original "work around" for "from a import b" case, elements in "fromlist" were imported with "level=level". But "level" might be grater than 1 (e.g. level=2 in "from .. import b" case) at demandimport() invocation, and importing direct sub-module in "fromlist" with level grater than 1 causes unexpected result. IMHO, this seems main reason of "errors for unknown reason" described in 4f1144c3c72b, and we don't have to worry about it, because this issue was already fixed by 78d05778907b. This is reason why this patch removes "errors for unknown reasons" comment.

import-checker: increase portability for python 2.6.x Before this patch, fromlocalfunc() assumes that "module" attribute of ast.ImportFrom is None for "from . import a", and Python 2.7.x satisfies this assumption. On the other hand, with Python 2.6.x, "module" attribute of ast.ImportFrom is an empty string for "from . import a", and this causes failure of test-check-module-imports.t.

scmutil: allow access to filecache descriptor on class To make it easier to patch the wrapped function, make it possible to access the filecache descriptor directly on the class (rather than have to use ClassObject.__dict__['attributename']). Returning `self` when the first argument to `__get__` is `None` makes the descriptor behave the same way `property` objects do.

rebase: do not abort if all changesets have equivalents in the destination

changegroup: don't send empty subdirectory manifest groups When grafting/rebasing, it is common for multiple changesets to make the same change to a subdirectory. When writing the revlog for the directory, the revlog code already takes care of not writing the entry again. In 0c2a088ffcc5 (changegroup: prune subdirectory dirlogs too, 2016-02-12), I added the corresponding code in changegroup (not sending entries the client already has), but I forgot to avoid sending the entire changegroup if no nodes remained in the pruned set. Although that's harmless besides the wasted network traffic, the receiving side was checking for it (copied from the changegroup code for handling files). This resulted in the client crashing with: abort: received dir revlog group is empty Fix by simply not emitting a changegroup for the directory if there were no changes is it. This matches how files are handled.

chg: ignore SIGINT while waiting pager termination Otherwise the terminal would be left with unclean state. This is what fcc4b55876c3 does.

chg: reset signal handlers to default before waiting pager Our signal handlers forward signals to the server process, but it will disappear soon after hgc_close(). So we should unregister handlers before hgc_close(). Otherwise chg would abort due to kill(perrpid, sig) failure. The problem is spotted by SIGWINCH while waiting pager termination.

help: document that [subpaths] may rewrite relative paths The subpaths substitution logic first attempts to match the absolute repository path, then the relative subrepository path if that failed.

doc: describe detail about checkambig optional argument This is followup for patches below, which add checkambig argument to existing function. - 731ced087a4b - 76f1ea360c7e - ce2d81aafbae - a109bf7e0dc2

ui: provide official way to reset internal state per command This will allow us to clear in-memory password storage per runcommand(). I've updated commandserver to call resetstate() of both ui and repo.ui because they may have different states in theory.

revset: extract function that validates sort() arguments This function will be used in _optimize() to get rid of noop sort() call while validating its arguments.

revset: build dict of extra sort options before evaluating set Prepares for extracting a function that only validates sort options.

revset: build list of (key, reverse) pairs before sorting Prepares for extracting a function that only validates sort options.

revset: fix crash on empty sort key Make it noop as before 2188f170f5b6. We could change it to an error, but allowing empty key makes some sense for scripting that builds a key string programmatically.

rebase: move local variable 'targetancestors' to the RR class

rebase: move local variable 'skipped' to the RR class

rebase: move local variable 'target' to the RR class

rebase: introduce a rebaseruntime (RR) class rebaseruntime is a class that will in future contain all of the state necessary to perform rebase operation and have pieces of rebase logic as its methods. This commit introduces the class and moves the following local variables to be its fields: - originalwd - external - state - activebookmark

chg: change default connect timeout to 60 seconds As discussed at https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-June/085290.html The default 10-second timeout is not enough if the machine is overloaded. Let's increase it to 60 seconds.

tests: increase test-https malform error glob The recently introduced (ecc9b788fd690a0a) test around malformed pem files hard codes an error message which doesn't appear to be cross platform agnostic. On our machines (centos6 if it matters) the test output differs: - abort: error: unknown error* (glob) + abort: error: _ssl.c:330: error:00000000:lib(0):func(0):reason(0) This patch increases the glob to cover the entire error message.

largefiles: make storefactory._openstore public In storefactory opening store is the main functionality, so it shouldn't be marked as private with underscore.

bookmarks: abort 'push -B .' when no active bookmark

transaction: avoid ambiguity of file stat at restoring from backup In some cases below, copying from backup is used to restore original contents of a file, which is backuped via addfilegenerator(). If copying keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - failure of transaction (from '.hg/journal.backup.*') - rollback of previous transaction (from '.hg/undo.backup.*') To avoid ambiguity of file stat at restoring, this patch invokes util.copyfile() with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

localrepo: make restoring from backup at rollback avoid ambiguity of file stat Rollback of previous transaction restores contents of files below by renaming from 'undo.*' file. If renaming keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - .hg/bookmarks - .hg/phaseroots To avoid ambiguity of file stat at restoring, this patch invokes vfs.rename() with checkambig=True. BTW, .hg/dirstate is also restored at rollback. But it is restored by dirstate.restorebackup(), and previous patch already made it invoke vfs.rename() with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

dirstate: make restoring from backup avoid ambiguity of file stat File .hg/dirstate is restored by renaming from backup in failure inside scopes below. If renaming keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - dirstateguard scope (from '.hg/dirstate.SUFFIX') - transaction scope (from '.hg/journal.dirstate') To avoid ambiguity of file stat at restoring, this patch invokes vfs.rename() with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

tests: drop a duplicated instruction

merge with stable

revset: add new topographical sort Sort revisions in reverse revision order but grouped by topographical branches. Visualised as a graph, instead of: o 4 | | o 3 | | | o 2 | | o | 1 |/ o 0 revisions on a 'main' branch are emitted before 'side' branches: o 4 | o 1 | | o 3 | | | o 2 |/ o 0 where what constitutes a 'main' branch is configurable, so the sort could also result in: o 3 | o 2 | | o 4 | | | o 1 |/ o 0 This sort was already available as an experimental option in the graphmod module, from which it is now removed. This sort is best used with hg log -G: $ hg log -G "sort(all(), topo)"

revset: move groupbranchiter over from graphmod This move is to prepare the adaptation of this function into a toposort predicate.

revset: record if a set is in topographical order A later revision adds actual topographical sorting. Recording if a set is in this order allows hg log -G to avoid re-sorting the revset.

chg: make timeout adjustable Before this patch, chg will give up when it cannot connect to the new server within 10 seconds. If the host has high load during that time, 10 seconds is not enough. This patch makes it adjustable using the CHGTIMEOUT environment variable.

chg: exec pager in child process Before this patch, chg uses the old pager behavior (pre 369741ef7253), which executes pager in the main process. The user will see the exit code of the pager, instead of the hg command. Like 369741ef7253, this patch fixes the behavior by executing the pager in the child process, and wait for it at the end of the main process.

tests: move chg pager test to test-pager.t The test is valid for both hg and chg. Since we are adding another chg-related pager test, let's put them together.

util: drop local aliases for md5, sha1, sha256, and sha512 This used to be needed to paper over hashlib not being in all Pythons we support, but that's not a problem anymore, so we can simplify things a little bit.

cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1 All versions of Python we support or hope to support make the hash functions available in the same way under the same name, so we may as well drop the util forwards.

pathencode: use hashlib.sha1 directly instead of indirecting through util

revlog: use hashlib.sha1 directly instead of through util Also remove module-local _sha alias, which was barely used.

store: use hashlib.sha1 directly instead of through util Also remove module-local alias to _sha, since it's not used that much.

similar: delete extra newline at EOF Spotted by my emacs config that cleans up extra whitespace.

scmutil: delete extra newline at EOF Spotted by my emacs config that cleans up extra whitespace.

graphmod: avoid sorting when already sorted This is somewhat redundant now, but allows us to add a toposort that should not be re-sorted either.

sslutil: per-host config option to define certificates Recent work has introduced the [hostsecurity] config section for defining per-host security settings. This patch builds on top of this foundation and implements the ability to define a per-host path to a file containing certificates used for verifying the server certificate. It is logically a per-host web.cacerts setting. This patch also introduces a warning when both per-host certificates and fingerprints are defined. These are mutually exclusive for host verification and I think the user should be alerted when security settings are ambiguous because, well, security is important. Tests validating the new behavior have been added. I decided against putting "ca" in the option name because a non-CA certificate can be specified and used to validate the server certificate (commonly this will be the exact public certificate used by the server). It's worth noting that the underlying Python API used is load_verify_locations(cafile=X) and it calls into OpenSSL's SSL_CTX_load_verify_locations(). Even OpenSSL's documentation seems to omit that the file can contain a non-CA certificate if it matches the server's certificate exactly. I thought a CA certificate was a special kind of x509 certificate. Perhaps I'm wrong and any x509 certificate can be used as a CA certificate [as far as OpenSSL is concerned]. In any case, I thought it best to drop "ca" from the name because this reflects reality.

tests: add basic tests for SMTP over SSL SSL handling in mail.py wasn't covered by our test suite, therefore it was sometimes broken. This patch introduces pretty minimal tests that only cover the default path. We can extend it later. Tested with python 2.6.9 and 2.7.11 on Debian sid.

tests: add dummy SMTP daemon for SSL tests Currently it only supports SMTP over SSL since SMTPS should be simpler than handling StartTLS. Since we don't need asynchronous server for our tests, it does TLS handshake in blocking way. But asyncore is required by Python smtpd module.

tests: extract SSL certificates from test-https.t They can be reused in SMTPS tests.

check-code: make 'ls' pattern less invasive I got false positive at "--tls smtps --certificate ...".

largefiles: fix support for local largefiles while using share extension Prior to revision 2a3f24786d09, largefiles were saved in the local repository, even if it was using the share extension. After that change, all largefiles are now stored in the shared repository. However, the backward compatibility for existing largefiles already placed in the local repository was never tested, and has been broken since.

merge with stable

crecord: drop unused "operation" parameter from filterpatch function

patch: define full messages for interactive record/revert Followup 14eee72c8d52 to provide complete context for proper localization. Also update cmdutil.recordfilter docstring to remove recommendation that "operation" argument should be translated. Indeed, for record/revert, we either go to patch.filterpatch or crecord.filterpatch (in curses mode) ; the former now build the full ui message from the operation parameter and the latter does not use this parameter (removing in a followup patch). For shelve, operation is not specified and this thus falls back to "record".

hgweb: remove unused code in annotate web command

py3: conditionalize cPickle import by adding in util The cPickle is renamed to _pickle in python3 and this C extension is available in pickle which was not included in earlier versions. So imports are conditionalized to import cPickle in py2 and pickle in py3. Moreover the use of pickle in py2 is switched to cPickle as the C extension is faster. The hack is added in util.py and the modules import util.pickle

bdiff: remove effectively dead code Now that we extend matches backwards in the inner loop, the final adjustment has no effect. (A similar extension for the forward direction is trickier and has less benefit.)

bdiff: extend matches across popular lines For very large diffs that have large numbers of identical lines (JSON dumps) that also have large blocks of identical text, bdiff could become confused about which block matches which because it can only match very limited regions. The result is very large diffs for small sets of edits. The earlier recursion rebalancing fix made this behavior more frequent because it's now more prone to match block 1 to block 2. One frequent user of large JSON files reported being unable to pass the resulting diffs through their code review system. Prior to this change, bdiff would calculate the length of a match at (i, j) as 1 + length found at (i-1, j-1). With large number of popular (ignored) lines, this often meant matches couldn't be extended backwards at all and thus all matching regions were very small. Disabling the popularity threshold is not an option because it brings back quadratic behavior. Instead, we extend a match backwards until we either found a previously discovered match or we find a mismatching line. This thus successfully bridges over any popular lines inside and before a matching region. The larger regions then significant reduce the probability of confusion.

test-revset: fix test vector for ordering issue of matching() 592e0beee8b0 fixed matching() to preserve the order of the input set, but the test was incorrect. Given "A and B", "A" should be the input set to "B". But thanks to our optimizer, the test expression was rewritten as "(2 or 3 or 1) and matching(1 or 2 or 3)", therefore it was working well. Since I'm going to fix the overall ordering issue, the test needs to be adjusted to do the right thing.

largefiles: rename match_ to matchmod import in lfutil

largefiles: rename match_ to matchmod import in reposetup

largefiles: rename match_ to matchmod import in overrides

largefiles: rename match_ to matchmod import in lfcommands

py3: make largefiles/wirestore.py use absolute_import

py3: make largefiles/uisetup.py use absolute_import

py3: make largefiles/reposetup.py use absolute_import

py3: make largefiles/remotestore.py use absolute_import

py3: make largefiles/proto.py use absolute_import

py3: make largefiles/overrides.py use absolute_import

py3: make largefiles/localstore.py use absolute_import

py3: make largefiles/lfutil.py use absolute_import

py3: make largefiles/lfcommands.py use absolute_import

py3: make largefiles/basestore.py use absolute_import

py3: make largefiles/__init__.py use absolute_import

largefiles: move basestore._openstore into new module to remove cycle

revset: make filteredset.__nonzero__ respect the order of the filteredset This fix allows __nonzero__ to respect the direction of iteration of the whole filteredset. Here's the case when it matters. Imagine that we have a very large repository and we want to execute a command like: $ hg log --rev '(tip:0) and user(ikostia)' --limit 1 (we want to get the latest commit by me). Mercurial will evaluate a filteredset lazy data structure, an instance of the filteredset class, which will know that it has to iterate in a descending order (isdescending() will return True if called). This means that when some code iterates over the instance of this filteredset, the 'and user(ikostia)' condition will be first checked on the latest revision, then on the second latest and so on, allowing Mercurial to print matches as it founds them. However, cmdutil.getgraphlogrevs contains the following code: revs = _logrevs(repo, opts) if not revs: return revset.baseset(), None, None The "not revs" expression is evaluated by calling filteredset.__nonzero__, which in its current implementation will try to iterate the filteredset in ascending order until it finds a revision that matches the 'and user(..' condition. If the condition is only true on late revisions, a lot of useless iterations will be done. These iterations could be avoided if __nonzero__ followed the order of the filteredset, which in my opinion is a sensible thing to do here. The problem gets even worse when instead of 'user(ikostia)' some more expensive check is performed, like grepping the commit diff. I tested this fix on a very large repo where tip is my commit and my very first commit comes fairly late in the revision history. Results of timing of the above command on that very large repo. -with my fix: real 0m1.795s user 0m1.657s sys 0m0.135s -without my fix: real 1m29.245s user 1m28.223s sys 0m0.929s I understand that this is a very specific kind of problem that presents itself very rarely, only on very big repositories and with expensive checks and so on. But I don't see any disadvantages to this kind of fix either.

phases: make writing phaseroots file out avoid ambiguity of file stat Cached attribute repo._phasecache uses stat of '.hg/phaseroots' file to examine validity of cached contents. If writing '.hg/phaseroots' file out keeps ctime, mtime and size of it, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/phaseroots' file out with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

dirstate: make writing branch file out avoid ambiguity of file stat Cached attribute dirstate._branch uses stat of '.hg/branch' file to examine validity of cached contents. If writing '.hg/branch' file out keeps ctime, mtime and size of it, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/branch' file out with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

dirstate: make writing dirstate file out avoid ambiguity of file stat Cached attribute repo.dirstate uses stat of '.hg/dirstate' file to examine validity of cached contents. If writing '.hg/dirstate' file out keeps ctime, mtime and size of it, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/dirstate' file out with checkambig=True. The former diff hunk changes the code path for "dirstate.write()", and the latter changes the code path for "dirstate.savebackup()". This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

bookmarks: make writing files out avoid ambiguity of file stat Cached attribute repo._bookmarks uses stat of '.hg/bookmarks' and '.hg/bookmarks.current' files to examine validity of cached contents. If writing these files out keeps ctime, mtime and size of them, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/bookmarks' and '.hg/bookmarks.current' files out with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

transaction: avoid ambiguity of file stat at closing transaction Files below, which might be changed at closing transaction, are used to examine validity of cached properties. If changing keeps ctime, mtime and size of a file, change is overlooked, and old contents cached before change isn't invalidated as expected. - .hg/bookmarks - .hg/dirstate - .hg/phaseroots To avoid ambiguity of file stat, this patch writes files out with checkambig=True at closing transaction. checkambig becomes True only at closing (= 'not suffix'), because stat information of '.pending' file isn't used to examine validity of cached properties. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: add __ne__ to filestat class for consistency This is follow up for ca4065028e00, which introduced filestat class.

style: remove namespace class For better or worse, our coding do not use use class for pure namespacing. We remove the class introduced in a5009789960c.

style: don't use capital letter for constant For better or worse, our coding do not use all caps for constants. We rename constant name introduced in a5009789960c.

tests-subrepo-git: use "f" to dump pwned.txt, for portability Rather than sometimes using a complicated shell construct to dump pwned.txt (if it wasn't expected to exist, but might, if something were broken) or just cat (if it was expected to exist), just use the "f" utility, which will be consistent in its behavior across different platforms. Also make sure that *something* gets put into pwned.txt, even if we ended up typoing the message variable.

bundle2: don't assume ordering of heads checked after push Usually, the heads will have the same ordering in handlecheckheads. Insisting on the same ordering is however an unnecessary constraint that in some custom cases can cause pushes to fail even though the actual heads didn't change. This caused production issues for us in combination with the current version of https://bitbucket.org/Unity-Technologies/hgwebcachingproxy/ .

sslutil: print the fingerprint from the last hash used Before, we would always print the unprefixed SHA-1 fingerprint when fingerprint comparison failed. Now, we print the fingerprint of the last hash used, including the prefix if necessary. This helps ensure that the printed hash type matches what is in the user configuration. There are still some cases where this can print a mismatched hash type. e.g. if there are both SHA-1 and SHA-256 fingerprints in the config, we could print a SHA-1 hash if it comes after the SHA-256 hash. But I'm inclined to ignore this edge case. While I was here, the "section" variable assignment has been moved to just above where it is used because it is now only needed for this error message and it makes the code easier to read.

sslutil: make cert fingerprints messages more actionable The previous warning and abort messages were difficult to understand. This patch makes them slightly better. I think there is still room to tweak the messaging. And as we adopt new security defaults, these messages will certainly change again. But at least this takes us a step in the right direction. References to "section" have been removed because if no fingerprint is defined, "section" can never be "hostfingerprints." So just print "hostsecurity" every time.

sslutil: refactor code for fingerprint matching We didn't need to use a temporary variable to indicate success because we just return anyway. This refactor makes the code simpler. While we're here, we also call into formatfingerprint() to ensure the fingerprint from the proper hashing algorithm is logged.

sslutil: print SHA-256 fingerprint by default The world is starting to move on from SHA-1. A few commits ago, we gained the ability to define certificate fingerprints using SHA-256 and SHA-512. Let's start printing the SHA-256 fingerprint instead of the SHA-1 fingerprint to encourage people to pin with a more secure hashing algorithm. There is still a bit of work to be done around the fingerprint messaging. This will be addressed in subsequent commits.

sslutil: move and change warning when cert verification is disabled A short time ago, validatesocket() didn't know the reasons why cert verification was disabled. Multiple code paths could lead to cert verification being disabled. e.g. --insecure and lack of loaded CAs. With the recent refactorings to sslutil.py, we now know the reasons behind security settings. This means we can recognize when the user requested security be disabled (as opposed to being unable to provide certificate verification due to lack of CAs). This patch moves the check for certificate verification being disabled and changes the wording to distinguish it from other states. The warning message is purposefully more dangerous sounding in order to help discourage people from disabling security outright. We may want to add a URL or hint to this message. I'm going to wait until additional changes to security defaults before committing to something.

sslutil: add devel.disableloaddefaultcerts to disable CA loading There are various tests for behavior when CA certs aren't loaded. Previously, we would pass --insecure to disable loading of CA certs. This has worked up to this point because the error message for --insecure and no CAs loaded is the same. Upcoming commits will change the error message for --insecure and will change behavior when CAs aren't loaded. This commit introduces the ability to disable loading of CA certs by setting devel.disableloaddefaultcerts. This allows a testing backdoor to disable loading of CA certs even if system/default CA certs are available. The flag is purposefully not exposed to end-users because there should not be a need for this in the wild: certificate pinning and --insecure provide workarounds to disable cert loading/validation. Tests have been updated to use the new method. The variable used to disable CA certs has been renamed because the method is not OS X specific.

sslutil: store flag for whether cert verification is disabled This patch effectively moves the ui.insecureconnections check to _hostsettings(). After this patch, validatesocket() no longer uses the ui instance for anything except writing messages. This patch also enables us to introduce a per-host config option for disabling certificate verification.

sslutil: remove "strict" argument from validatesocket() It was only used by mail.py as part of processing smtp.verifycert, which was just removed.

mail: unsupport smtp.verifycert (BC) smtp.verifycert was accidentally broken by cca59ef27e60. And, I believe the "loose" value has been broken for longer than that. The current code refuses to talk to a remote server unless the CA is trusted or the fingerprint is validated. In other words, we lost the ability for smtp.verifycert to lower/disable security. There are special considerations for smtp.verifycert in sslutil.validatesocket() (the "strict" argument). This violates the direction sslutil is evolving towards, which has all security options determined at wrapsocket() time and a unified code path and configs for determining security options. Since smtp.verifycert is broken and since we'll soon have new security defaults and new mechanisms for controlling host security, this patch formally deprecates smtp.verifycert. With this patch, the socket security code in mail.py now effectively mirrors code in url.py and other places we're doing socket security. For the record, removing smtp.verifycert because it was accidentally broken is a poor excuse to remove it. However, I would have done this anyway because smtp.verifycert is a one-off likely used by few people (users of the patchbomb extension) and I don't think the existence of this seldom-used one-off in security code can be justified, especially when you consider that better mechanisms are right around the corner.

update: fix bare --clean to work on new branch (issue5003) (BC) Before this commit bare update --clean on newly created branch updates to the parent commit, even if there are later commits on the parent commit's branch. Update to the latest head on the parent commit's branch instead. This seems reasonable as clean should discard uncommited changes, branch is one of them.

revert: use "discard"/"revert" verb when reverting interactively (issue5143) Instead of "record this change to 'FILE'?" now prompt with: * "discard this change to 'FILE'?" when reverting to the parent of working directory, and, * "revert this change to 'FILE'?" otherwise.

run-tests: add support for RTUNICODEPEDANTRY environment variable based on 73e4a02e6d23

obsolete: fix grammar

tests: add run-test .testtimes basic testing

check-code: make repquote distinguish more characters for exact detection This patch makes repquote() distinguish more characters below, as a preparation for exact detection in subsequent patch. - "%" as "%" - "\\" as "b"(ackslash) - "*" as "A"(sterisk) - "+" as "P"(lus) - "-" as "M"(inus) Characters other than "%" don't use itself as replacement, because they are treated as special ones in regexp.

check-code: centralize rules depending on implementation of repquote This decreases the cost of checking which regexp should be adjusted at change of repquote().

check-code: use fixedmap for replacement of space characters This can centralize management of fixed replacement into fixedmap.

check-code: replace quoted characters correctly 169cb9e47f8e tried to detect '.. note::' more exactly. But implementation of it seems not correct, because: - fromc.find(c) returns -1 for other than "." and ":" - tochr[-1] returns "q" for such characters, but - expected result for them is "o" This patch uses dict to manage replacement instead of replacing str.find() by str.index(), for improvement/refactoring in subsequent patches. Examination by fixedmap is placed just after examination for ' ' and '\n', because subsequent patch will integrate the latter into the former. This patch also changes regexp for 'string join across lines with no space' rule, and adds detailed test for it, because 169cb9e47f8e did: - make repquote() distinguish "." (as "p") and ":" (as "q") from others (as "o"), but - not change this regexp without any reason (in commit log, at least), even though this regexp depends on what "o" means This patch doesn't focuses on deciding whether "." and/or ":" should be followed by whitespace or not in translatable messages.

test-chg: add basic tests for server lifecycle I'm going to move around the codes in AutoExitMixIn. This test should catch a subtle bug of unlinking sockets which I made in draft patches.

test-chg: run only with chg It doesn't make sense to run test-chg.t without chg, so ignore it with vanilla hg, and specify chg executable explicitly. test-chg.t can host chg-specific tests.

distate: add assertions to backup functions Those assertions will prevent the backup functions from overwriting the dirstate file in case both: suffix and prefix are empty. (foozy suggested making that change and I agree with him)

Added signature for changeset a9764ab80e11

Added tag 3.8.3 for changeset a9764ab80e11

shelve: use backup functions instead of manually copying dirstate This increases encapsulation of dirstate: the dirstate file is private to the dirstate module and shouldn't be touched by extensions directly.

dirstate: don't use actualfilename to name the backup file The issue with using actualfilename is that dirstate saved during transaction with "pending" in filename will be impossible to recover from outside of the transaction because the recover method will be looking for the name without "pending".

sslutil: reference appropriate config section in messaging Error messages reference the config section defining the host fingerprint. Now that we have multiple sections where this config setting could live, we need to point the user at the appropriate one. We default to the new "hostsecurity" section. But we will still refer them to the "hostfingerprint" section if a value is defined there. There are some corner cases where the messaging might be off. e.g. they could define a SHA-1 fingerprint in both sections. IMO the messaging needs a massive overhaul. I plan to do this as part of future refactoring to security settings.

sslutil: allow fingerprints to be specified in [hostsecurity] We introduce the [hostsecurity] config section. It holds per-host security settings. Currently, the section only contains a "fingerprints" option, which behaves like [hostfingerprints] but supports specifying the hashing algorithm. There is still some follow-up work, such as changing some error messages.

debuginstall: expose modulepolicy With this, you can check for pure easily: $ HGMODULEPOLICY=py ./hg debuginstall -T "{hgmodulepolicy}" py

revset: define table of sort() key functions This should be more readable than big "if" branch.

revset: factor out reverse flag of sort() key Prepares for making a table of sort keys. This assumes 'k' has at least one character, which should be guaranteed by keys.split().

tests: don't save host fingerprints in hgrc Previously, the test saved the host fingerprints in hgrc. Many tests override the fingerprint at run-time. This was a bit dangerous and was too magical for my liking. It will also interfere with a future patch that adds a new source for obtaining fingerprints. So change the test to require the fingerprint on every command invocation.

sslutil: calculate host fingerprints from additional algorithms Currently, we only support defining host fingerprints with SHA-1. A future patch will introduce support for defining fingerprints using other hashing algorithms. In preparation for that, we rewrite the fingerprint verification code to support multiple fingerprints, namely SHA-256 and SHA-512 fingerprints. We still only display the SHA-1 fingerprint. We'll have to revisit this code once we support defining fingerprints with other hash functions. As part of this, I snuck in a change to use range() instead of xrange() because xrange() isn't necessary for such small values.

util: add sha256 Upcoming patches will teach host fingerprint checking to verify non-SHA1 fingerprints. Many x509 certificates these days are SHA-256. And modern browsers often display the SHA-256 fingerprint for certificates. Since SHA-256 fingerprints are highly visible and easy to obtain, we want to support them for fingerprint pinning. So add SHA-256 support to util. I did not add SHA-256 to DIGESTS and DIGESTS_BY_STRENGTH because this will advertise the algorithm on the wire protocol. I wasn't sure if that would be appropriate. I'm playing it safe by leaving it out for now.

sslutil: move CA file processing into _hostsettings() The CA file processing code has been moved from _determinecertoptions into _hostsettings(). As part of the move, the logic has been changed slightly and the "cacerts" variable has been renamed to "cafile" to match the argument used by SSLContext.load_verify_locations(). Since _determinecertoptions() no longer contains any meaningful code, it has been removed.

sslutil: move SSLContext.verify_mode value into _hostsettings _determinecertoptions() and _hostsettings() are redundant with each other. _hostsettings() is used the flexible API we want. We start the process of removing _determinecertoptions() by moving some of the logic for the verify_mode value into _hostsettings(). As part of this, _determinecertoptions() now takes a settings dict as its argument. This is technically API incompatible. But since _determinecertoptions() came into existence a few days ago as part of this release, I'm not flagging it as such.

sslutil: introduce a function for determining host-specific settings This patch marks the beginning of a series that introduces a new, more configurable, per-host security settings mechanism. Currently, we have global settings (like web.cacerts and the --insecure argument). We also have per-host settings via [hostfingerprints]. Global security settings are good for defaults, but they don't provide the amount of control often wanted. For example, an organization may want to require a particular CA is used for a particular hostname. [hostfingerprints] is nice. But it currently assumes SHA-1. Furthermore, there is no obvious place to put additional per-host settings. Subsequent patches will be introducing new mechanisms for defining security settings, some on a per-host basis. This commits starts the transition to that world by introducing the _hostsettings function. It takes a ui and hostname and returns a dict of security settings. Currently, it limits itself to returning host fingerprint info. We foreshadow the future support of non-SHA1 hashing algorithms for verifying the host fingerprint by making the "certfingerprints" key a list of tuples instead of a list of hashes. We add this dict to the hgstate property on the socket and use it during socket validation for checking fingerprints. There should be no change in behavior.

tests-subrepo-git: emit a different "pwned" message based on the test Having a single "pwned" message which may or may not be emitted during the tests for CVE-2016-3068 leads to extra confusion. Allow each test to emit a more detailed message based on what the expectations are. In both cases, we expect a version of git which has had the vulnerability plugged, as well as a version of mercurial which also knows about GIT_ALLOW_PROTOCOL. For the first test, we make sure GIT_ALLOW_PROTOCOL is unset, meaning that the ext-protocol subrepo should be ignored; if it isn't, there's either a problem with mercurial or the installed copy of git. For the second test, we explicitly allow ext-protocol subrepos, which means that the subrepo will be accessed and a message emitted confirming that this was, in fact, our intention.

tests-subrepo-git: make the "pwned" message output in a stable order The "pwned" message from this test gets gets sent to stderr, and so may get emitted in different places from run to run in the rest of mercurial's output. This patch forces the message to go to a specific file instead, whose existence and contents we can examine at a stable point in the test's execution.

test-cache-abuse: correct for different hunk headers between Solaris and GNU When diffing against an empty file, Solaris diff uses 1 to designate the first line of the empty file (either -1,0 on the left or +1,0 on the right) while GNU diff uses 0 (-0,0 and +0,0). We use a glob here to make sure the test passes with either toolchain. I've not added tests to check-code because there are scads of places in the tests where the GNU format is used due to that being the format that "hg diff" and "hg export" use, and changing those to use globs seems wrong.

lazymanifest: fix typo s/typles/tuples/

sslutil: remove sslkwargs() (API) It is now unused.

url: remove use of sslkwargs

mail: remove use of sslkwargs

httpconnection: remove use of sslkwargs It now does nothing.

sslutil: move sslkwargs logic into internal function (API) As the previous commit documented, sslkwargs() doesn't add any value since its return is treated as a black box and proxied to wrapsocket(). We formalize its uselessness by moving its logic into a new, internal function and make sslkwargs() return an empty dict. The certificate arguments that sslkwargs specified have been removed from wrapsocket() because they should no longer be set.

sslutil: remove ui from sslkwargs (API) Arguments to sslutil.wrapsocket() are partially determined by calling sslutil.sslkwargs(). This function receives a ui and a hostname and determines what settings, if any, need to be applied when the socket is wrapped. Both the ui and hostname are passed into wrapsocket(). The other arguments to wrapsocket() provided by sslkwargs() (ca_certs and cert_reqs) are not looked at or modified anywhere outside of sslutil.py. So, sslkwargs() doesn't need to exist as a separate public API called before wrapsocket(). This commit starts the process of removing external consumers of sslkwargs() by removing the "ui" key/argument from its return. All callers now pass the ui argument explicitly.

dirstate: remove file from copymap on drop As the copymap is short-lived object regenerated from dirstate on each read this didn't affect us in any serious way. But since I've started working on permanent storage of copymap in my experiments with sqldirstate[1] I've seen this bug leaving the copy information in copymap after reverting the file moves and copies. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

run-tests: use json.dumps(separators=) Followup to daff05dcd184 per Martijn Pieters

debugignore: make messages translatable These messages have been overlooked by check-code, because they start with non-alphabet character ('%' or '('). Making these messages translatable seems reasonable, because messages for ui.note(), ui.status(), ui.progress() and descriptive messages for ui.write() in "debug" commands are already translatable in many cases. This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

grep: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character (' '). This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

subrepo: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('%'). This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

merge: make messages translatable These messages have been overlooked by check-code, because they start with non-alphabet character (' '). Making these messages translatable seems reasonable, because all other 'ui.note()'-ed messages in calculateupdates() are already translatable. This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

httppeer: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('('). Making this message translatable seems reasonable, because exception message below in same function is already translatable - 'cannot create new http repository' This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

notify: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('\'). Making this message translatable seems reasonable, because messages below in same function are already translatable - '\ndiffs (truncated from %d to %d lines):\n\n' - '\ndiffs (%d lines):\n\n' This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

gpg: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('%'). This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

revset: use getargsdict for sort() This makes it possible to use keyword arguments to specify per-sort options. For example, a hypothetical 'first' option for the user sort could sort certain users first with: sort(all(), user, user.first=mpm@selenic.com)

merge with stable

changegroup: extract method that sorts nodes to send The current implementation of narrowhg needs to influence the order in which nodes are sent to the client. adgar@ and I think this is fixable, but it's going to require pretty substantial time investment, so in the interim we'd like to extract this method. I think it makes the group() code a little more obvious, as it took us a couple of tries to isolate the exact behavior we were observing.

hg: disable demandimport for py3

tests: enable import checker for all python files (including no .py files) i18n/posplit is excluded as it couldn't be trivially fixed. That's the same as 99a2bdad0fda.

tests: make 'f' utility import hashlib unconditionally It must exist on Python 2.5+.

tests: fix typo of shebang prefix in test-check-pyflakes.t

hghave: silence future pyflakes warning of unused import

tests: remove unused import from 'f' utility It should have been caught by pyflakes.

bookmarks: allow pushing active bookmark on new remote head (issue5236) Before 'hg push -B .' on new remote head complained with: abort: push creates new remote head ... It was because _nowarnheads was not expanding active bookmark name, so it didn't add active bookmark "proper" name to no warn heads list.

sslutil: remove redundant check of sslsocket.cipher() We are doing this check in both wrapsocket() and validatesocket(). The check was added to the validator in 4bb59919c905 and the commit message justifies the redundancy with a "might." The check in wrapsocket() was added in 0cc4ad757c77, which appears to be part of the same series. I'm going to argue the redundancy isn't needed. I choose to keep the check in wrapsocket() because it is working around a bug in Python's wrap_socket() and I feel the check for the bug should live next to the function call exhibiting the bug.

sslutil: convert socket validation from a class to a function (API) Now that the socket validator doesn't have any instance state, we can make it a generic function. The "validator" class has been converted into the "validatesocket" function and all consumers have been updated.

sslutil: store and use hostname and ui in socket instance Currently, we pass a hostname and ui to sslutil.wrap_socket() then create a separate sslutil.validator instance also from a hostname and ui. There is a 1:1 mapping between a wrapped socket and a validator instance. This commit lays the groundwork for making the validation function generic by storing the hostname and ui instance in the state dict attached to the socket instance and then using these variables in the validator function. Since the arguments to sslutil.validator.__init__ are no longer used, we make them optional and make __init__ a no-op.

sslutil: use a dict for hanging hg state off the wrapped socket I plan on introducing more state on the socket instance. Instead of using multiple variables, let's just use one to minimize risk of name collision.

sslutil: require serverhostname argument (API) All callers now specify it. So we can require it. Requiring the argument means SNI will always work if supported by Python. The main reason for this change is to store state on the socket instance to make the validation function generic. This will be evident in subsequent commits.

annotate: optimize line counting We used len(text.splitlines()) to count lines. This allocates, copies, and deallocates an object for every line in a file. Instead, we use count("\n") to count newlines and adjust based on whether there's a trailing newline. This improves the speed of annotating localrepo.py from 4.2 to 4.0 seconds.

purge: use opts.get() Most commands use opts.get() to retrieve values for options that may not be explicitly passed. purge wasn't. This makes it easier to call purge() from 3rd party extensions.

test-run-tests: clean up inuse server eagerly

tests: refactor run-tests helpers test-run-tests-rev.t will need them

tests: silence test-repo obsolete warning refactoring test-check-commit.t HGRCPATH bits as helpers-testrepo.sh

largefiles: send statlfile remote calls only for nonexisting locally files Files that are already in local store should be checked locally. The problem with this implementation is how difference in messages between local and remote checks should look like. For now local errors for file missing and content corrupted looks like this: 'changeset cset: filename references missing storepath\n' 'changeset cset: filename references corrupted storepath\n' for remote it looks like: 'changeset cset: filename missing\n' 'changeset cset: filename: contents differ\n' Contents differ error for remote calls is never raised currently - for now statlfile implementation lacks checking file content.

check-code: reject .next(...)

py3: convert to next() function next(..) was introduced in py2.6 and .next() is not available in py3 https://docs.python.org/2/library/functions.html#next

revset: rename variable to avoid shadowing with builtin next() function https://docs.python.org/2/library/functions.html#next

histedit: add experimental config for using the first word of the commit This allows users to start a commit with "verb! ..." so that when this is opened in histedit, the default action will be "verb". For example, "roll! foo" will default to the action "roll". Currently, we'll allow any known verb to be used but this is experimental.

histedit: add optional parameter for determining intial editor line A simple refactor to allow us to change the default verb for the initial editor display.

tests: enable import checker for all **.py files Several known-bad files are excluded as they couldn't be trivially fixed. That's the same as 99a2bdad0fda.

py3: make contrib/import-checker.py get along with itself Indent these imports to disable the rule of "not lexically sorted."

py3: make contrib/revsetbenchmarks.py not import symbols from stdlib modules

py3: make contrib/bdiff-torture.py conform to our import style

import-checker: extend check of symbol-import order to all local modules It doesn't make sense that (a) is allowed whereas (b) is disallowed. a) from mercurial import hg from mercurial.i18n import _ b) from . import hg from .i18n import _

import-checker: always build a list of imported symbols The next patch will rely on it.

import-checker: fix test to make a real package Otherwise "testpackage" wouldn't be counted as a package when building a list of imported symbols.

py3: move up symbol imports to enforce import-checker rules Since (b) is banned, we should do the same for (a) for consistency. a) from mercurial import hg from mercurial.i18n import _ b) from . import hg from .i18n import _

util: make copyfile avoid ambiguity of file stat if needed In some cases below, copying from backup is used to restore original contents of a file. If copying keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - failure of transaction before closing (from '.hg/journal.backup.*') - rollback of previous transaction (from '.hg/undo.backup.*') To avoid such problem, this patch makes copyfile() avoid ambiguity of file stat, if needed. Ambiguity check is executed, only if: - checkambig=True is specified (not all copying needs ambiguity check), and - destination file exists before copying This patch also adds 'not (copystat and checkambig)' assertion, because combination of copystat and checkambig is meaningless. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

vfs: make rename avoid ambiguity of file stat if needed In some cases below, renaming from backup is used to restore original contents of a file. If renaming keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - failure of transaction before closing (only from '.hg/journal.dirstate') - rollback of previous transaction (from '.hg/undo.*') - failure in dirstateguard scope (from '.hg/dirstate.SUFFIX') To avoid such problem, this patch makes vfs.rename() avoid ambiguity of file stat, if needed. Ambiguity check is executed, only if: - checkambig=True is specified (not all renaming needs ambiguity check), and - destination file exists before renaming This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

vfs: make atomictempfile avoid ambiguity of file stat if needed This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: make atomictempfile avoid ambiguity of file stat if needed Ambiguity check is executed at close(), only if: - atomictempfile is created with checkambig=True, and - target file exists before renaming This restriction avoids performance decrement by needless examination of file stat (for example, filelog doesn't need exact cache validation, even though it uses atomictempfile to write changes out). See description of filestat class for detail about why the logic in this patch works as expected. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: add filestat class to detect ambiguity of file stat Current posix.cachestat implementation might overlook change of a file, if changing keeps ctime, mtime and size of file. Comparison of inode number also overlooks changing in such situation, because inode number is rapidly reused. Contents of a file cached before changing isn't invalidated as expected, if change of a file is overlooked for this "ambiguity" of file stat. This patch adds filestat class to detect ambiguity of file stat. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

run-tests: handle json.dumps divergence In py2, json.dumps includes a trailing space after a comma at the end of lines. The py3 behavior which omits the trailing space is preferable, so we're going to strip it.

tests: use debuginstall to retrieve hg version

debuginstall: add mercurial version

strip: invalidate phase cache after stripping changeset (issue5235) When we remove a changeset from the changelog, the phase cache must be invalidated, otherwise it could refer to changesets that are no longer in the repo. To reproduce the failure, I created an extension querying the phase cache after the strip transaction is over. To do that, I stripped two commits with a bookmark on one of them to force another transaction (we open a transaction for moving bookmarks) after the strip transaction. Without the fix in this patch, the test leads to a stacktrace showing the issue: repair.strip(ui, repo, revs, backup) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/repair.py", line 205, in strip tr.close() File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/transaction.py", line 44, in _active return func(self, *args, **kwds) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/transaction.py", line 490, in close self._postclosecallback[cat](self) File "$TESTTMP/crashstrip2.py", line 4, in test [repo.changelog.node(r) for r in repo.revs("not public()")] File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/changelog.py", line 337, in node return super(changelog, self).node(rev) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/revlog.py", line 377, in node return self.index[rev][7] IndexError: revlog index out of range The situation was encountered in inhibit (evolve's repo) where we would crash following the volatile set invalidation submitted by Augie in e6f490e328635312ee214a12bc7fd3c7d46bf9ce. Before his patch the issue was masked as we were not accessing the phasecache after stripping a revision. This bug uncovered another but in histedit (see explanation in issue5235). I changed the histedit test accordingly to avoid fixing two things at once.

py3: make tests/svn-safe-append.py use absolute_import

py3: make tests/test-atomictempfile.py use absolute_import

py3: tests/test-check-py3-compat.t output updated The lower part of the tests runs with Python 3.5 so its remains unchanged with new commits.

py3: use setattr() to assign new class attribute The old method produces error 'object does not supports item assignment'. So setattr() is used to assign a new class attribute via __dict__ .

localrepo: use dirstate savebackup instead of handling dirstate file manually This is one step towards having dirstate manage its own storage. It will be useful for the implementation of sql dirstate [1]. This introduced a small test change: now we always write the dirstate before saving backup so in some cases where dirstate file didn't exist yet savebackup can create it. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

localrepo: use dirstate restorebackup instead of copying dirstate manually This is one step towards having dirstate manage its own storage. It will be useful for the implementation of sqldirstate [1]. I'm deleting two of the dirstate.invalidate() calls in localrepo because restorebackup method does that for us. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

dirstate: add prefix and suffix arguments to backup This would allow the code explicitly copying dirstate to use this method instead. Use of this method will increase encapsulation (the dirstate class will be sole owner of its on-disk storage).

tests: mark test-atomictempfile.py write as binary

tests: mark test-context.py write as binary

transaction: turn lack of locking into a hard failure (API) We have been warning about transactions without locks for about a year (and three releases), third party extensions had a fair grace period to fix their code, we are moving lack of locking to a hard failure in order to protect users against repository corruption.

test: extract develwarn transaction testing in its own command The lack of locking for a transation is about to change from a warning to an error. We first extract the test decidated to this warning to make the next changeset clearer.

graphmod: update edgemap in-place The edgemap update was not actually propagated to future asciiedge calls; update the edge state dictionary in-place instead.

tests: test-archive.t use mercurial.util for urllib compat

tests: test-archive.t use sys.stdout.buffer for binary output in py3

localrepo: prevent executable-bit only changes from being lost on amend If you have just executable-bit change and amend it twice it will vanish: * After the first amend the commit will have the proper executable bit set in manifest but it won't have the the file on the list of files in changelog. * The second amend will read the wrong list of files from changelog and it will copy the manifest entry from parent for this file. * Voila! The change is lost. This change repairs the bug in localrepo causing this and adds a test for it.

tests: escape bytes setting MSB in input of grep for portability GNU grep (2.21-2 or later) assumes that input is encoded in LC_CTYPE, and input is binary if it contains byte sequence not valid for that encoding. For example, if locale is configured as C, a byte setting most significant bit (MSB) makes such GNU grep show "Binary file <FILENAME> matches" message instead of matched lines unintentionally. This behavior is recognized as a bug, and fixed in GNU grep 2.25-1 or later. But some distributions are shipped with such buggy version (e.g. Ubuntu xenial, which is used by launchpad buildbot). http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19230 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800670 http://packages.ubuntu.com/xenial/grep This causes failure of test-commit-interactive.t, which applies grep on CP932 byte sequence since 1111e84de635. But, explicit setting LC_CTYPE for CP932 might cause another problem, because it can't be assumed that all environment running Mercurial tests allows arbitrary locale setting. To resolve this issue, this patch escapes bytes setting MSB in input of grep. For this purpose: - str.encode('string-escape') isn't useful, because it escapes also control code (less than 0x20), and makes EOL handling complicated - "f --hexdump" isn't useful, because it isn't line-oriented - "sed -n" seems reasonable, but "sed" itself sometimes causes portability issue, too (e.g. 900767dfa80d or afb86ee925bf) This patch is posted with "stable" flag, because 1111e84de635 is on stable branch.

tests: test-archive.t use absolute_import This is a step to adding a mercurial dependency to simplify py3 compat

tests: test-archive.t use open() instead of file() for py3 compat

tests: test-archive.t use print_function

merge with stable

readlink: use print_function

tests: test-addremove-similar.t use print() for py3

tests: add coverage for run-tests.py --whitelist

hg: limit HGUNICODEPEDANTRY to py2 reload is not available in py3, and py3 is fatal anyway

py3: make i18n/hggettext use print_function

py3: make i18n/hggettext use absolute_import

py3: make doc/docchecker use print_function

py3: make doc/docchecker use absolute_import

py3: make contrib/undumprevlog use absolute_import

py3: make contrib/dumprevlog use print_function

py3: make contrib/dumprevlog use absolute_import

py3: make contrib/check-commit use print_function

py3: make contrib/check-commit use absolute_import

hgcia: remove hgcia (BC) As discussed at: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-March/081018.html, cia service is down for years. It also uses socket.setdefaulttimeout() which will break chg. This patch removes the extension.

py3: make tests/hghave use absolute_import

py3: make tests/f use absolute_import

py3: make tests/dummyssh use absolute_import

py3: make raise statement python3 compatible In python3 raise error, message has been changed to raise error(message) In additional to that nodes.SkipNode is changed to nodes.SkipNode() so that it creates an instance directly.

Added signature for changeset aaabed77791a

Added tag 3.8.2 for changeset aaabed77791a

help: search section of help topic by translated section name correctly Before this patch, "hg help topic.section" might show unexpected section of help topic in some encoding. It applies str.lower() instead of encoding.lower(str) on translated message to search section case-insensitively, but some encoding uses 0x41(A) - 0x5a(Z) as the second or later byte of multi-byte character (for example, ja_JP.cp932), and str.lower() causes unexpected result. To search section of help topic by translated section name correctly, this patch replaces str.lower() by encoding.lower(str) for both query string (in commands.help()) and translated help text (in minirst.getsections()).

patch: show lower-ed translated message correctly Before this patch, patch.filterpatch() shows meaningless translation of help message for chunk selection in some encoding. It applies str.lower() instead of encoding.lower(str) on translated message, but some encoding uses 0x41(A) - 0x5a(Z) as the second or later byte of multi-byte character (for example, ja_JP.cp932), and str.lower() causes unexpected result. To show lower-ed translated message correctly, this patch replaces str.lower() by encoding.lower(str).

py3: make i18n/posplit use print_function

py3: make i18n/posplit use absolute_import

wireproto: optimize handling of large batch responses Now that batch can be used by remotefilelog, the quadratic string copying this was doing was actually disastrous. In my local testing, fetching a 56 meg file used to take 3 minutes, and now takes only a few seconds.

cleanup: replace False identity testing with an explicit token object The recommended way to check default value (when None is not as option) is a token object. Identity testing to integer is less explicit and not guaranteed to work in all implementations.

devel: officially deprecate dirstate.write without transaction argument When we introduce the develwarning, we did not had an official deprecation API and infrastructure. We can now officially deprecate the old way with a version deadline.

devel: officially deprecate update without destination When we introduce the develwarning, we did not had an official deprecation API and infrastructure. We can now officially deprecate the old way with a version deadline.

devel: fix a typo in a deprecation warning Credit goes to Sean Farley for spotting it.

devel: officially deprecate old style revset When we introduce the develwarning, we did not had an official deprecation API and infrastructure. We can now officially deprecate the old way with a version deadline.

check-code: handle py3 open divergence open() really wants an encoding attribute

check-code: switch to opener

check-code: handle range/xrange divergence

check-code: fix py3 complaint about \NNN being invalid unicode

hghave: switch from iteritems to items With this, test-hghave.t passes on python 3. Four features fail because mercurial still is not py3 safe: absimport cacheable hardlink defaultcacerts But that will be resolved automatically eventually.

hghave: matchoutput needs to use bytes for regexp file output is bytes in py3, so we need each regexp to be bytes

revset: make dagrange preserve order of input set Unlike range, dagrange has no inverted range (such as '10:0'). So there should be no practical reason to keep dagrange as a function that forces its own ordering. No performance regression is spotted in contrib/base-revsets.txt.

tests: mark test-check-pyflakes.t as requiring hg1.0+ hg does not yet run with py3, so if you try: ./run-tests.py --local test-check-pyflakes.t ... it will try to run the local hg, which does not work and thus, hg locate will return no output to stdout (and stderr is sent to /dev/null). If you do: ./run-tests.py --with-hg=~/bin/hg test-check-pyflakes.t Then it should work, if your hg is new enough to have a locate command (hg0.6 does not have locate).

dirstate: make backup methods public They are called from outside of dirstate anyway and I want the localrepo to use them too.

check-code: add a rule banning `env -u`

subrepo: use unset instead of env -u to fix test on BSDs (issue5229)

graphmod: partial edge styling Allow for a style to only apply to the last N lines (for positive N) or everything but the first N lines (for negative N) of the section along the current node. This allows for more subtle grandparent styling. So from the default: $ hg log -G ... o Lorem ipsum dolor sit :\ amet, consectetur : : adipiscing elit, sed : : do eiusmod tempor : : o : incididunt ut labore | : et dolore magna | : aliqua. Ut enim ad | : minim veniam, quis |/ o nostrud exercitation : ullamco laboris nisi : ut aliquip ex ea : commodo consequat. : o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu to $ hg log -G --config "experimental.graphstyle.grandparent=2." ... o Lorem ipsum dolor sit |\ amet, consectetur | | adipiscing elit, sed . . do eiusmod tempor . . o | incididunt ut labore | | et dolore magna | | aliqua. Ut enim ad | | minim veniam, quis |/ o nostrud exercitation | ullamco laboris nisi | ut aliquip ex ea . commodo consequat. . o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu or $ hg log -G --config "experimental.graphstyle.grandparent=1:" ... o Lorem ipsum dolor sit |\ amet, consectetur | | adipiscing elit, sed | | do eiusmod tempor : : o | incididunt ut labore | | et dolore magna | | aliqua. Ut enim ad | | minim veniam, quis |/ o nostrud exercitation | ullamco laboris nisi | ut aliquip ex ea | commodo consequat. : o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu or $ hg log -G --config "experimental.graphstyle.grandparent=-2!" ... o Lorem ipsum dolor sit |\ amet, consectetur ! ! adipiscing elit, sed ! ! do eiusmod tempor ! ! o | incididunt ut labore | | et dolore magna | | aliqua. Ut enim ad | | minim veniam, quis |/ o nostrud exercitation | ullamco laboris nisi ! ut aliquip ex ea ! commodo consequat. ! o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu

pure: write a really lazy version of pure indexObject On PyPy this version performs reasonably well compared to C version. Example command is "hg id" which gets faster, depending on details of your operating system and hard drive (it's bottlenecked on stat mostly) There is potential for improvements by storing extra as a condensed struct too.

dispatch: always load extensions before running shell aliases (issue5230) Before this patch, we may or may not load extensions for shell aliases depending on whether the command is abbreviated or not. Loading extensions may have useful side effects to shell aliases. For example, the pager extension does not work for shell aliases. This patch removes the code checking shell aliases before loading extensions to give the user a more consistent experience. It may hurt performance for shell aliases a bit without chg but the correctness seems worth it. It will also make the behavior consistent with chg since chg will always load all extensions before running commands.

httpclient: update to upstream revision 2995635573d2 This is mostly Python 3 compat work thanks to timeless.

crecord: call prevsibling() and nextsibling() directly The 3 classes for items used in crecord (uiheader, uihunk, uihunkline) all have prevsibling() and nextsibling() methods. The two methods are used to get the previous/next item of the same type of the same parent element as the current one: when `a` is a uihunkline instance, a.nextsibling() returns the next line in this hunk (or None, if `a` is the last line). There are also two similar methods: previtem() and nextitem(). When called with constrainlevel=True (the default) they simply returned the result of prevsibling()/nextsibling(). Only when called with constrainlevel=False they did something different: they returned previous/next item regardless of its type (so if `a` is the last line in a hunk, a.nextitem(constrainlevel=False) could return the next hunk or the next file -- something that is not a line). Let's simplify this logic and make code call -sibling() methods when only siblings are needed and -item() methods when any item would do, and then remove the constrainlevel argument from previtem() and nextitem().

dispatch: add fail-* family of hooks The post-* family of hooks will not run in case a command fails (i.e. raises an exception). This makes it inconvenient to hook into events such as doing something in case of a failed push. We catch all exceptions to run the failure hook. I am not sure if this is too aggressive, but tests apparently pass.

py3: make hgext/rebase.py use absolute_import

py3: make hgext/mq.py use absolute_import

py3: make hgext/hisedit.py use absolute_import

py3: make hgext/hgk.py use absolute_import

py3: make hgext/gpg.py use absolute_import

py3: make hgext/graphlog.py use absolute_import

import-checker: recognize relative imports from parents of current package So far fromlocal recognizes relative imports of the form: from . import D from .. import E It wasn't prepared for recognizing relative imports like: from ..F import G The bug was not found so far because all relative imports starting from the parent was in the list of allowsymbolicimports like: from ..i18n import from ..node import

py3: make hgext/fetch.py use absolute_import

tests: test-check-py3-compat.t output updated The test output was not updated as the lower section of the test updates with python3.5, so it might be the case that people have updated the modules but the test was left as it was. So this patch updates the test output.

revset: factor out public optimize() function from recursion New optimize() hides internal arguments and return values. This makes it easy to add more parameters and return values to _optimize().

revset: introduce temporary variables in optimize() where they look better

revset: construct arguments of only() against matched tree Since _isonly() knows the structure of 'revs' and 'bases', it should be slightly easier to understand than destructuring 'ta' and 'tb'.

revset: unnest isonly() closure from optimize() There were no variables to be captured.

sslutil: stop checking for web.cacerts=! (BC) The previous patch stopped setting web.cacerts=! to indicate --insecure. That left user configs as the only source that could introduce web.cacerts=!. The practical impact of this patch is we no longer honor web.cacerts=! in configs. Instead, we always treat web.cacerts as a path. The patch is therefore technically BC. However, since I don't believe web.cacerts=! is documented, it should be safe to remove. a939f08fae9c (which introduced --insecure) has no indication that web.cacerts=! is anything but an implementation detail, reinforcing my belief it can be removed without major debate.

dispatch: stop setting web.cacerts=! to indicate --insecure Consumers needing to know if --insecure was used have already transitioned to using ui.insecureconnections. The previous patch removed the last meaningful consumer looking for web.cacerts=!.

sslutil: use CA loaded state to drive validation logic Until now, sslkwargs may set web.cacerts=! to indicate that system certs could not be found. This is really obtuse because sslkwargs effectively sets state on a global object which bypasses wrapsocket() and is later consulted by validator.__call__. This is madness. This patch introduces an attribute on the wrapped socket instance indicating whether system CAs were loaded. We can set this directly inside wrapsocket() because that function knows everything that sslkwargs() does - and more. With this attribute set on the socket, we refactor validator.__call__ to use it. Since we no longer have a need for setting web.cacerts=! in sslkwargs, we remove that. I think the new logic is much easier to understand and will enable behavior to be changed more easily.

sslutil: handle ui.insecureconnections in validator Right now, web.cacerts=! means one of two things: 1) Use of --insecure 2) No CAs could be found and were loaded (see sslkwargs) This isn't very obvious and makes changing behavior of these different scenarios independent of the other impossible. This patch changes the validator code to explicit handle the case of --insecure being used. As the inline comment indicates, there is room to possibly change messaging and logic here. For now, we are backwards compatible.

sslutil: check for ui.insecureconnections in sslkwargs The end result of this function is the same. We now have a more explicit return branch. We still keep the old code looking at web.cacerts=! a few lines below because we're still setting web.cacerts=! and need to react to the variable. This will be removed in an upcoming patch.

dispatch: set ui.insecureconnections when --insecure is used

ui: add an instance flag to hold --insecure bit Currently, when --insecure is used we set web.cacerts=! and socket validation takes this value into account. web.cacerts=! is not documented AFAICT and is purely an internal implementation detail. Let's be more explicit about what is going on by introducing a dedicated variable outside of the config values to track that --insecure is used.

sslutil: make sslkwargs code even more explicit The ways in which this code can interact with socket wrapping and validation later are mind numbing. This patch helps make it even more clear. The end behavior should be identical.

sslutil: move code examining _canloaddefaultcerts out of _defaultcacerts Before, the return of _defaultcacerts() was 1 of 3 types. This was difficult to read. Make it return a path or None. We had to update hghave.py in the same patch because it was also looking at this internal function. I wasted dozens of minutes trying to figure out why tests were failing until I found the code in hghave.py...

sslutil: further refactor sslkwargs The logic here and what happens with web.cacerts is mind numbing. Make the code even more explicit.

sslutil: document and slightly refactor sslkwargs This will help me and any reviewers keep sane as this code is refactored.

localrepo: remove a couple of local type aliases The local aliases are unused now, and were confusing mypy's type checker.

cmdutil: typo fix in comment

tests: add globs for Windows

chgserver: add [alias] to confighash The [alias] config section affects a global state: commands.table. It's hard to trace whether an alias is added by a config option or an extension, and add/remove aliases in a safe way per chg request. It will hurt performance a bit if we clean the table and parse aliases every time. Therefore let's just add it to confighash. This will make chg pass test-pager.t.

devel: use the 'config' argument for the dirstate normalisation develwarn

devel: use the new 'config' argument for the update develwarn

devel: use the new 'config' argument for the revset develwarn

devel: use the new 'config' argument for the dirstate develwarn

devel: use the new 'config' argument of the develwarn in deprecwarn Controling all deprecation warnings with the same config seems sensible. This mirror a fix (about missing gating) submitted for stable but with the new API.

develwarn: move config gating inside the develwarn function The config gating is almost always the same and contributor tend to forget it. We move the logic inside the function. Call site will be updated in later changeset. We might make the sub config mandatory in the future (once all old call sites are gone).

help: fix the display for `hg help internals.revlogs` (issue5227) It previously aborted saying the help section wasn't found. Credit to Yuya for figuring out the fix.

builddeb: add distroseries to tagged versions This is needed so that launchpad and friends have a unique version number for each distroseries (trusty, wily, xenial, etc). It was discovered when trying to upload 3.8 to launchpad.

debian: forgot to make debian/rules executable in 6b95a623ec90

debian: add wish to suggests Debian maintainers already have this and lintian warns us about not listing 'wish' as a dependency or suggestion so this patch does indeed just that. The issue, by the way, is that we are shipping hgk (which is written in tcl/tk) so we should be good citizens and list wish (a meta package for tcl/tk) as a dependency.

hg-ssh: copy doc string to man page This corrects a warning from lintian that we're shipping an executable without a man page. Since there is a doc string in the text, let's use that for the man page.

progress: stop excessive clearing (issue4801) The progress bar was being cleared on every write(), regardless of whether it was currently displayed. This could foul up the display of any writes that didn't include a linebreak. In particular, the win32 mode of the color extension was turning single prompt string writes into two writes, and the resulting clear/write/clear/write pattern was making the prompt invisible. We fix this by insisting that we have shown a progress bar and haven't just cleared it (setting lastprint to 0). Conveniently, the test suite already had instances of duplicate clears.. that are now cleared up.

chgserver: remove _clearenvaliases Since we expand environment variables in alias lazily, the _clearenvaliases hack is no longer necessary. This resolves an issue that a non-shell alias which has environment variables in its arguments and is set to use pager will not use pager running with chg.

dispatch: defer environment variable resolution in alias commands (BC) Before this patch, if there are environment variables in an alias command, they will be expanded immediately when we first see the alias. This will cause issues with chg, because environment variable updates will not propagate to expanded arguments. This patch makes "args" of "cmdalias" a property that will be calculated every time when accessed.

rollback: add a config knob for entirely disabling the command This is of pretty high value for organizations that used to use p4 (as an example), since `p4 rollback` is what we call `hg backout`.

templater: add separate() template function A pretty common pattern in templates is adding conditional separators like so: {node}{if(bookmarks, " {bookmarks}")}{if(tags, " {tags}")} With this patch, the above can be simplified to: {separate(" ", node, bookmarks, tags)} The function is similar to the already existing join(), but with a few differences: * separate() skips empty arguments * join() expects a single list argument, while separate() expects each item as a separate argument * separate() takes the separator first in order to allow a variable number of arguments after it

bookmarks: jettison bmstore's write() method per deprecation policy

merge with stable

deprecation: gate deprecation warning behind devel configuration Regular users are not supposed to be exposed to the API deprecation warnings. We now only issue them when the developper warnings are enabled.

crecord: update downarrowshiftevent() docstring, remove todo The phrasing is mostly taken from uparrowshiftevent().

crecord: remove things that don't happen in functions from their docstrings Scrolling screen is currently done in a different place. The things that had been described in the docstrings may still happen, but the functions touched by this patch don't do any scrolling, they only set self.currentselecteditem and nothing more.

crecord: remove skipfolded keyword argument from patchnode.previtem() It wasn't used, it wasn't implemented. Probably was a copy-paste bonus from patchnode.nextitem()

crecord: update a copy-pasted comment in downarrowshiftevent()

crecord: drop the version condition for amend The UI is now shipped in core, amend feature is always available.

crecord: add/remove blank lines (coding style)

localrepo: jettison parents() method per deprecation policy (API)

revset: define _parsealias() in _aliasrules class It's short. It doesn't make sense to define _parsealias() outside of the class.

revset: factor out common parsing function

revset: inline _tokenizealias() into _parsealias() This helps factoring out common part between _parsealias() and parse().

store: treat range as a generator instead of a list for py3 compat

ui: add new config option for help text width Before this patch, when printing help text using `hg help`, or `hg log -h`, the output will wrap at 78 chars even if the user has a bigger terminal width and there is no config option to change it, making the experience different from the commonly used `man` tool. This patch introduces a new config option `ui.textwidth`, which replaces the hardcoded number. It's set to 78 by default to maintain compatibility. When set to 0, `hg help` will behave more like `man`.

tests: test histedit base command plan help

largefiles: makes verify batching stat calls to remote Instead of sending stat calls for each files separately, it sends one batch call with stat invocations for all files.

largefiles: change basestore._verifyfile to take list of files to check Makes it easier to use batch stat calls in remotestore to decrease number of round trips.

bookmarks: properly invalidate volatile sets when writing bookmarks This corrects a regression introduced during the 3.7 cycle, but which went undetected due to the surviving-but-deprecated write() method on bmstore.

test-obsolete: update extension in test to actually work This hasn't been testing anything since partway through the 3.7 cycle due to unrelated refactoring. Sadly, the behavior it was trying to prevent reemerged in the codebase at that time. A fix is in the next patch, because proving that the fix was actually correct ended up being trickier than I expected.

bundle2: properly request phases during getbundle getbundle was requesting the "phase" namespace instead of the "phases" namespace, which led to the client still requesting the phases separately after getbundle finished.

rebase: handle successor targets (issue5198) When a parent has a successor (indicated by revprecursor in state), we need to use it.

debugbundle: add tests for debugbundle output with bundle2

revert: mention ui.origbackuppath in the command help

help: wrap ".orig" in rst quotes Apparently, .orig. is a macro for man pages so we need to wrap it in quotes to silence lintian warnings.

parser: shorten prefix of alias parsing errors These messages seemed to be a bit long. We should try making them fit to 80-col console.

parser: rephrase "'$' not for alias arguments" message Say which symbol caused the error. The word "alias" is removed since these messages are prefixed by "failed to parse ... revset alias "...":".

debugbundle: handle the --all option for bundle2

manifest: improve filesnotin performance by using lazymanifest diff lazymanifests can compute diffs significantly faster than taking the set of two manifests and calculating the delta. when running hg diff --git -c . on Facebook's big repo, this reduces the run time from 2.1s to 1.5s.

keyword: replace use of _filerev with _filenode To be independent of rev numbers. Analogous to ba8257cb53e8.

merge with stable

Added signature for changeset a56296f55a5e

Added tag 3.8.1 for changeset a56296f55a5e

convert: pass absolute paths to git (SEC) Fixes CVE-2016-3105 (1/1). Previously, it was possible for the repository path passed to git-ls-remote to be misinterpreted as a URL. Always passing an absolute path to git is a simple way to avoid this.

Added signature for changeset f85de28eae32

Added tag 3.8 for changeset f85de28eae32

debian: alphabetize build deps

debian: fix lintian warning about debhelper It seems this is correct but does it work on older distros? I ran the docker-jessie rule and didn't get any warnings.

builddeb: remove chmod as lintian tells us It turns out we just need debian/rules to be executable, so we do just that.

builddeb: use codename in version Apparently, this is needed to allow ppas to be built for multiple distros.

rebase: restrict rebase destination to the pulled set (issue5214) Before this patch, `hg pull --rebase` would be a strict sequence of `hg pull` followed by `hg rebase` if anything was pulled. Now that rebase pick his default destination the same way than merge, than `hg rebase` step would abort in the case the repo already had multiple anonymous heads (because of the ambiguity). (changed in fac3a24be50e) The intend of the user with `hg pull --rebase` is clearly to rebase on pulled content. This used to be (mostly) enforced by the former default destination for rebase, "tipmost changeset of the branch" as the tipmost would likely a changeset that just got pulled. But this intended was no longer enforced with the new defaul destination (unified with merge). This changeset makes use of the '_destspace' mechanism introduced in the previous changeset to enforce this. This partially fixes issue5214 as no change at all have been made to the new handling of the case with bookmark (unified with merge).

destutil: add the ability to specify a search space for rebase destination In the 'hg pull --rebase', we don't want to pick a rebase destination unrelated to the pull, we lay down basic infrastructure to allow such restriction on stable (before 3.8 release) in this case. See issue 5214 for details. Actual usage and test will be in the next patch.

sslutil: restore old behavior not requiring a hostname argument (issue5210) This effectively backs out changeset 1fde84d42f9c. The http library behind ui.http2=true isn't specifying the hostname. It is the day before the expected 3.8 release and we don't want to ship a regression. I'll try to restore this requirement in the 3.9 release cycle as part of planned improvements to Mercurial's SSL/TLS interactions.

tests: test a variety of cache invariants We've historically had a problem maintaining the expected invariants on our caches, especially when introducing new caches. This tests documents the invariants and exercises them across most of our existing cache files.

repoview: ignore unwritable hidden cache The atomictemp.close() file attempts to do a rename, which can fail. Moving the close inside the exception handler fixes it. This doesn't fit well with the with: pattern, as it's the finalizer that's failing.

tags: silence hgtagsfnodes reading failures tryread() doesn't handle "is a directory" errors and presumably others. We might not want to globally swallow such tryread errors, so we replace with our own try/except handling. An upcoming test will use directories as a portable stand-in for various bizarre circumstances that cache read/write code should be robust to.

tags: silence cache parsing errors Follow our standard STFU cache-handling pattern

i18n-pt_BR: synchronized with 18c1b107898e

ubuntu-xenial-ppa: add makefile rule

ubuntu-wily-ppa: add makefile rule

ubuntu-trusty-ppa: add makefile rule

ubuntu-xenial: add makefile rule to build deb

ubuntu-wily: add makefile rule to build deb

make: turn ubuntu docker into template This allows us to easily add more ubuntu docker targets (which following patches will do). Also, we no longer need the mkdir command.

revsets: add docs for '%' operator

graft: fix printing of --continue command Properly shell quote arguments, to avoid printing commands that won't work when run literally. For example, a date string with timestamp needs to be quoted: --date '1456953053 28800'

hghave: remove unused check for bdist_mpkg

osx: create a modern package including manpages Instead of using bdist_mpkg, we use the modern Apple-provided tools to build an OS X Installer package directly. This has several advantages: * Avoids bdist_mpkg which seems to be barely maintained and is hard to use. * Creates a single unified .pkg instead of a .mpkg. * The package we produce is in the modern, single-file format instead of a directory bundle that we have to zip up for download. In addition, this way of building the package now correctly: * Installs the manpages, bringing the `make osx`-generated package in line with the official Mac packages we publish on the website. * Installs files with the correct permissions instead of encoding the UID of the user who happened to build the package. Thanks to Augie for updating the test expectations.

hghave: add check for OS X packaging tools

tests: add test for Mac OS X package construction

osx: add support for keeping mpkgs This is a bit of a hack, but I don't really want to mount a dmg during a test, and I don't see an option with hdiutil to take a dmg and spit out a folder, so this is what we've got for now.

osx: add support for dumping built dmg into OUTPUTDIR

hghave: add check for bdist_mpkg

verify: don't init subrepo when missing one is referenced (issue5128) (API) Initializing a subrepo when one doesn't exist is the right thing to do when the parent is being updated, but in few other cases. Unfortunately, there isn't enough context in the subrepo module to distinguish this case. This same issue can be caused with other subrepo aware commands, so there is a general issue here beyond the scope of this fix. A simpler attempt I tried was to add an '_updating' boolean to localrepo, and set/clear it around the call to mergemod.update() in hg.updaterepo(). That mostly worked, but doesn't handle the case where archive will clone the subrepo if it is missing. (I vaguely recall that there may be other commands that will clone if needed like this, but certainly not all do. It seems both handy, and a bit surprising for what should be a read only operation. It might be nice if all commands did this consistently, but we probably need Angel's subrepo caching first, to not make a mess of the working directory.) I originally handled 'Exception' in order to pick up the Aborts raised in subrepo.state(), but this turns out to be unnecessary because that is called once and cached by ctx.sub() when iterating the subrepos. It was suggested in the bug discussion to skip looking at the subrepo links unless -S is specified. I don't really like that idea because missing a subrepo or (less likely, but worse) a corrupt .hgsubstate is a problem of the parent repo when checking out a revision. The -S option seems like a better fit for functionality that would recurse into each subrepo and do a full verification. Ultimately, the default value for 'allowcreate' should probably be flipped, but since the default behavior was to allow creation, this is less risky for now.

setup: detect Python DLL filename from loaded DLL Attempting to build Mercurial from source using MinGW from msys2 on Windows produces a hg.exe that attempts to load e.g. python27.dll. MinGW prefixes its library name with "lib" and adds a period between the major and minor versions. e.g. "libpython2.7.dll." Before this patch, hg.exe files in a MinGW environment would either fail to find a Python DLL or would attempt to load a non-MinGW DLL, which would summarily explode. Either way, hg.exe wouldn't work. This patch improves the code that determines the Python DLL filename to actually use the loaded Python DLL instead of inferring it. Basically we take the handle of the loaded DLL from sys.dllhandle and call a Windows API to try to resolve that handle to a filename.

exewrapper: add .dll to LoadLibrary() argument LoadLibrary() changes behavior depending on whether the argument passed to it contains a period. From the MSDN docs: If no file name extension is specified in the lpFileName parameter, the default library extension .dll is appended. However, the file name string can include a trailing point character (.) to indicate that the module name has no extension. When no path is specified, the function searches for loaded modules whose base name matches the base name of the module to be loaded. If the name matches, the load succeeds. Otherwise, the function searches for the file. As the subsequent patch will show, some environments on Windows define their Python library as e.g. "libpython2.7.dll." The existing code would pass "libpython2.7" into LoadLibrary(). It would assume "7" was the file extension and look for a "libpython2.dll" to load. By passing ".dll" into LoadLibrary(), we force it to search for the exact basename we want, even if it contains a period.

update: correct description of --check option The old "update across branches if no uncommitted changes" made it sound like updating across branches (with no uncommitted changes) was allowed only with this option, which was not true. Also, the option did not care whether it was linear or across branches. Instead, it checked that there were no uncommitted changes. Let's explain what it does instead of trying to suggest what happens without it.

util: fix race in makedirs() Update makedirs() to ignore EEXIST in case someone else has already created the directory in question. Previously the ensuredirs() function existed, and was nearly identical to makedirs() except that it fixed this race. Unfortunately ensuredirs() was only used in 3 places, and most code uses the racy makedirs() function. This fixes makedirs() to be non-racy, and replaces calls to ensuredirs() with makedirs(). In particular, mercurial.scmutil.origpath() used the racy makedirs() code, which could cause failures during "hg update" as it tried to create backup directories. This does slightly change the behavior of call sites using ensuredirs(): previously ensuredirs() would throw EEXIST if the path existed but was a regular file instead of a directory. It did this by explicitly checking os.path.isdir() after getting EEXIST. The makedirs() code did not do this and swallowed all EEXIST errors. I kept the makedirs() behavior, since it seemed preferable to avoid the extra stat call in the common case where this directory already exists. If the path does happen to be a file, the caller will almost certainly fail with an ENOTDIR error shortly afterwards anyway. I checked the 3 existing call sites of ensuredirs(), and this seems to be the case for them.

chg: initialize sockdirfd to -1 instead of AT_FDCWD As we don't use sockdirfd yet, this is the simplest workaround to compile chg on old Unices where AT_FDCWD does not exist. Foozy pointed out Mac OS X 10.10 is required for AT_FDCWD as well as xxxat() functions.

bdiff: further restrain potential quadratic performance This causes the longest_match search to limit itself to a window of 30000 lines during search (roughly 1MB), thus avoiding a full O(N*M) search that might occur in repetitive structured inputs. For a particular class of many MB pathological test cases, this generated the following timings: size before after 10x 1.25s 1.24s 100x 57s 33s 1000x >8400s 400s The times on the right quickly become much faster and appear more linear. While windowing means deltas are no longer "optimal", the resulting deltas were within a couple percent of expected size. While we've yet to have a report of a file with the level of repetition necessary to hit this case, some JSON/XML database dump scenario is fairly likely to hit it. This may also slightly improve the average-case performance for deltas of large binaries.

bdiff: balance recursion to avoid quadratic behavior (issue4704) For highly structured files like JSON or XML dumps with large numbers of duplicate lines (eg braces) and isolated matching lines, bdiff could find large numbers of equally good spans. Because it prefers earlier matches, this would result in pathologically unbalance recursion that resulted in quadratic performance. This patch makes it prefer matches closer to the middle that tend to balance recursion. This change improves the speed of a pathological test case from 1100s to 9s. Included is a smaller test that has a roughly 50x safety margin on the performance it accepts. It's likely to fail on pure builds because difflib also has a recursion-balancing problem.

bdiff: deal better with duplicate lines The longest_match code compares all the possible positions in two files to find the best match. Given a pair of sequences, it effectively searches a grid like this: a b b b c . d e . f 0 1 2 3 4 5 6 7 8 9 a 1 - - - - - - - - - b - 2 1 1 - - - - - - b - 1 3 2 - - - - - - b - 1 2 4 - - - - - - . - - - - - 1 - - 1 - Here, the 4 in the middle says "the first four lines of the file match", which it can compute be comparing the fourth lines and then adding one to the result found when comparing the third lines in the entry to the upper left. We generally avoid the quadratic worst case by only looking at lines that match, which is precomputed. We also avoid quadratic storage by only keeping a single column vector and then keeping track of the best match. Unfortunately, this can get us into trouble with the sequences above. Because we want to reuse the '3' value when calculating the '4', we need to be careful not to overwrite it with the '2' we calculate immediately before. If we scan left to right, top to bottom, we're going to have a problem: we'll overwrite our 3 before we use it and calculate a suboptimal best match. To address this, we can either keep two column vectors and swap between them (which significantly complicates bookkeeping), or change our scanning order. If we instead scan from left to right, bottom to top, we'll avoid ever overwriting values we'll need in the future. This unfortunately needs several changes to be made simultaneously: - change the order we build the initial hash chains for the b sequence - change the sentinel values from INT_MAX to -1 - change the visit order in the longest_match inner loop - add a tie-breaker preference for earlier matches This last is needed because we previously had an implicit tie-breaker from our visitation order that our test suite relies on. Later matches can also trigger a bug in the normalization code in diff().

bdiff: fix latent normalization bug This bug is hidden by the current bias towards matches at the beginning of the file. When this bias is tweaked later to address recursion balancing, the normalization code could cause the next block to shrink to a negative length, thus creating invalid delta chunks. We add checks here to disallow that. This bug requires test cases that are an awkwardly large size for the test suite, but is very rapidly picked up by the included torture tester.

bdiff: fold in shift calculation in normalize This just makes the code harder to read without any performance advantage. We're going to make the check here more complex, let's make it simpler first.

bdiff: unify duplicate normalize loops We're about to make the while loop check more complicated, so let's simplify first.

make: backout changeset 51f5fae84e43 Support for '!=' was only added in GNU Make 4.0, and CentOS versions as new as CentOS 7 only carry 3.82. I will leave figuring out compatibility with BSD make as an exercise for interested folks.

tests: test-lock-badness.t message could come later I got this on gcc112: @@ -58,8 +58,8 @@ $ hg -R b ci -A -m b --config hooks.precommit="python:`pwd`/hooks.py:sleepone" > stdout & $ hg -R b up -q --config hooks.pre-update="python:`pwd`/hooks.py:sleephalf" waiting for lock on working directory of b held by '*:*' (glob) - got lock after ? seconds (glob) $ wait + got lock after 1 seconds $ cat stdout adding b

dockerdeb: pass the rest of the args to the builder script It seems this was the original intent of the script so this patch passes the remanining arguments to builddeb.

dockerdeb: fix incorrect number of shifts From the comment, it appears that the original intent was to remove the first two arguments, so this patch does just that.

make: use shell-command assignment instead of $(eval ...) This is portable between BSD and GNU make. As of this change, our Makefile appears to work in both BSD and GNU make, with the caveat that the test-% and testpy-% wildcard rules don't work on BSD make. That said, this still seems worthwhile because it lets the buildbots work more consistently across platforms.

make: do assignment and export in a single statement This is portable between GNU and BSD make, whereas doing the export on its own line confuses BSD make.

make: alter how we compute compiler flags for setup.py This is portable between BSD and GNU make. I'm not thrilled with how it worked out, but it's portable and solves the problem.

revset: unindent "if True" block in sort() It was there to make the previous patch readable.

revset: make sort() do dumb multi-pass sorting for multiple keys (issue5218) Our invert() function was too clever to not take length into account. I could fix the problem by appending '\xff' as a terminator (opposite to '\0'), but it turned out to be slower than simple multi-pass sorting. New implementation is pretty straightforward, which just calls sort() from the last key. We can do that since Python sort() is guaranteed to be stable. It doesn't sound nice to call sort() multiple times, but actually it is faster. That's probably because we have fewer Python codes in hot loop, and can avoid heavy string and list manipulation. revset #0: sort(0:10000, 'branch') 0) 0.412753 1) 0.393254 revset #1: sort(0:10000, '-branch') 0) 0.455377 1) 0.389191 85% revset #2: sort(0:10000, 'date') 0) 0.408082 1) 0.376332 92% revset #3: sort(0:10000, '-date') 0) 0.406910 1) 0.380498 93% revset #4: sort(0:10000, 'desc branch user date rev') 0) 0.542996 1) 0.486397 89% revset #5: sort(0:10000, '-desc -branch -user -date -rev') 0) 0.965032 1) 0.518426 53%

log: fix status template to list copy source per dest (issue5155) Before, copied files were assumed as "A" (added) and listed followed by non-copy added files. This could double entries of a copy if it had "M" (modified) state. So, this patch makes the template check if a file is included in copies dict. This way, entries should never be doubled. The output of "log -Tstatus -C" does not always agree with "status -C --change" due to the bug of "status", which is documented in test-status.t. See also 2963d5c9d90b.

graphmod: disable graph styling when HGPLAIN is set (issue5212) Produce stable output for tools to rely on by hardcoding all edge styles to "|". This ensures that any tool parsing the output of hg log -G still gets the same behaviour as pre-3.8 releases.

graphmod: fix seen state handling for > 2 parents (issue5174) When there are more than 2 parents for a given node (in a sparse graph), extra dummy nodes are inserted to transition the lines more gradually. However, since the seen state was not updated when yielding the extra nodes, the wrong graph styles were being applied to the nodes.

httpclient: reverse accidental damage from 86db5cb55d46

tests: tolerate http2 You can run tests like this: run-tests.py -l --extra-config-opt ui.usehttp2=true And ideally, no tests should fail...

make: add rule for building an ubuntu ppa

builddeb: add flag for a source-only deb This is required for building a ppa for ubuntu which following patches will use.

builddeb: create source archive for ubuntu

builddeb: ignore vcs and build results This one is a no-brainer. Previously, if you tried to build a deb on ubuntu, it would try to diff files in the .hg store. These flags prevent that.

builddeb: copy over .gz and .dsc files We were forgetting to copy over the signature (if it exists) and the zipped diff, so let's do that.

builddeb: ignore errors about find not finding files The debuild command may output less files than we explicitly list so let's not error out if none exist.

builddeb: use the os codename instead of 'unstable' This fixes a lintian error (and indeed, launchpad rejects it) by using the distribution's codename (e.g. xenial, trusty, etc).

builddeb: use sed -i Notice that there is no space after '-i'. This makes it work on both GNU and BSD versions of sed.

dockerdeb: redirect 'cd' in export command to /dev/null This had the unfortunate side effect of causing the environment to have a newline due to the fact that some 'cd' outputs the result of the directory change. So, let's just redirect the meaningless output.

help: avoid using "$n" parameter in revsetalias example Because parsing "$n" requires a crafted tokenizer, it exists only for backward compatibility (as documented in revset._tokenizealias.) This patch updates the examples so that users are encouraged to use symbolic names instead of "$n"s. I'm going to implement alias expansion in templater, which won't support "$n" parameters to make my life easier. Templater is more complicated than revset because tokenizer and parser call each other.

debian: add missing netbase dependency Apparently, some machines don't have this service (launchpad builders are one such example). This adds the correct dependency for test-serve.t.

debian: add missing zip/unzip dependencies

debian: add missing python-docutils dependency

debian: add missing python-all-dev dependency

patchbomb: use single quotes around command hint Windows command lines use double quotes to quote arguments with spaces. This change is in a series to unify around using single quotes around commands, and double quotes around interior arguments. This changeset is taken on stable for consistency with similar update done before the freeze. See 2e58dc022caa, ad2cd2ef25d9, fc1d75e7a98d and 9dcc9ed26d33.

chg: forward SIGWINCH to worker Before this patch, if the user uses chg and ncurses interface, resizing the terminal window will mess up its content. This patch fixes the issue by forwarding SIGWINCH to the worker process.

Added signature for changeset 740156eedf2c

Added tag 3.8-rc for changeset 740156eedf2c

merge default into stable for 3.8 code freeze

make: remove packages directory in clean rule

make: add forgotten hgext3rd to clean rule

make: add chg to clean rule

test-docker-packaging: add new line to test output It seems we changed our build but didn't update the docker test.

tests: relax pattern matching for newer docker

py3: make factotum use absolute_import check-code complains for using urllib2 so that too was fixed.

py3: make extdiff use absolute_import

py3: make eol use absolute_import

py3: make color use absolute_import

py3: make hgmanpage use absolute_import

py3: make gendoc use absolute_import Fixed direct imports even the tests were not complaining.

py3: make check-seclevel use absolute_import Also fixed direct symbol imports even the tests were not complaining.

fetch: use single quotes around command hint Windows command lines use double quotes to quote arguments with spaces. This change is in a series to unify around using single quotes around commands, and double quotes around interior arguments.

graft: use single quotes around command hint Windows command lines use double quotes to quote arguments with spaces. This change is in a series to unify around using single quotes around commands, and double quotes around interior arguments.

config: use single quotes around command hint Windows command lines use double quotes to quote arguments with spaces. This change is in a series to unify around using single quotes around commands, and double quotes around interior arguments.

debugcreatestreamclonebundle: use single quotes around command hint Windows command lines use double quotes to quote arguments with spaces. This change is in a series to unify around using single quotes around commands, and double quotes around interior arguments.

transaction: clear callback instances after usage Prevents double usage and helps reduce reference cycles, which were observed to occur in `hg convert` and other scenarios where there are multiple transactions per process.

lock: clear postrelease hooks list after usage Post release hooks should only be called once. Setting the list to None after usage will prevent accidental usage after they are used. In addition, it is easy for reference cycles to sneak into hook functions. Clearing the hooks after usage helps prevent these cycles.

ui: drop template aliases by HGPLAIN Otherwise, scripting output could be suffered from user aliases.

templater: load and expand aliases by template engine (API) (issue4842) Now template aliases are fully supported in log and formatter templates. As I said before, aliases are not expanded in map files. This avoids possible corruption of our stock styles and web templates. This behavior is undocumented since no map file nor [templates] section are documented at all. Later on, we might want to add [aliases] section to map files if it appears to be useful.

templater: inline compiletemplate() function into engine This allows the template engine to modify parsed tree.

templater: factor out function that creates templater from string template This function will host loading of template aliases. It is not defined at templater, but at formatter, since formatter is the module handling ui stuff in front of templater.

templater: separate function to create templater from map file (API) New frommapfile() function will make it clear when template aliases will be loaded. They should be applied to command arguments and templates in hgrc, but not to map files. Otherwise, our stock styles and web templates (i.e map-file templates) could be modified unintentionally. Future patches will add "aliases" argument to __init__(), but not to frommapfile().

templater: extract function that loads template map file Prepares for API change. See the next patch for details. 'map' variable is renamed to avoid shadowing map() function.

templater: demote "base" directory of map file to local variable It isn't referenced from other places.

notify: do not load style file if template is specified (BC) This patch makes sure that either "tmpl" or "mapfile" is exclusively set, which is the same behavior as common log-like templates and formatter outputs. See the previous patch for why.

bugzilla: do not load style file if template is specified (BC) This prepares for the API change to support template aliases. I'm going to extract a factory function of templater that reads a map file: # original templater(mapfile, ..., cache, ...) # new templater.frommapfile(mapfile, ...) # read mapfile to build cache/map templater(..., cache, ...) # use specified cache (= map elements) This will make it clear to isolate stock styles (i.e. map files) from user aliases. Template aliases should be applied to command arguments and templates in hgrc, but not to map files. Otherwise, our stock styles and web templates could be modified unintentionally. This patch makes sure that either "tmpl" or "mapfile" is exclusively set. It's theoretically a behavior change, since you could put new keywords in template by defining them in a map file before: # mapfile foo = "{rev}" # hgrc [bugzilla] style = mapfile template = {foo} But the old behavior would be a bug because bugzilla.template is documented as "overrides style if specified". Also, common log-like templates and formatter doesn't allow using mapfile-keywords in a separate template. So I decided to make a BC. Since there was no test for the bugzilla extension, this adds new test that covers style/template output.

subrepo: disable localizations when calling Git (issue5176) Spotted by Aidar Sayfullin.

py3: make test-demandimport use print_function Replacing print statements with print function.

py3: use absolute_import in svnxml.py

py3: use absolute_import in sitecustomize.py

py3: use absolute_import in revlog-formatv0.py

py3: use absolute_import in printenv.py

py3: use absolute_import in mockblackbox.py

py3: use absolute_import in killdaemons.py

test-shelve: shorten a long path so it works on Windows

test-convert-git: skip tests with invalid path characters on Windows

test-subrepo-git: add globs for Windows

hook: report untrusted hooks as failure (issue5110) (BC) Before this patch, there was no way for a repository owner to ensure that validation hooks would be run by people with write access. If someone had write access but did not trust the user owning the repository, the config and its hook would simply be ignored. After this patch, hooks from untrusted configs are taken into account but never actually run. Instead they are reported as failures right away. This will ensure validation performed by a hook is not ignored. As a side effect writer can be forced to trust a repository hgrc by adding a 'pretxnopen.trust=true' hook to the file. This was discussed during the 3.8 sprint with Matt Mackall, Augie Fackler and Kevin Bullock.

hook: split config reading further We want an easy way to fetch the hook config with and without honoring "trusted" so that we can compare the values. So we extract the part retrieving raw hook data from the config to be able to call it twice in the next patch.

hook: small refactor to store hooks as dict instead of list We are about to take untrusted hooks into account (to report them as failures) so we need to rearrange the code a bit to allow config overwriting each other in a later patch.

mail: retain hostname for sslutil.wrapsocket (issue5203) SMTPS + STARTTLS need to provide serverhostname, and we can't store it in sslkwargs because that breaks something involving the https protocol.