sslutil: make cert fingerprints messages more actionable The previous warning and abort messages were difficult to understand. This patch makes them slightly better. I think there is still room to tweak the messaging. And as we adopt new security defaults, these messages will certainly change again. But at least this takes us a step in the right direction. References to "section" have been removed because if no fingerprint is defined, "section" can never be "hostfingerprints." So just print "hostsecurity" every time.

sslutil: refactor code for fingerprint matching We didn't need to use a temporary variable to indicate success because we just return anyway. This refactor makes the code simpler. While we're here, we also call into formatfingerprint() to ensure the fingerprint from the proper hashing algorithm is logged.

sslutil: print SHA-256 fingerprint by default The world is starting to move on from SHA-1. A few commits ago, we gained the ability to define certificate fingerprints using SHA-256 and SHA-512. Let's start printing the SHA-256 fingerprint instead of the SHA-1 fingerprint to encourage people to pin with a more secure hashing algorithm. There is still a bit of work to be done around the fingerprint messaging. This will be addressed in subsequent commits.

sslutil: move and change warning when cert verification is disabled A short time ago, validatesocket() didn't know the reasons why cert verification was disabled. Multiple code paths could lead to cert verification being disabled. e.g. --insecure and lack of loaded CAs. With the recent refactorings to sslutil.py, we now know the reasons behind security settings. This means we can recognize when the user requested security be disabled (as opposed to being unable to provide certificate verification due to lack of CAs). This patch moves the check for certificate verification being disabled and changes the wording to distinguish it from other states. The warning message is purposefully more dangerous sounding in order to help discourage people from disabling security outright. We may want to add a URL or hint to this message. I'm going to wait until additional changes to security defaults before committing to something.

sslutil: add devel.disableloaddefaultcerts to disable CA loading There are various tests for behavior when CA certs aren't loaded. Previously, we would pass --insecure to disable loading of CA certs. This has worked up to this point because the error message for --insecure and no CAs loaded is the same. Upcoming commits will change the error message for --insecure and will change behavior when CAs aren't loaded. This commit introduces the ability to disable loading of CA certs by setting devel.disableloaddefaultcerts. This allows a testing backdoor to disable loading of CA certs even if system/default CA certs are available. The flag is purposefully not exposed to end-users because there should not be a need for this in the wild: certificate pinning and --insecure provide workarounds to disable cert loading/validation. Tests have been updated to use the new method. The variable used to disable CA certs has been renamed because the method is not OS X specific.

sslutil: store flag for whether cert verification is disabled This patch effectively moves the ui.insecureconnections check to _hostsettings(). After this patch, validatesocket() no longer uses the ui instance for anything except writing messages. This patch also enables us to introduce a per-host config option for disabling certificate verification.

sslutil: remove "strict" argument from validatesocket() It was only used by mail.py as part of processing smtp.verifycert, which was just removed.

mail: unsupport smtp.verifycert (BC) smtp.verifycert was accidentally broken by cca59ef27e60. And, I believe the "loose" value has been broken for longer than that. The current code refuses to talk to a remote server unless the CA is trusted or the fingerprint is validated. In other words, we lost the ability for smtp.verifycert to lower/disable security. There are special considerations for smtp.verifycert in sslutil.validatesocket() (the "strict" argument). This violates the direction sslutil is evolving towards, which has all security options determined at wrapsocket() time and a unified code path and configs for determining security options. Since smtp.verifycert is broken and since we'll soon have new security defaults and new mechanisms for controlling host security, this patch formally deprecates smtp.verifycert. With this patch, the socket security code in mail.py now effectively mirrors code in url.py and other places we're doing socket security. For the record, removing smtp.verifycert because it was accidentally broken is a poor excuse to remove it. However, I would have done this anyway because smtp.verifycert is a one-off likely used by few people (users of the patchbomb extension) and I don't think the existence of this seldom-used one-off in security code can be justified, especially when you consider that better mechanisms are right around the corner.

update: fix bare --clean to work on new branch (issue5003) (BC) Before this commit bare update --clean on newly created branch updates to the parent commit, even if there are later commits on the parent commit's branch. Update to the latest head on the parent commit's branch instead. This seems reasonable as clean should discard uncommited changes, branch is one of them.

revert: use "discard"/"revert" verb when reverting interactively (issue5143) Instead of "record this change to 'FILE'?" now prompt with: * "discard this change to 'FILE'?" when reverting to the parent of working directory, and, * "revert this change to 'FILE'?" otherwise.

run-tests: add support for RTUNICODEPEDANTRY environment variable based on 73e4a02e6d23

obsolete: fix grammar

tests: add run-test .testtimes basic testing

check-code: make repquote distinguish more characters for exact detection This patch makes repquote() distinguish more characters below, as a preparation for exact detection in subsequent patch. - "%" as "%" - "\\" as "b"(ackslash) - "*" as "A"(sterisk) - "+" as "P"(lus) - "-" as "M"(inus) Characters other than "%" don't use itself as replacement, because they are treated as special ones in regexp.

check-code: centralize rules depending on implementation of repquote This decreases the cost of checking which regexp should be adjusted at change of repquote().

check-code: use fixedmap for replacement of space characters This can centralize management of fixed replacement into fixedmap.

check-code: replace quoted characters correctly 169cb9e47f8e tried to detect '.. note::' more exactly. But implementation of it seems not correct, because: - fromc.find(c) returns -1 for other than "." and ":" - tochr[-1] returns "q" for such characters, but - expected result for them is "o" This patch uses dict to manage replacement instead of replacing str.find() by str.index(), for improvement/refactoring in subsequent patches. Examination by fixedmap is placed just after examination for ' ' and '\n', because subsequent patch will integrate the latter into the former. This patch also changes regexp for 'string join across lines with no space' rule, and adds detailed test for it, because 169cb9e47f8e did: - make repquote() distinguish "." (as "p") and ":" (as "q") from others (as "o"), but - not change this regexp without any reason (in commit log, at least), even though this regexp depends on what "o" means This patch doesn't focuses on deciding whether "." and/or ":" should be followed by whitespace or not in translatable messages.

test-chg: add basic tests for server lifecycle I'm going to move around the codes in AutoExitMixIn. This test should catch a subtle bug of unlinking sockets which I made in draft patches.

test-chg: run only with chg It doesn't make sense to run test-chg.t without chg, so ignore it with vanilla hg, and specify chg executable explicitly. test-chg.t can host chg-specific tests.

distate: add assertions to backup functions Those assertions will prevent the backup functions from overwriting the dirstate file in case both: suffix and prefix are empty. (foozy suggested making that change and I agree with him)

Added signature for changeset a9764ab80e11

Added tag 3.8.3 for changeset a9764ab80e11

shelve: use backup functions instead of manually copying dirstate This increases encapsulation of dirstate: the dirstate file is private to the dirstate module and shouldn't be touched by extensions directly.

dirstate: don't use actualfilename to name the backup file The issue with using actualfilename is that dirstate saved during transaction with "pending" in filename will be impossible to recover from outside of the transaction because the recover method will be looking for the name without "pending".

sslutil: reference appropriate config section in messaging Error messages reference the config section defining the host fingerprint. Now that we have multiple sections where this config setting could live, we need to point the user at the appropriate one. We default to the new "hostsecurity" section. But we will still refer them to the "hostfingerprint" section if a value is defined there. There are some corner cases where the messaging might be off. e.g. they could define a SHA-1 fingerprint in both sections. IMO the messaging needs a massive overhaul. I plan to do this as part of future refactoring to security settings.

sslutil: allow fingerprints to be specified in [hostsecurity] We introduce the [hostsecurity] config section. It holds per-host security settings. Currently, the section only contains a "fingerprints" option, which behaves like [hostfingerprints] but supports specifying the hashing algorithm. There is still some follow-up work, such as changing some error messages.

debuginstall: expose modulepolicy With this, you can check for pure easily: $ HGMODULEPOLICY=py ./hg debuginstall -T "{hgmodulepolicy}" py

revset: define table of sort() key functions This should be more readable than big "if" branch.

revset: factor out reverse flag of sort() key Prepares for making a table of sort keys. This assumes 'k' has at least one character, which should be guaranteed by keys.split().

tests: don't save host fingerprints in hgrc Previously, the test saved the host fingerprints in hgrc. Many tests override the fingerprint at run-time. This was a bit dangerous and was too magical for my liking. It will also interfere with a future patch that adds a new source for obtaining fingerprints. So change the test to require the fingerprint on every command invocation.

sslutil: calculate host fingerprints from additional algorithms Currently, we only support defining host fingerprints with SHA-1. A future patch will introduce support for defining fingerprints using other hashing algorithms. In preparation for that, we rewrite the fingerprint verification code to support multiple fingerprints, namely SHA-256 and SHA-512 fingerprints. We still only display the SHA-1 fingerprint. We'll have to revisit this code once we support defining fingerprints with other hash functions. As part of this, I snuck in a change to use range() instead of xrange() because xrange() isn't necessary for such small values.

util: add sha256 Upcoming patches will teach host fingerprint checking to verify non-SHA1 fingerprints. Many x509 certificates these days are SHA-256. And modern browsers often display the SHA-256 fingerprint for certificates. Since SHA-256 fingerprints are highly visible and easy to obtain, we want to support them for fingerprint pinning. So add SHA-256 support to util. I did not add SHA-256 to DIGESTS and DIGESTS_BY_STRENGTH because this will advertise the algorithm on the wire protocol. I wasn't sure if that would be appropriate. I'm playing it safe by leaving it out for now.

sslutil: move CA file processing into _hostsettings() The CA file processing code has been moved from _determinecertoptions into _hostsettings(). As part of the move, the logic has been changed slightly and the "cacerts" variable has been renamed to "cafile" to match the argument used by SSLContext.load_verify_locations(). Since _determinecertoptions() no longer contains any meaningful code, it has been removed.

sslutil: move SSLContext.verify_mode value into _hostsettings _determinecertoptions() and _hostsettings() are redundant with each other. _hostsettings() is used the flexible API we want. We start the process of removing _determinecertoptions() by moving some of the logic for the verify_mode value into _hostsettings(). As part of this, _determinecertoptions() now takes a settings dict as its argument. This is technically API incompatible. But since _determinecertoptions() came into existence a few days ago as part of this release, I'm not flagging it as such.

sslutil: introduce a function for determining host-specific settings This patch marks the beginning of a series that introduces a new, more configurable, per-host security settings mechanism. Currently, we have global settings (like web.cacerts and the --insecure argument). We also have per-host settings via [hostfingerprints]. Global security settings are good for defaults, but they don't provide the amount of control often wanted. For example, an organization may want to require a particular CA is used for a particular hostname. [hostfingerprints] is nice. But it currently assumes SHA-1. Furthermore, there is no obvious place to put additional per-host settings. Subsequent patches will be introducing new mechanisms for defining security settings, some on a per-host basis. This commits starts the transition to that world by introducing the _hostsettings function. It takes a ui and hostname and returns a dict of security settings. Currently, it limits itself to returning host fingerprint info. We foreshadow the future support of non-SHA1 hashing algorithms for verifying the host fingerprint by making the "certfingerprints" key a list of tuples instead of a list of hashes. We add this dict to the hgstate property on the socket and use it during socket validation for checking fingerprints. There should be no change in behavior.

tests-subrepo-git: emit a different "pwned" message based on the test Having a single "pwned" message which may or may not be emitted during the tests for CVE-2016-3068 leads to extra confusion. Allow each test to emit a more detailed message based on what the expectations are. In both cases, we expect a version of git which has had the vulnerability plugged, as well as a version of mercurial which also knows about GIT_ALLOW_PROTOCOL. For the first test, we make sure GIT_ALLOW_PROTOCOL is unset, meaning that the ext-protocol subrepo should be ignored; if it isn't, there's either a problem with mercurial or the installed copy of git. For the second test, we explicitly allow ext-protocol subrepos, which means that the subrepo will be accessed and a message emitted confirming that this was, in fact, our intention.

tests-subrepo-git: make the "pwned" message output in a stable order The "pwned" message from this test gets gets sent to stderr, and so may get emitted in different places from run to run in the rest of mercurial's output. This patch forces the message to go to a specific file instead, whose existence and contents we can examine at a stable point in the test's execution.

test-cache-abuse: correct for different hunk headers between Solaris and GNU When diffing against an empty file, Solaris diff uses 1 to designate the first line of the empty file (either -1,0 on the left or +1,0 on the right) while GNU diff uses 0 (-0,0 and +0,0). We use a glob here to make sure the test passes with either toolchain. I've not added tests to check-code because there are scads of places in the tests where the GNU format is used due to that being the format that "hg diff" and "hg export" use, and changing those to use globs seems wrong.

lazymanifest: fix typo s/typles/tuples/

sslutil: remove sslkwargs() (API) It is now unused.

url: remove use of sslkwargs

mail: remove use of sslkwargs

httpconnection: remove use of sslkwargs It now does nothing.

sslutil: move sslkwargs logic into internal function (API) As the previous commit documented, sslkwargs() doesn't add any value since its return is treated as a black box and proxied to wrapsocket(). We formalize its uselessness by moving its logic into a new, internal function and make sslkwargs() return an empty dict. The certificate arguments that sslkwargs specified have been removed from wrapsocket() because they should no longer be set.

sslutil: remove ui from sslkwargs (API) Arguments to sslutil.wrapsocket() are partially determined by calling sslutil.sslkwargs(). This function receives a ui and a hostname and determines what settings, if any, need to be applied when the socket is wrapped. Both the ui and hostname are passed into wrapsocket(). The other arguments to wrapsocket() provided by sslkwargs() (ca_certs and cert_reqs) are not looked at or modified anywhere outside of sslutil.py. So, sslkwargs() doesn't need to exist as a separate public API called before wrapsocket(). This commit starts the process of removing external consumers of sslkwargs() by removing the "ui" key/argument from its return. All callers now pass the ui argument explicitly.

dirstate: remove file from copymap on drop As the copymap is short-lived object regenerated from dirstate on each read this didn't affect us in any serious way. But since I've started working on permanent storage of copymap in my experiments with sqldirstate[1] I've seen this bug leaving the copy information in copymap after reverting the file moves and copies. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

run-tests: use json.dumps(separators=) Followup to daff05dcd184 per Martijn Pieters

debugignore: make messages translatable These messages have been overlooked by check-code, because they start with non-alphabet character ('%' or '('). Making these messages translatable seems reasonable, because messages for ui.note(), ui.status(), ui.progress() and descriptive messages for ui.write() in "debug" commands are already translatable in many cases. This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

grep: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character (' '). This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

subrepo: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('%'). This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

merge: make messages translatable These messages have been overlooked by check-code, because they start with non-alphabet character (' '). Making these messages translatable seems reasonable, because all other 'ui.note()'-ed messages in calculateupdates() are already translatable. This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

httppeer: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('('). Making this message translatable seems reasonable, because exception message below in same function is already translatable - 'cannot create new http repository' This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

notify: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('\'). Making this message translatable seems reasonable, because messages below in same function are already translatable - '\ndiffs (truncated from %d to %d lines):\n\n' - '\ndiffs (%d lines):\n\n' This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

gpg: make a message translatable This message has been overlooked by check-code, because it starts with non-alphabet character ('%'). This is also a part of preparation for making "missing _() in ui message" detection of check-code more exact.

revset: use getargsdict for sort() This makes it possible to use keyword arguments to specify per-sort options. For example, a hypothetical 'first' option for the user sort could sort certain users first with: sort(all(), user, user.first=mpm@selenic.com)

merge with stable

changegroup: extract method that sorts nodes to send The current implementation of narrowhg needs to influence the order in which nodes are sent to the client. adgar@ and I think this is fixable, but it's going to require pretty substantial time investment, so in the interim we'd like to extract this method. I think it makes the group() code a little more obvious, as it took us a couple of tries to isolate the exact behavior we were observing.

hg: disable demandimport for py3

tests: enable import checker for all python files (including no .py files) i18n/posplit is excluded as it couldn't be trivially fixed. That's the same as 99a2bdad0fda.

tests: make 'f' utility import hashlib unconditionally It must exist on Python 2.5+.

tests: fix typo of shebang prefix in test-check-pyflakes.t

hghave: silence future pyflakes warning of unused import

tests: remove unused import from 'f' utility It should have been caught by pyflakes.

bookmarks: allow pushing active bookmark on new remote head (issue5236) Before 'hg push -B .' on new remote head complained with: abort: push creates new remote head ... It was because _nowarnheads was not expanding active bookmark name, so it didn't add active bookmark "proper" name to no warn heads list.

sslutil: remove redundant check of sslsocket.cipher() We are doing this check in both wrapsocket() and validatesocket(). The check was added to the validator in 4bb59919c905 and the commit message justifies the redundancy with a "might." The check in wrapsocket() was added in 0cc4ad757c77, which appears to be part of the same series. I'm going to argue the redundancy isn't needed. I choose to keep the check in wrapsocket() because it is working around a bug in Python's wrap_socket() and I feel the check for the bug should live next to the function call exhibiting the bug.

sslutil: convert socket validation from a class to a function (API) Now that the socket validator doesn't have any instance state, we can make it a generic function. The "validator" class has been converted into the "validatesocket" function and all consumers have been updated.

sslutil: store and use hostname and ui in socket instance Currently, we pass a hostname and ui to sslutil.wrap_socket() then create a separate sslutil.validator instance also from a hostname and ui. There is a 1:1 mapping between a wrapped socket and a validator instance. This commit lays the groundwork for making the validation function generic by storing the hostname and ui instance in the state dict attached to the socket instance and then using these variables in the validator function. Since the arguments to sslutil.validator.__init__ are no longer used, we make them optional and make __init__ a no-op.

sslutil: use a dict for hanging hg state off the wrapped socket I plan on introducing more state on the socket instance. Instead of using multiple variables, let's just use one to minimize risk of name collision.

sslutil: require serverhostname argument (API) All callers now specify it. So we can require it. Requiring the argument means SNI will always work if supported by Python. The main reason for this change is to store state on the socket instance to make the validation function generic. This will be evident in subsequent commits.

annotate: optimize line counting We used len(text.splitlines()) to count lines. This allocates, copies, and deallocates an object for every line in a file. Instead, we use count("\n") to count newlines and adjust based on whether there's a trailing newline. This improves the speed of annotating localrepo.py from 4.2 to 4.0 seconds.

purge: use opts.get() Most commands use opts.get() to retrieve values for options that may not be explicitly passed. purge wasn't. This makes it easier to call purge() from 3rd party extensions.

test-run-tests: clean up inuse server eagerly

tests: refactor run-tests helpers test-run-tests-rev.t will need them

tests: silence test-repo obsolete warning refactoring test-check-commit.t HGRCPATH bits as helpers-testrepo.sh

largefiles: send statlfile remote calls only for nonexisting locally files Files that are already in local store should be checked locally. The problem with this implementation is how difference in messages between local and remote checks should look like. For now local errors for file missing and content corrupted looks like this: 'changeset cset: filename references missing storepath\n' 'changeset cset: filename references corrupted storepath\n' for remote it looks like: 'changeset cset: filename missing\n' 'changeset cset: filename: contents differ\n' Contents differ error for remote calls is never raised currently - for now statlfile implementation lacks checking file content.

check-code: reject .next(...)

py3: convert to next() function next(..) was introduced in py2.6 and .next() is not available in py3 https://docs.python.org/2/library/functions.html#next

revset: rename variable to avoid shadowing with builtin next() function https://docs.python.org/2/library/functions.html#next

histedit: add experimental config for using the first word of the commit This allows users to start a commit with "verb! ..." so that when this is opened in histedit, the default action will be "verb". For example, "roll! foo" will default to the action "roll". Currently, we'll allow any known verb to be used but this is experimental.

histedit: add optional parameter for determining intial editor line A simple refactor to allow us to change the default verb for the initial editor display.

tests: enable import checker for all **.py files Several known-bad files are excluded as they couldn't be trivially fixed. That's the same as 99a2bdad0fda.

py3: make contrib/import-checker.py get along with itself Indent these imports to disable the rule of "not lexically sorted."

py3: make contrib/revsetbenchmarks.py not import symbols from stdlib modules

py3: make contrib/bdiff-torture.py conform to our import style

import-checker: extend check of symbol-import order to all local modules It doesn't make sense that (a) is allowed whereas (b) is disallowed. a) from mercurial import hg from mercurial.i18n import _ b) from . import hg from .i18n import _

import-checker: always build a list of imported symbols The next patch will rely on it.

import-checker: fix test to make a real package Otherwise "testpackage" wouldn't be counted as a package when building a list of imported symbols.

py3: move up symbol imports to enforce import-checker rules Since (b) is banned, we should do the same for (a) for consistency. a) from mercurial import hg from mercurial.i18n import _ b) from . import hg from .i18n import _

util: make copyfile avoid ambiguity of file stat if needed In some cases below, copying from backup is used to restore original contents of a file. If copying keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - failure of transaction before closing (from '.hg/journal.backup.*') - rollback of previous transaction (from '.hg/undo.backup.*') To avoid such problem, this patch makes copyfile() avoid ambiguity of file stat, if needed. Ambiguity check is executed, only if: - checkambig=True is specified (not all copying needs ambiguity check), and - destination file exists before copying This patch also adds 'not (copystat and checkambig)' assertion, because combination of copystat and checkambig is meaningless. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

vfs: make rename avoid ambiguity of file stat if needed In some cases below, renaming from backup is used to restore original contents of a file. If renaming keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - failure of transaction before closing (only from '.hg/journal.dirstate') - rollback of previous transaction (from '.hg/undo.*') - failure in dirstateguard scope (from '.hg/dirstate.SUFFIX') To avoid such problem, this patch makes vfs.rename() avoid ambiguity of file stat, if needed. Ambiguity check is executed, only if: - checkambig=True is specified (not all renaming needs ambiguity check), and - destination file exists before renaming This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

vfs: make atomictempfile avoid ambiguity of file stat if needed This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: make atomictempfile avoid ambiguity of file stat if needed Ambiguity check is executed at close(), only if: - atomictempfile is created with checkambig=True, and - target file exists before renaming This restriction avoids performance decrement by needless examination of file stat (for example, filelog doesn't need exact cache validation, even though it uses atomictempfile to write changes out). See description of filestat class for detail about why the logic in this patch works as expected. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: add filestat class to detect ambiguity of file stat Current posix.cachestat implementation might overlook change of a file, if changing keeps ctime, mtime and size of file. Comparison of inode number also overlooks changing in such situation, because inode number is rapidly reused. Contents of a file cached before changing isn't invalidated as expected, if change of a file is overlooked for this "ambiguity" of file stat. This patch adds filestat class to detect ambiguity of file stat. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

run-tests: handle json.dumps divergence In py2, json.dumps includes a trailing space after a comma at the end of lines. The py3 behavior which omits the trailing space is preferable, so we're going to strip it.

tests: use debuginstall to retrieve hg version

debuginstall: add mercurial version

strip: invalidate phase cache after stripping changeset (issue5235) When we remove a changeset from the changelog, the phase cache must be invalidated, otherwise it could refer to changesets that are no longer in the repo. To reproduce the failure, I created an extension querying the phase cache after the strip transaction is over. To do that, I stripped two commits with a bookmark on one of them to force another transaction (we open a transaction for moving bookmarks) after the strip transaction. Without the fix in this patch, the test leads to a stacktrace showing the issue: repair.strip(ui, repo, revs, backup) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/repair.py", line 205, in strip tr.close() File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/transaction.py", line 44, in _active return func(self, *args, **kwds) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/transaction.py", line 490, in close self._postclosecallback[cat](self) File "$TESTTMP/crashstrip2.py", line 4, in test [repo.changelog.node(r) for r in repo.revs("not public()")] File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/changelog.py", line 337, in node return super(changelog, self).node(rev) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/revlog.py", line 377, in node return self.index[rev][7] IndexError: revlog index out of range The situation was encountered in inhibit (evolve's repo) where we would crash following the volatile set invalidation submitted by Augie in e6f490e328635312ee214a12bc7fd3c7d46bf9ce. Before his patch the issue was masked as we were not accessing the phasecache after stripping a revision. This bug uncovered another but in histedit (see explanation in issue5235). I changed the histedit test accordingly to avoid fixing two things at once.

py3: make tests/svn-safe-append.py use absolute_import

py3: make tests/test-atomictempfile.py use absolute_import

py3: tests/test-check-py3-compat.t output updated The lower part of the tests runs with Python 3.5 so its remains unchanged with new commits.

py3: use setattr() to assign new class attribute The old method produces error 'object does not supports item assignment'. So setattr() is used to assign a new class attribute via __dict__ .

localrepo: use dirstate savebackup instead of handling dirstate file manually This is one step towards having dirstate manage its own storage. It will be useful for the implementation of sql dirstate [1]. This introduced a small test change: now we always write the dirstate before saving backup so in some cases where dirstate file didn't exist yet savebackup can create it. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

localrepo: use dirstate restorebackup instead of copying dirstate manually This is one step towards having dirstate manage its own storage. It will be useful for the implementation of sqldirstate [1]. I'm deleting two of the dirstate.invalidate() calls in localrepo because restorebackup method does that for us. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

dirstate: add prefix and suffix arguments to backup This would allow the code explicitly copying dirstate to use this method instead. Use of this method will increase encapsulation (the dirstate class will be sole owner of its on-disk storage).

tests: mark test-atomictempfile.py write as binary

tests: mark test-context.py write as binary

transaction: turn lack of locking into a hard failure (API) We have been warning about transactions without locks for about a year (and three releases), third party extensions had a fair grace period to fix their code, we are moving lack of locking to a hard failure in order to protect users against repository corruption.

test: extract develwarn transaction testing in its own command The lack of locking for a transation is about to change from a warning to an error. We first extract the test decidated to this warning to make the next changeset clearer.

graphmod: update edgemap in-place The edgemap update was not actually propagated to future asciiedge calls; update the edge state dictionary in-place instead.

tests: test-archive.t use mercurial.util for urllib compat

tests: test-archive.t use sys.stdout.buffer for binary output in py3

localrepo: prevent executable-bit only changes from being lost on amend If you have just executable-bit change and amend it twice it will vanish: * After the first amend the commit will have the proper executable bit set in manifest but it won't have the the file on the list of files in changelog. * The second amend will read the wrong list of files from changelog and it will copy the manifest entry from parent for this file. * Voila! The change is lost. This change repairs the bug in localrepo causing this and adds a test for it.

tests: escape bytes setting MSB in input of grep for portability GNU grep (2.21-2 or later) assumes that input is encoded in LC_CTYPE, and input is binary if it contains byte sequence not valid for that encoding. For example, if locale is configured as C, a byte setting most significant bit (MSB) makes such GNU grep show "Binary file <FILENAME> matches" message instead of matched lines unintentionally. This behavior is recognized as a bug, and fixed in GNU grep 2.25-1 or later. But some distributions are shipped with such buggy version (e.g. Ubuntu xenial, which is used by launchpad buildbot). http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19230 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800670 http://packages.ubuntu.com/xenial/grep This causes failure of test-commit-interactive.t, which applies grep on CP932 byte sequence since 1111e84de635. But, explicit setting LC_CTYPE for CP932 might cause another problem, because it can't be assumed that all environment running Mercurial tests allows arbitrary locale setting. To resolve this issue, this patch escapes bytes setting MSB in input of grep. For this purpose: - str.encode('string-escape') isn't useful, because it escapes also control code (less than 0x20), and makes EOL handling complicated - "f --hexdump" isn't useful, because it isn't line-oriented - "sed -n" seems reasonable, but "sed" itself sometimes causes portability issue, too (e.g. 900767dfa80d or afb86ee925bf) This patch is posted with "stable" flag, because 1111e84de635 is on stable branch.

tests: test-archive.t use absolute_import This is a step to adding a mercurial dependency to simplify py3 compat

tests: test-archive.t use open() instead of file() for py3 compat

tests: test-archive.t use print_function

merge with stable

readlink: use print_function

tests: test-addremove-similar.t use print() for py3

tests: add coverage for run-tests.py --whitelist