test-revset: fix test vector for ordering issue of matching() 592e0beee8b0 fixed matching() to preserve the order of the input set, but the test was incorrect. Given "A and B", "A" should be the input set to "B". But thanks to our optimizer, the test expression was rewritten as "(2 or 3 or 1) and matching(1 or 2 or 3)", therefore it was working well. Since I'm going to fix the overall ordering issue, the test needs to be adjusted to do the right thing.

largefiles: rename match_ to matchmod import in lfutil

largefiles: rename match_ to matchmod import in reposetup

largefiles: rename match_ to matchmod import in overrides

largefiles: rename match_ to matchmod import in lfcommands

py3: make largefiles/wirestore.py use absolute_import

py3: make largefiles/uisetup.py use absolute_import

py3: make largefiles/reposetup.py use absolute_import

py3: make largefiles/remotestore.py use absolute_import

py3: make largefiles/proto.py use absolute_import

py3: make largefiles/overrides.py use absolute_import

py3: make largefiles/localstore.py use absolute_import

py3: make largefiles/lfutil.py use absolute_import

py3: make largefiles/lfcommands.py use absolute_import

py3: make largefiles/basestore.py use absolute_import

py3: make largefiles/__init__.py use absolute_import

largefiles: move basestore._openstore into new module to remove cycle

revset: make filteredset.__nonzero__ respect the order of the filteredset This fix allows __nonzero__ to respect the direction of iteration of the whole filteredset. Here's the case when it matters. Imagine that we have a very large repository and we want to execute a command like: $ hg log --rev '(tip:0) and user(ikostia)' --limit 1 (we want to get the latest commit by me). Mercurial will evaluate a filteredset lazy data structure, an instance of the filteredset class, which will know that it has to iterate in a descending order (isdescending() will return True if called). This means that when some code iterates over the instance of this filteredset, the 'and user(ikostia)' condition will be first checked on the latest revision, then on the second latest and so on, allowing Mercurial to print matches as it founds them. However, cmdutil.getgraphlogrevs contains the following code: revs = _logrevs(repo, opts) if not revs: return revset.baseset(), None, None The "not revs" expression is evaluated by calling filteredset.__nonzero__, which in its current implementation will try to iterate the filteredset in ascending order until it finds a revision that matches the 'and user(..' condition. If the condition is only true on late revisions, a lot of useless iterations will be done. These iterations could be avoided if __nonzero__ followed the order of the filteredset, which in my opinion is a sensible thing to do here. The problem gets even worse when instead of 'user(ikostia)' some more expensive check is performed, like grepping the commit diff. I tested this fix on a very large repo where tip is my commit and my very first commit comes fairly late in the revision history. Results of timing of the above command on that very large repo. -with my fix: real 0m1.795s user 0m1.657s sys 0m0.135s -without my fix: real 1m29.245s user 1m28.223s sys 0m0.929s I understand that this is a very specific kind of problem that presents itself very rarely, only on very big repositories and with expensive checks and so on. But I don't see any disadvantages to this kind of fix either.

phases: make writing phaseroots file out avoid ambiguity of file stat Cached attribute repo._phasecache uses stat of '.hg/phaseroots' file to examine validity of cached contents. If writing '.hg/phaseroots' file out keeps ctime, mtime and size of it, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/phaseroots' file out with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

dirstate: make writing branch file out avoid ambiguity of file stat Cached attribute dirstate._branch uses stat of '.hg/branch' file to examine validity of cached contents. If writing '.hg/branch' file out keeps ctime, mtime and size of it, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/branch' file out with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

dirstate: make writing dirstate file out avoid ambiguity of file stat Cached attribute repo.dirstate uses stat of '.hg/dirstate' file to examine validity of cached contents. If writing '.hg/dirstate' file out keeps ctime, mtime and size of it, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/dirstate' file out with checkambig=True. The former diff hunk changes the code path for "dirstate.write()", and the latter changes the code path for "dirstate.savebackup()". This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

bookmarks: make writing files out avoid ambiguity of file stat Cached attribute repo._bookmarks uses stat of '.hg/bookmarks' and '.hg/bookmarks.current' files to examine validity of cached contents. If writing these files out keeps ctime, mtime and size of them, change is overlooked, and old contents cached before change isn't invalidated as expected. To avoid ambiguity of file stat, this patch writes '.hg/bookmarks' and '.hg/bookmarks.current' files out with checkambig=True. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

transaction: avoid ambiguity of file stat at closing transaction Files below, which might be changed at closing transaction, are used to examine validity of cached properties. If changing keeps ctime, mtime and size of a file, change is overlooked, and old contents cached before change isn't invalidated as expected. - .hg/bookmarks - .hg/dirstate - .hg/phaseroots To avoid ambiguity of file stat, this patch writes files out with checkambig=True at closing transaction. checkambig becomes True only at closing (= 'not suffix'), because stat information of '.pending' file isn't used to examine validity of cached properties. This patch is a part of "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: add __ne__ to filestat class for consistency This is follow up for ca4065028e00, which introduced filestat class.

style: remove namespace class For better or worse, our coding do not use use class for pure namespacing. We remove the class introduced in a5009789960c.

style: don't use capital letter for constant For better or worse, our coding do not use all caps for constants. We rename constant name introduced in a5009789960c.

tests-subrepo-git: use "f" to dump pwned.txt, for portability Rather than sometimes using a complicated shell construct to dump pwned.txt (if it wasn't expected to exist, but might, if something were broken) or just cat (if it was expected to exist), just use the "f" utility, which will be consistent in its behavior across different platforms. Also make sure that *something* gets put into pwned.txt, even if we ended up typoing the message variable.

bundle2: don't assume ordering of heads checked after push Usually, the heads will have the same ordering in handlecheckheads. Insisting on the same ordering is however an unnecessary constraint that in some custom cases can cause pushes to fail even though the actual heads didn't change. This caused production issues for us in combination with the current version of https://bitbucket.org/Unity-Technologies/hgwebcachingproxy/ .

sslutil: print the fingerprint from the last hash used Before, we would always print the unprefixed SHA-1 fingerprint when fingerprint comparison failed. Now, we print the fingerprint of the last hash used, including the prefix if necessary. This helps ensure that the printed hash type matches what is in the user configuration. There are still some cases where this can print a mismatched hash type. e.g. if there are both SHA-1 and SHA-256 fingerprints in the config, we could print a SHA-1 hash if it comes after the SHA-256 hash. But I'm inclined to ignore this edge case. While I was here, the "section" variable assignment has been moved to just above where it is used because it is now only needed for this error message and it makes the code easier to read.

sslutil: make cert fingerprints messages more actionable The previous warning and abort messages were difficult to understand. This patch makes them slightly better. I think there is still room to tweak the messaging. And as we adopt new security defaults, these messages will certainly change again. But at least this takes us a step in the right direction. References to "section" have been removed because if no fingerprint is defined, "section" can never be "hostfingerprints." So just print "hostsecurity" every time.

sslutil: refactor code for fingerprint matching We didn't need to use a temporary variable to indicate success because we just return anyway. This refactor makes the code simpler. While we're here, we also call into formatfingerprint() to ensure the fingerprint from the proper hashing algorithm is logged.

sslutil: print SHA-256 fingerprint by default The world is starting to move on from SHA-1. A few commits ago, we gained the ability to define certificate fingerprints using SHA-256 and SHA-512. Let's start printing the SHA-256 fingerprint instead of the SHA-1 fingerprint to encourage people to pin with a more secure hashing algorithm. There is still a bit of work to be done around the fingerprint messaging. This will be addressed in subsequent commits.

sslutil: move and change warning when cert verification is disabled A short time ago, validatesocket() didn't know the reasons why cert verification was disabled. Multiple code paths could lead to cert verification being disabled. e.g. --insecure and lack of loaded CAs. With the recent refactorings to sslutil.py, we now know the reasons behind security settings. This means we can recognize when the user requested security be disabled (as opposed to being unable to provide certificate verification due to lack of CAs). This patch moves the check for certificate verification being disabled and changes the wording to distinguish it from other states. The warning message is purposefully more dangerous sounding in order to help discourage people from disabling security outright. We may want to add a URL or hint to this message. I'm going to wait until additional changes to security defaults before committing to something.

sslutil: add devel.disableloaddefaultcerts to disable CA loading There are various tests for behavior when CA certs aren't loaded. Previously, we would pass --insecure to disable loading of CA certs. This has worked up to this point because the error message for --insecure and no CAs loaded is the same. Upcoming commits will change the error message for --insecure and will change behavior when CAs aren't loaded. This commit introduces the ability to disable loading of CA certs by setting devel.disableloaddefaultcerts. This allows a testing backdoor to disable loading of CA certs even if system/default CA certs are available. The flag is purposefully not exposed to end-users because there should not be a need for this in the wild: certificate pinning and --insecure provide workarounds to disable cert loading/validation. Tests have been updated to use the new method. The variable used to disable CA certs has been renamed because the method is not OS X specific.

sslutil: store flag for whether cert verification is disabled This patch effectively moves the ui.insecureconnections check to _hostsettings(). After this patch, validatesocket() no longer uses the ui instance for anything except writing messages. This patch also enables us to introduce a per-host config option for disabling certificate verification.

sslutil: remove "strict" argument from validatesocket() It was only used by mail.py as part of processing smtp.verifycert, which was just removed.

mail: unsupport smtp.verifycert (BC) smtp.verifycert was accidentally broken by cca59ef27e60. And, I believe the "loose" value has been broken for longer than that. The current code refuses to talk to a remote server unless the CA is trusted or the fingerprint is validated. In other words, we lost the ability for smtp.verifycert to lower/disable security. There are special considerations for smtp.verifycert in sslutil.validatesocket() (the "strict" argument). This violates the direction sslutil is evolving towards, which has all security options determined at wrapsocket() time and a unified code path and configs for determining security options. Since smtp.verifycert is broken and since we'll soon have new security defaults and new mechanisms for controlling host security, this patch formally deprecates smtp.verifycert. With this patch, the socket security code in mail.py now effectively mirrors code in url.py and other places we're doing socket security. For the record, removing smtp.verifycert because it was accidentally broken is a poor excuse to remove it. However, I would have done this anyway because smtp.verifycert is a one-off likely used by few people (users of the patchbomb extension) and I don't think the existence of this seldom-used one-off in security code can be justified, especially when you consider that better mechanisms are right around the corner.

update: fix bare --clean to work on new branch (issue5003) (BC) Before this commit bare update --clean on newly created branch updates to the parent commit, even if there are later commits on the parent commit's branch. Update to the latest head on the parent commit's branch instead. This seems reasonable as clean should discard uncommited changes, branch is one of them.

revert: use "discard"/"revert" verb when reverting interactively (issue5143) Instead of "record this change to 'FILE'?" now prompt with: * "discard this change to 'FILE'?" when reverting to the parent of working directory, and, * "revert this change to 'FILE'?" otherwise.

run-tests: add support for RTUNICODEPEDANTRY environment variable based on 73e4a02e6d23

obsolete: fix grammar

tests: add run-test .testtimes basic testing

check-code: make repquote distinguish more characters for exact detection This patch makes repquote() distinguish more characters below, as a preparation for exact detection in subsequent patch. - "%" as "%" - "\\" as "b"(ackslash) - "*" as "A"(sterisk) - "+" as "P"(lus) - "-" as "M"(inus) Characters other than "%" don't use itself as replacement, because they are treated as special ones in regexp.

check-code: centralize rules depending on implementation of repquote This decreases the cost of checking which regexp should be adjusted at change of repquote().

check-code: use fixedmap for replacement of space characters This can centralize management of fixed replacement into fixedmap.

check-code: replace quoted characters correctly 169cb9e47f8e tried to detect '.. note::' more exactly. But implementation of it seems not correct, because: - fromc.find(c) returns -1 for other than "." and ":" - tochr[-1] returns "q" for such characters, but - expected result for them is "o" This patch uses dict to manage replacement instead of replacing str.find() by str.index(), for improvement/refactoring in subsequent patches. Examination by fixedmap is placed just after examination for ' ' and '\n', because subsequent patch will integrate the latter into the former. This patch also changes regexp for 'string join across lines with no space' rule, and adds detailed test for it, because 169cb9e47f8e did: - make repquote() distinguish "." (as "p") and ":" (as "q") from others (as "o"), but - not change this regexp without any reason (in commit log, at least), even though this regexp depends on what "o" means This patch doesn't focuses on deciding whether "." and/or ":" should be followed by whitespace or not in translatable messages.

test-chg: add basic tests for server lifecycle I'm going to move around the codes in AutoExitMixIn. This test should catch a subtle bug of unlinking sockets which I made in draft patches.

test-chg: run only with chg It doesn't make sense to run test-chg.t without chg, so ignore it with vanilla hg, and specify chg executable explicitly. test-chg.t can host chg-specific tests.

distate: add assertions to backup functions Those assertions will prevent the backup functions from overwriting the dirstate file in case both: suffix and prefix are empty. (foozy suggested making that change and I agree with him)

Added signature for changeset a9764ab80e11

Added tag 3.8.3 for changeset a9764ab80e11

shelve: use backup functions instead of manually copying dirstate This increases encapsulation of dirstate: the dirstate file is private to the dirstate module and shouldn't be touched by extensions directly.

dirstate: don't use actualfilename to name the backup file The issue with using actualfilename is that dirstate saved during transaction with "pending" in filename will be impossible to recover from outside of the transaction because the recover method will be looking for the name without "pending".

sslutil: reference appropriate config section in messaging Error messages reference the config section defining the host fingerprint. Now that we have multiple sections where this config setting could live, we need to point the user at the appropriate one. We default to the new "hostsecurity" section. But we will still refer them to the "hostfingerprint" section if a value is defined there. There are some corner cases where the messaging might be off. e.g. they could define a SHA-1 fingerprint in both sections. IMO the messaging needs a massive overhaul. I plan to do this as part of future refactoring to security settings.

sslutil: allow fingerprints to be specified in [hostsecurity] We introduce the [hostsecurity] config section. It holds per-host security settings. Currently, the section only contains a "fingerprints" option, which behaves like [hostfingerprints] but supports specifying the hashing algorithm. There is still some follow-up work, such as changing some error messages.

debuginstall: expose modulepolicy With this, you can check for pure easily: $ HGMODULEPOLICY=py ./hg debuginstall -T "{hgmodulepolicy}" py

revset: define table of sort() key functions This should be more readable than big "if" branch.

revset: factor out reverse flag of sort() key Prepares for making a table of sort keys. This assumes 'k' has at least one character, which should be guaranteed by keys.split().

tests: don't save host fingerprints in hgrc Previously, the test saved the host fingerprints in hgrc. Many tests override the fingerprint at run-time. This was a bit dangerous and was too magical for my liking. It will also interfere with a future patch that adds a new source for obtaining fingerprints. So change the test to require the fingerprint on every command invocation.

sslutil: calculate host fingerprints from additional algorithms Currently, we only support defining host fingerprints with SHA-1. A future patch will introduce support for defining fingerprints using other hashing algorithms. In preparation for that, we rewrite the fingerprint verification code to support multiple fingerprints, namely SHA-256 and SHA-512 fingerprints. We still only display the SHA-1 fingerprint. We'll have to revisit this code once we support defining fingerprints with other hash functions. As part of this, I snuck in a change to use range() instead of xrange() because xrange() isn't necessary for such small values.