sslutil: use a dict for hanging hg state off the wrapped socket I plan on introducing more state on the socket instance. Instead of using multiple variables, let's just use one to minimize risk of name collision.

sslutil: require serverhostname argument (API) All callers now specify it. So we can require it. Requiring the argument means SNI will always work if supported by Python. The main reason for this change is to store state on the socket instance to make the validation function generic. This will be evident in subsequent commits.

annotate: optimize line counting We used len(text.splitlines()) to count lines. This allocates, copies, and deallocates an object for every line in a file. Instead, we use count("\n") to count newlines and adjust based on whether there's a trailing newline. This improves the speed of annotating localrepo.py from 4.2 to 4.0 seconds.

purge: use opts.get() Most commands use opts.get() to retrieve values for options that may not be explicitly passed. purge wasn't. This makes it easier to call purge() from 3rd party extensions.

test-run-tests: clean up inuse server eagerly

tests: refactor run-tests helpers test-run-tests-rev.t will need them

tests: silence test-repo obsolete warning refactoring test-check-commit.t HGRCPATH bits as helpers-testrepo.sh

largefiles: send statlfile remote calls only for nonexisting locally files Files that are already in local store should be checked locally. The problem with this implementation is how difference in messages between local and remote checks should look like. For now local errors for file missing and content corrupted looks like this: 'changeset cset: filename references missing storepath\n' 'changeset cset: filename references corrupted storepath\n' for remote it looks like: 'changeset cset: filename missing\n' 'changeset cset: filename: contents differ\n' Contents differ error for remote calls is never raised currently - for now statlfile implementation lacks checking file content.

check-code: reject .next(...)

py3: convert to next() function next(..) was introduced in py2.6 and .next() is not available in py3 https://docs.python.org/2/library/functions.html#next

revset: rename variable to avoid shadowing with builtin next() function https://docs.python.org/2/library/functions.html#next

histedit: add experimental config for using the first word of the commit This allows users to start a commit with "verb! ..." so that when this is opened in histedit, the default action will be "verb". For example, "roll! foo" will default to the action "roll". Currently, we'll allow any known verb to be used but this is experimental.

histedit: add optional parameter for determining intial editor line A simple refactor to allow us to change the default verb for the initial editor display.

tests: enable import checker for all **.py files Several known-bad files are excluded as they couldn't be trivially fixed. That's the same as 99a2bdad0fda.

py3: make contrib/import-checker.py get along with itself Indent these imports to disable the rule of "not lexically sorted."

py3: make contrib/revsetbenchmarks.py not import symbols from stdlib modules

py3: make contrib/bdiff-torture.py conform to our import style

import-checker: extend check of symbol-import order to all local modules It doesn't make sense that (a) is allowed whereas (b) is disallowed. a) from mercurial import hg from mercurial.i18n import _ b) from . import hg from .i18n import _

import-checker: always build a list of imported symbols The next patch will rely on it.

import-checker: fix test to make a real package Otherwise "testpackage" wouldn't be counted as a package when building a list of imported symbols.

py3: move up symbol imports to enforce import-checker rules Since (b) is banned, we should do the same for (a) for consistency. a) from mercurial import hg from mercurial.i18n import _ b) from . import hg from .i18n import _

util: make copyfile avoid ambiguity of file stat if needed In some cases below, copying from backup is used to restore original contents of a file. If copying keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - failure of transaction before closing (from '.hg/journal.backup.*') - rollback of previous transaction (from '.hg/undo.backup.*') To avoid such problem, this patch makes copyfile() avoid ambiguity of file stat, if needed. Ambiguity check is executed, only if: - checkambig=True is specified (not all copying needs ambiguity check), and - destination file exists before copying This patch also adds 'not (copystat and checkambig)' assertion, because combination of copystat and checkambig is meaningless. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

vfs: make rename avoid ambiguity of file stat if needed In some cases below, renaming from backup is used to restore original contents of a file. If renaming keeps ctime, mtime and size of a file, restoring is overlooked, and old contents cached before restoring isn't invalidated as expected. - failure of transaction before closing (only from '.hg/journal.dirstate') - rollback of previous transaction (from '.hg/undo.*') - failure in dirstateguard scope (from '.hg/dirstate.SUFFIX') To avoid such problem, this patch makes vfs.rename() avoid ambiguity of file stat, if needed. Ambiguity check is executed, only if: - checkambig=True is specified (not all renaming needs ambiguity check), and - destination file exists before renaming This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

vfs: make atomictempfile avoid ambiguity of file stat if needed This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: make atomictempfile avoid ambiguity of file stat if needed Ambiguity check is executed at close(), only if: - atomictempfile is created with checkambig=True, and - target file exists before renaming This restriction avoids performance decrement by needless examination of file stat (for example, filelog doesn't need exact cache validation, even though it uses atomictempfile to write changes out). See description of filestat class for detail about why the logic in this patch works as expected. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

util: add filestat class to detect ambiguity of file stat Current posix.cachestat implementation might overlook change of a file, if changing keeps ctime, mtime and size of file. Comparison of inode number also overlooks changing in such situation, because inode number is rapidly reused. Contents of a file cached before changing isn't invalidated as expected, if change of a file is overlooked for this "ambiguity" of file stat. This patch adds filestat class to detect ambiguity of file stat. This patch is a part of preparation for "Exact Cache Validation Plan": https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

run-tests: handle json.dumps divergence In py2, json.dumps includes a trailing space after a comma at the end of lines. The py3 behavior which omits the trailing space is preferable, so we're going to strip it.

tests: use debuginstall to retrieve hg version

debuginstall: add mercurial version

strip: invalidate phase cache after stripping changeset (issue5235) When we remove a changeset from the changelog, the phase cache must be invalidated, otherwise it could refer to changesets that are no longer in the repo. To reproduce the failure, I created an extension querying the phase cache after the strip transaction is over. To do that, I stripped two commits with a bookmark on one of them to force another transaction (we open a transaction for moving bookmarks) after the strip transaction. Without the fix in this patch, the test leads to a stacktrace showing the issue: repair.strip(ui, repo, revs, backup) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/repair.py", line 205, in strip tr.close() File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/transaction.py", line 44, in _active return func(self, *args, **kwds) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/transaction.py", line 490, in close self._postclosecallback[cat](self) File "$TESTTMP/crashstrip2.py", line 4, in test [repo.changelog.node(r) for r in repo.revs("not public()")] File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/changelog.py", line 337, in node return super(changelog, self).node(rev) File "/Users/lcharignon/facebook-hg-rpms/hg-crew/mercurial/revlog.py", line 377, in node return self.index[rev][7] IndexError: revlog index out of range The situation was encountered in inhibit (evolve's repo) where we would crash following the volatile set invalidation submitted by Augie in e6f490e328635312ee214a12bc7fd3c7d46bf9ce. Before his patch the issue was masked as we were not accessing the phasecache after stripping a revision. This bug uncovered another but in histedit (see explanation in issue5235). I changed the histedit test accordingly to avoid fixing two things at once.

py3: make tests/svn-safe-append.py use absolute_import

py3: make tests/test-atomictempfile.py use absolute_import

py3: tests/test-check-py3-compat.t output updated The lower part of the tests runs with Python 3.5 so its remains unchanged with new commits.

py3: use setattr() to assign new class attribute The old method produces error 'object does not supports item assignment'. So setattr() is used to assign a new class attribute via __dict__ .

localrepo: use dirstate savebackup instead of handling dirstate file manually This is one step towards having dirstate manage its own storage. It will be useful for the implementation of sql dirstate [1]. This introduced a small test change: now we always write the dirstate before saving backup so in some cases where dirstate file didn't exist yet savebackup can create it. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

localrepo: use dirstate restorebackup instead of copying dirstate manually This is one step towards having dirstate manage its own storage. It will be useful for the implementation of sqldirstate [1]. I'm deleting two of the dirstate.invalidate() calls in localrepo because restorebackup method does that for us. [1] https://www.mercurial-scm.org/wiki/SQLDirstatePlan

dirstate: add prefix and suffix arguments to backup This would allow the code explicitly copying dirstate to use this method instead. Use of this method will increase encapsulation (the dirstate class will be sole owner of its on-disk storage).

tests: mark test-atomictempfile.py write as binary

tests: mark test-context.py write as binary

transaction: turn lack of locking into a hard failure (API) We have been warning about transactions without locks for about a year (and three releases), third party extensions had a fair grace period to fix their code, we are moving lack of locking to a hard failure in order to protect users against repository corruption.

test: extract develwarn transaction testing in its own command The lack of locking for a transation is about to change from a warning to an error. We first extract the test decidated to this warning to make the next changeset clearer.

graphmod: update edgemap in-place The edgemap update was not actually propagated to future asciiedge calls; update the edge state dictionary in-place instead.

tests: test-archive.t use mercurial.util for urllib compat

tests: test-archive.t use sys.stdout.buffer for binary output in py3

localrepo: prevent executable-bit only changes from being lost on amend If you have just executable-bit change and amend it twice it will vanish: * After the first amend the commit will have the proper executable bit set in manifest but it won't have the the file on the list of files in changelog. * The second amend will read the wrong list of files from changelog and it will copy the manifest entry from parent for this file. * Voila! The change is lost. This change repairs the bug in localrepo causing this and adds a test for it.

tests: escape bytes setting MSB in input of grep for portability GNU grep (2.21-2 or later) assumes that input is encoded in LC_CTYPE, and input is binary if it contains byte sequence not valid for that encoding. For example, if locale is configured as C, a byte setting most significant bit (MSB) makes such GNU grep show "Binary file <FILENAME> matches" message instead of matched lines unintentionally. This behavior is recognized as a bug, and fixed in GNU grep 2.25-1 or later. But some distributions are shipped with such buggy version (e.g. Ubuntu xenial, which is used by launchpad buildbot). http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19230 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800670 http://packages.ubuntu.com/xenial/grep This causes failure of test-commit-interactive.t, which applies grep on CP932 byte sequence since 1111e84de635. But, explicit setting LC_CTYPE for CP932 might cause another problem, because it can't be assumed that all environment running Mercurial tests allows arbitrary locale setting. To resolve this issue, this patch escapes bytes setting MSB in input of grep. For this purpose: - str.encode('string-escape') isn't useful, because it escapes also control code (less than 0x20), and makes EOL handling complicated - "f --hexdump" isn't useful, because it isn't line-oriented - "sed -n" seems reasonable, but "sed" itself sometimes causes portability issue, too (e.g. 900767dfa80d or afb86ee925bf) This patch is posted with "stable" flag, because 1111e84de635 is on stable branch.

tests: test-archive.t use absolute_import This is a step to adding a mercurial dependency to simplify py3 compat

tests: test-archive.t use open() instead of file() for py3 compat

tests: test-archive.t use print_function

merge with stable

readlink: use print_function

tests: test-addremove-similar.t use print() for py3

tests: add coverage for run-tests.py --whitelist

hg: limit HGUNICODEPEDANTRY to py2 reload is not available in py3, and py3 is fatal anyway

py3: make i18n/hggettext use print_function

py3: make i18n/hggettext use absolute_import

py3: make doc/docchecker use print_function

py3: make doc/docchecker use absolute_import

py3: make contrib/undumprevlog use absolute_import

py3: make contrib/dumprevlog use print_function

py3: make contrib/dumprevlog use absolute_import

py3: make contrib/check-commit use print_function

py3: make contrib/check-commit use absolute_import

hgcia: remove hgcia (BC) As discussed at: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-March/081018.html, cia service is down for years. It also uses socket.setdefaulttimeout() which will break chg. This patch removes the extension.

py3: make tests/hghave use absolute_import

py3: make tests/f use absolute_import

py3: make tests/dummyssh use absolute_import

py3: make raise statement python3 compatible In python3 raise error, message has been changed to raise error(message) In additional to that nodes.SkipNode is changed to nodes.SkipNode() so that it creates an instance directly.

Added signature for changeset aaabed77791a

Added tag 3.8.2 for changeset aaabed77791a

help: search section of help topic by translated section name correctly Before this patch, "hg help topic.section" might show unexpected section of help topic in some encoding. It applies str.lower() instead of encoding.lower(str) on translated message to search section case-insensitively, but some encoding uses 0x41(A) - 0x5a(Z) as the second or later byte of multi-byte character (for example, ja_JP.cp932), and str.lower() causes unexpected result. To search section of help topic by translated section name correctly, this patch replaces str.lower() by encoding.lower(str) for both query string (in commands.help()) and translated help text (in minirst.getsections()).

patch: show lower-ed translated message correctly Before this patch, patch.filterpatch() shows meaningless translation of help message for chunk selection in some encoding. It applies str.lower() instead of encoding.lower(str) on translated message, but some encoding uses 0x41(A) - 0x5a(Z) as the second or later byte of multi-byte character (for example, ja_JP.cp932), and str.lower() causes unexpected result. To show lower-ed translated message correctly, this patch replaces str.lower() by encoding.lower(str).

py3: make i18n/posplit use print_function

py3: make i18n/posplit use absolute_import

wireproto: optimize handling of large batch responses Now that batch can be used by remotefilelog, the quadratic string copying this was doing was actually disastrous. In my local testing, fetching a 56 meg file used to take 3 minutes, and now takes only a few seconds.

cleanup: replace False identity testing with an explicit token object The recommended way to check default value (when None is not as option) is a token object. Identity testing to integer is less explicit and not guaranteed to work in all implementations.

devel: officially deprecate dirstate.write without transaction argument When we introduce the develwarning, we did not had an official deprecation API and infrastructure. We can now officially deprecate the old way with a version deadline.

devel: officially deprecate update without destination When we introduce the develwarning, we did not had an official deprecation API and infrastructure. We can now officially deprecate the old way with a version deadline.

devel: fix a typo in a deprecation warning Credit goes to Sean Farley for spotting it.

devel: officially deprecate old style revset When we introduce the develwarning, we did not had an official deprecation API and infrastructure. We can now officially deprecate the old way with a version deadline.

check-code: handle py3 open divergence open() really wants an encoding attribute

check-code: switch to opener

check-code: handle range/xrange divergence

check-code: fix py3 complaint about \NNN being invalid unicode

hghave: switch from iteritems to items With this, test-hghave.t passes on python 3. Four features fail because mercurial still is not py3 safe: absimport cacheable hardlink defaultcacerts But that will be resolved automatically eventually.

hghave: matchoutput needs to use bytes for regexp file output is bytes in py3, so we need each regexp to be bytes

revset: make dagrange preserve order of input set Unlike range, dagrange has no inverted range (such as '10:0'). So there should be no practical reason to keep dagrange as a function that forces its own ordering. No performance regression is spotted in contrib/base-revsets.txt.

tests: mark test-check-pyflakes.t as requiring hg1.0+ hg does not yet run with py3, so if you try: ./run-tests.py --local test-check-pyflakes.t ... it will try to run the local hg, which does not work and thus, hg locate will return no output to stdout (and stderr is sent to /dev/null). If you do: ./run-tests.py --with-hg=~/bin/hg test-check-pyflakes.t Then it should work, if your hg is new enough to have a locate command (hg0.6 does not have locate).

dirstate: make backup methods public They are called from outside of dirstate anyway and I want the localrepo to use them too.

check-code: add a rule banning `env -u`

subrepo: use unset instead of env -u to fix test on BSDs (issue5229)

graphmod: partial edge styling Allow for a style to only apply to the last N lines (for positive N) or everything but the first N lines (for negative N) of the section along the current node. This allows for more subtle grandparent styling. So from the default: $ hg log -G ... o Lorem ipsum dolor sit :\ amet, consectetur : : adipiscing elit, sed : : do eiusmod tempor : : o : incididunt ut labore | : et dolore magna | : aliqua. Ut enim ad | : minim veniam, quis |/ o nostrud exercitation : ullamco laboris nisi : ut aliquip ex ea : commodo consequat. : o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu to $ hg log -G --config "experimental.graphstyle.grandparent=2." ... o Lorem ipsum dolor sit |\ amet, consectetur | | adipiscing elit, sed . . do eiusmod tempor . . o | incididunt ut labore | | et dolore magna | | aliqua. Ut enim ad | | minim veniam, quis |/ o nostrud exercitation | ullamco laboris nisi | ut aliquip ex ea . commodo consequat. . o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu or $ hg log -G --config "experimental.graphstyle.grandparent=1:" ... o Lorem ipsum dolor sit |\ amet, consectetur | | adipiscing elit, sed | | do eiusmod tempor : : o | incididunt ut labore | | et dolore magna | | aliqua. Ut enim ad | | minim veniam, quis |/ o nostrud exercitation | ullamco laboris nisi | ut aliquip ex ea | commodo consequat. : o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu or $ hg log -G --config "experimental.graphstyle.grandparent=-2!" ... o Lorem ipsum dolor sit |\ amet, consectetur ! ! adipiscing elit, sed ! ! do eiusmod tempor ! ! o | incididunt ut labore | | et dolore magna | | aliqua. Ut enim ad | | minim veniam, quis |/ o nostrud exercitation | ullamco laboris nisi ! ut aliquip ex ea ! commodo consequat. ! o Duis aute irure dolor | in reprehenderit in ~ voluptate velit esse cillum dolore eu

pure: write a really lazy version of pure indexObject On PyPy this version performs reasonably well compared to C version. Example command is "hg id" which gets faster, depending on details of your operating system and hard drive (it's bottlenecked on stat mostly) There is potential for improvements by storing extra as a condensed struct too.

dispatch: always load extensions before running shell aliases (issue5230) Before this patch, we may or may not load extensions for shell aliases depending on whether the command is abbreviated or not. Loading extensions may have useful side effects to shell aliases. For example, the pager extension does not work for shell aliases. This patch removes the code checking shell aliases before loading extensions to give the user a more consistent experience. It may hurt performance for shell aliases a bit without chg but the correctness seems worth it. It will also make the behavior consistent with chg since chg will always load all extensions before running commands.

httpclient: update to upstream revision 2995635573d2 This is mostly Python 3 compat work thanks to timeless.

crecord: call prevsibling() and nextsibling() directly The 3 classes for items used in crecord (uiheader, uihunk, uihunkline) all have prevsibling() and nextsibling() methods. The two methods are used to get the previous/next item of the same type of the same parent element as the current one: when `a` is a uihunkline instance, a.nextsibling() returns the next line in this hunk (or None, if `a` is the last line). There are also two similar methods: previtem() and nextitem(). When called with constrainlevel=True (the default) they simply returned the result of prevsibling()/nextsibling(). Only when called with constrainlevel=False they did something different: they returned previous/next item regardless of its type (so if `a` is the last line in a hunk, a.nextitem(constrainlevel=False) could return the next hunk or the next file -- something that is not a line). Let's simplify this logic and make code call -sibling() methods when only siblings are needed and -item() methods when any item would do, and then remove the constrainlevel argument from previtem() and nextitem().

dispatch: add fail-* family of hooks The post-* family of hooks will not run in case a command fails (i.e. raises an exception). This makes it inconvenient to hook into events such as doing something in case of a failed push. We catch all exceptions to run the failure hook. I am not sure if this is too aggressive, but tests apparently pass.

py3: make hgext/rebase.py use absolute_import

py3: make hgext/mq.py use absolute_import

py3: make hgext/hisedit.py use absolute_import

py3: make hgext/hgk.py use absolute_import

py3: make hgext/gpg.py use absolute_import

py3: make hgext/graphlog.py use absolute_import

import-checker: recognize relative imports from parents of current package So far fromlocal recognizes relative imports of the form: from . import D from .. import E It wasn't prepared for recognizing relative imports like: from ..F import G The bug was not found so far because all relative imports starting from the parent was in the list of allowsymbolicimports like: from ..i18n import from ..node import

py3: make hgext/fetch.py use absolute_import

tests: test-check-py3-compat.t output updated The test output was not updated as the lower section of the test updates with python3.5, so it might be the case that people have updated the modules but the test was left as it was. So this patch updates the test output.

revset: factor out public optimize() function from recursion New optimize() hides internal arguments and return values. This makes it easy to add more parameters and return values to _optimize().

revset: introduce temporary variables in optimize() where they look better

revset: construct arguments of only() against matched tree Since _isonly() knows the structure of 'revs' and 'bases', it should be slightly easier to understand than destructuring 'ta' and 'tb'.

revset: unnest isonly() closure from optimize() There were no variables to be captured.

sslutil: stop checking for web.cacerts=! (BC) The previous patch stopped setting web.cacerts=! to indicate --insecure. That left user configs as the only source that could introduce web.cacerts=!. The practical impact of this patch is we no longer honor web.cacerts=! in configs. Instead, we always treat web.cacerts as a path. The patch is therefore technically BC. However, since I don't believe web.cacerts=! is documented, it should be safe to remove. a939f08fae9c (which introduced --insecure) has no indication that web.cacerts=! is anything but an implementation detail, reinforcing my belief it can be removed without major debate.

dispatch: stop setting web.cacerts=! to indicate --insecure Consumers needing to know if --insecure was used have already transitioned to using ui.insecureconnections. The previous patch removed the last meaningful consumer looking for web.cacerts=!.

sslutil: use CA loaded state to drive validation logic Until now, sslkwargs may set web.cacerts=! to indicate that system certs could not be found. This is really obtuse because sslkwargs effectively sets state on a global object which bypasses wrapsocket() and is later consulted by validator.__call__. This is madness. This patch introduces an attribute on the wrapped socket instance indicating whether system CAs were loaded. We can set this directly inside wrapsocket() because that function knows everything that sslkwargs() does - and more. With this attribute set on the socket, we refactor validator.__call__ to use it. Since we no longer have a need for setting web.cacerts=! in sslkwargs, we remove that. I think the new logic is much easier to understand and will enable behavior to be changed more easily.

sslutil: handle ui.insecureconnections in validator Right now, web.cacerts=! means one of two things: 1) Use of --insecure 2) No CAs could be found and were loaded (see sslkwargs) This isn't very obvious and makes changing behavior of these different scenarios independent of the other impossible. This patch changes the validator code to explicit handle the case of --insecure being used. As the inline comment indicates, there is room to possibly change messaging and logic here. For now, we are backwards compatible.

sslutil: check for ui.insecureconnections in sslkwargs The end result of this function is the same. We now have a more explicit return branch. We still keep the old code looking at web.cacerts=! a few lines below because we're still setting web.cacerts=! and need to react to the variable. This will be removed in an upcoming patch.

dispatch: set ui.insecureconnections when --insecure is used

ui: add an instance flag to hold --insecure bit Currently, when --insecure is used we set web.cacerts=! and socket validation takes this value into account. web.cacerts=! is not documented AFAICT and is purely an internal implementation detail. Let's be more explicit about what is going on by introducing a dedicated variable outside of the config values to track that --insecure is used.

sslutil: make sslkwargs code even more explicit The ways in which this code can interact with socket wrapping and validation later are mind numbing. This patch helps make it even more clear. The end behavior should be identical.

sslutil: move code examining _canloaddefaultcerts out of _defaultcacerts Before, the return of _defaultcacerts() was 1 of 3 types. This was difficult to read. Make it return a path or None. We had to update hghave.py in the same patch because it was also looking at this internal function. I wasted dozens of minutes trying to figure out why tests were failing until I found the code in hghave.py...

sslutil: further refactor sslkwargs The logic here and what happens with web.cacerts is mind numbing. Make the code even more explicit.