util: don't use mutable default argument value I don't think this is any tight loops and we'd need to worry about PyObject creation overhead. Also, I'm pretty sure strptime() will be much slower than PyObject creation (date parsing is surprisingly slow).

match: don't use mutable default argument value There shouldn't be a big perf hit creating a new object because this function is complicated and does things that dwarf the cost of creating a new PyObject.

hgweb: don't use mutable default argument value

hgweb: don't use mutable default argument value

filemerge: don't use mutable default argument value

context: don't use mutable default argument value Mutable default argument values are a Python gotcha and can represent subtle, hard-to-find bugs. Lets rid our code base of them.

heads: enable pager

branches: enable pager

py3: fix slicing of bytes in revset.formatspec()

py3: make set of revset operators and quotes in bytes

py3: convert set of revset initial symbols back to bytes Otherwise tokenize() would fail due to comparison between unicode and bytes.

pycompat: add helper to iterate each char in bytes

branchmap: fix python 2.6 by using util.buffer() instead of passing bytearray

rebase: allow rebasing children of wd to wd if a new branch has been set (BC) The named branch of the leaf changeset can be changed by updating to it, setting the branch, and amending. But previously, there was no good way to *just* change the branch of several linear changes. If rebasing changes with another parent to '.', it would pick up a pending branch change up. But when rebasing changes that have the same parent, it would fail with 'nothing to rebase', even when the branch name was set differently. To fix this, allow rebasing to same parent when a branch has been set.

merge: check current wc branch for 'nothing to merge', not its p1 The working directory will usually be clean or very clean, and wc will usually have the same branch as its parent. This change will thus usually not make any difference and is done as a separate change to show that. It will be used in a later change.

lock: do not encode result of gethostname on Python 2 If a hostname contained non-ascii character, str.encode() would first try to decode it to a unicode and raise UnicodeDecodeError.

py3: prove `hg files --rev` works

tests: make a variable for hg binary location in test-check-py3-commands The number of which calls in here is starting to get silly.

lock: encode result of gethostname into a bytestring

config: avoid using a mutable default Nothing *currently* mutates this list, but the moment something does it'll be shared between all config instances. Avoid this eventuality.

localrepo: deprecate 'repo.join' in favor of 'repo.vfs.join' localrepo have an insane amount of method. Accessing the feature through the vfs is not really harder and allow us to schedule that method for removal.

pycompat: move imports of cStringIO/io to where they are used There's no point to import cStringIO as io since we have to select StringIO or BytesIO conditionally.

rbc: empty (and invalid) rbc-names file should give an empty name list An empty file (if it somehow should exist) used to give a list with an empty name. That didn't do any harm, but it was "wrong". Fix that.

rbc: use struct unpack_from and pack_into instead of unpack and pack These functions were introduced in Python 2.5 and are faster and simpler than the old ones ... mainly because we can avoid intermediate buffers: $ python -m timeit -s "_rbcrecfmt='>4sI'" -s 's = "x"*10000' -s 'from struct import unpack' 'unpack(_rbcrecfmt, buffer(s, 16, 8))' 1000000 loops, best of 3: 0.543 usec per loop $ python -m timeit -s "_rbcrecfmt='>4sI'" -s 's = "x"*10000' -s 'from struct import unpack_from' 'unpack_from(_rbcrecfmt, s, 16)' 1000000 loops, best of 3: 0.323 usec per loop $ python -m timeit -s "from array import array" -s "_rbcrecfmt='>4sI'" -s "s = array('c')" -s 's.fromstring("x"*10000)' -s 'from struct import pack' -s "rec = array('c')" 'rec.fromstring(pack(_rbcrecfmt, "asdf", 7))' 1000000 loops, best of 3: 0.364 usec per loop $ python -m timeit -s "from array import array" -s "_rbcrecfmt='>4sI'" -s "s = array('c')" -s 's.fromstring("x"*10000)' -s 'from struct import pack_into' -s "rec = array('c')" -s 'rec.fromstring("x"*100)' 'pack_into(_rbcrecfmt, rec, 0, "asdf", 7)' 1000000 loops, best of 3: 0.229 usec per loop

revlog: use bytes() instead of str() to get data from memoryview Fixes `files -v` on Python 3.

util: teach url object about __bytes__ __str__ tries to do something reasonable, but someone else more familiar with encoding bugs should check my work.

manifest: ensure paths are bytes (not str) in pure parser

manifest: now that node.bin is available, use it directly Previously we were getting it through revlog, which is a little unusual.

manifest: use node.bin instead of .decode('hex') The latter doesn't work in Python 3.

manifest: add __next__ methods for Python 3 Python 3 renamed .next() in the iterator protocol to __next__().

files: use native string type to load rev opt from dict

store: fix many single-byte ops to use slicing in _auxencode

py3: add "b" prefix to string literals related to module policy String literals without explicit prefix in __init__.py and policy.py are treated as unicode object on Python3, because these modules are loaded before setup of our specific code transformation (the later module is imported at the beginning of __init__.py). BTW, "modulepolicy" in __init__.py is initialized by "policy.policy". This causes issues below; - checking "policy" value in other modules causes unintentional result For example, "b'py' not in (u'c', u'py')" returns True unintentionally on Python3. - writing "policy" out fails at conversion from unicode to bytes 62939e0148f1 fixed this issue for default code path, but "policy" can be overridden by HGMODULEPOLICY environment variable (it should be rare case for developer using Python3, though). This patch does: - add "b" prefix to all string literals, which are related to module policy, in modules above. - check existence of HGMODULEPOLICY, and overwrite "policy" only if it exists For simplicity, this patch omits checking "supports_bytes_environ", switching os.environ/os.environb, and so on (Yuya agreed this in personal talking)

py3: drop unused aliases to array.array which are replaced with bytearray

pycompat: default to BytesIO instead of StringIO

repoview: specify setattr values as native strings

revlog: use bytes() to ensure text from _chunks is a reasonable type

revlog: extract first byte of revlog with a slice so it's portable

revsetlang: slice out single bytes instead of indexing For portability with Python 3.

lock: use %d to format integer into a bytestring

parser: use %d instead of %s for interpolating error position Error position is an int, so we should use %d instead of %s. Fixes failures on Python 3.

manifest: unbreak pure-python manifest parsing on Python 3

context: use portable construction to verify int parsing

ui: portably bytestring-ify url object

scmutil: fix key generation to portably bytestringify integer

branchmap: stringify int in a portable way We actually need a bytes in Python 3, and thanks to our nasty source loader this will portably do the right thing.

branchmap: don't use buffer() on Python 3 This is certainly slower than the Python 2 code, but it works, and we can revisit it later if it's a problem.

py3: use bytearray() instead of array('c', ...) constructions Portable from 2.6-3.6.

summary: don't explicitly str() something we're about to %s str() is wrong on Python 3 here, and %s implicitly calls str() anyway, so this was just extra dancing for no reason.

context: implement both __bytes__ and __str__ for Python 3 They're very similar, for obvious reasons.

context: work around `long` not existing on Python 3 I can't figure out what this branch is even trying to accomplish, and it was introduced in ac89a23ca814 which doesn't really shed any insight into why longs are treated differently from ints.

phases: explicitly evaluate list returned by map On Python 3 map() returns a generator, which bool()s to true even if it had an empty input set. Work around this by using list() on the map() result.

ui: check for --debugger in sys.argv using r-string to avoid bytes on py3 Our source loader was errantly turning this --debugger into a bytes, which was then causing me to still get a pager when I was using the debugger on py3. That made life hard.

minirst: use bytes.strip instead of str.strip bytes.strip exists in Python 2.6 and Python 2.7 also.

smcposix: pass unicode as first argument to array.array This is an instance where we can safely convert the first argument, rest are the cases except one where we are using 'c' which is not there in Python 3. So that needs to be handled differently. This will help in making `hg help` run on Python 3.

util: pass encoding.[encoding|encodingmode] as unicodes We need to pass str to encode() and decode().

win32text: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

transplant: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

shelve: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

share: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

mq: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

largefiles: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

keyword: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

journal: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

histedit: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

eol: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

convert: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

test-bundle2-remote-changegroup: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

subrepo: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

repair: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

merge: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

hg-mod: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

commands: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

cmdutil: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

localrepo: directly use repo.vfs.join The 'repo.join' method is about to be deprecated.

minirst: make encoding.encoding unicodes to pass into encode() and decode()

minirst: make regular expressions bytes

setup: convert setupversion to unicode Something deep in the bowels of distutils expects "version" passed to setup() to be a str/unicode. So, convert the type. This still works on Python 2 because the string is ascii and an implicit coercion back to str/bytes should work without issue. If it does cause problems, we can always make the unicode conversion dependent on running Python 3. This change makes `python3.5 setup.py install` work.

util: add debugstacktrace depth limit Useful when you don't care about the start of the stack, but only want to see the last entries.

util: strip trailing newline from debugstacktrace message This makes the function more convenient to use as drop-in replacement for ui.write & co.

tests: rework util.debugstacktrace tests Prepare for adding another test.

vfs: use repo.vfs.rename

vfs: use repo.vfs.unlinkpath

merge: use repo.wvfs.unlinkpath

vfs: use repo.wvfs.unlinkpath

policy: try and always have a bytes for module policy debuginstall now runs cleanly in Python 3.

init: zstd is already python3-ready, so don't run it through our importer

config: guard against setconfig specifying unicode values on py3 This was leading to some difficult to trace problems because the values were set in one place, but then blew up much later in the program. Exploding violently with an assertion seems reasonable here.

dispatch: enforce bytes when converting boolean flags to config items This fixes --verbose on Python 3.

py3: add a test that proves %include and missing-extension code works

extensions: tapdance to get reasonable import error formatting I'm not thrilled with this, but it seems to work.

rebase: abort if *any* commit in rebase set is public

merge with stable

httpconnection: rename config to groups Because that is what it is.

httpconnection: don't use dict() Not sure the history here. But we don't do this elsewhere.

formatter: set _first on formatter, not ui The _first field is used for tracking when to emit a separator between items. It seems like it's clearly formatter state, not ui state, so let's move it there.

rebase: unhide original working directory node as well (issue5219) By including the working directory revision at the start of rebase in the repo._rebaseset, we make sure it's not hidden when we update back to it at the end of the rebase. This feels like abusing the set a bit given its name (_rebaseset), but I couldn't think of another name that's clearly better.

rebase: pass in a regular set to _setrebasesetvisibility() Trivial refactoring to simplify the next patch.

tests: demonstrate broken rebase from obsolete commit This shows the bug I reported in issue 5219.

treemanifest: make node reuse match flat manifest behavior In a flat manifest, a node with the same content but different parents is still considered a new node. In the current tree manifests however, if the content is the same, we ignore the parents entirely and just reuse the existing node. In our external treemanifest extension, we want to allow having one treemanifest for every flat manifests, as a way of easeing the migration to treemanifests. To make this possible, let's change the root node treemanifest behavior to match the behavior for flat manifests, so we can have a 1:1 relationship. While this sounds like a BC breakage, it's not actually a state users can normally get in because: A) you can't make empty commits, and B) even if you try to make an empty commit (by making a commit then amending it's changes away), the higher level commit logic in localrepo.commitctx() forces the commit to use the original p1 manifest node if no files were changed. So this would only affect extensions and automation that reached passed the normal localrepo.commit() logic straight into the manifest logic.

help: update help.internalstable for new censor docs

parsers: drop old nonnormalentries method This is okay to do because the Python will fall back transparently if the method is missing.

parsers: avoid leak of nonnset and otherpset Py_BuildValue increments the refcount, rather than stealing the reference, which I missed in code review.

sslutil: issue warning when [hostfingerprint] is used Mercurial 3.9 added the [hostsecurity] section, which is better than [hostfingerprints] in every way. One of the ways that [hostsecurity] is better is that it supports SHA-256 and SHA-512 fingerprints, not just SHA-1 fingerprints. The world is moving away from SHA-1 because it is borderline secure. Mercurial should be part of that movement. This patch adds a warning when a valid SHA-1 fingerprint from the [hostfingerprints] section is being used. The warning informs users to switch to [hostsecurity]. It even prints the config option they should set. It uses the SHA-256 fingerprint because recommending a SHA-1 fingerprint in 2017 would be ill-advised. The warning will print itself on every connection to a server until it is fixed. There is no way to suppress the warning. I admit this is annoying. But given the security implications of sticking with SHA-1, I think this is justified. If this patch is accepted, I'll likely send a follow-up to start warning on SHA-1 certificates in [hostsecurity] as well. Then sometime down the road, we can drop support for SHA-1 fingerprints. Credit for this idea comes from timeless in issue 5466.

setup: use setuptools on Windows (issue5400) We've had a long, complicated history with setuptools. We want to make it the universal default. But when we do, it breaks things. `python setup.py build` is broken on Windows today. Forcing the use of setuptools via FORCE_SETUPTOOLS=1 unbreaks things. Since the previous bustage with making setuptools the default was on !Windows, it seems safe to move ahead with the setuptools transition on Windows. So this patch does that.

schemes: use br'' literal to define bytes regexp

help: fix layout of pre-formatted text

help: fix example of revs() fileset

filecache: make 'join' abstract All subclasses redefine this method, so we can make it abstract.

filecache: explicitly test 'repofilecache' The tests is actually about testing a repofilecache (it uses a fake repo). We make this clear to prevent blockers while cleaning theses API.

repofilecache: directly use 'repo.vfs.join' The 'vfs' attribute already have all methods we need, the value of going through the repository for this is low. so we removes it.

repofilecache: define a 'join' method We are about to turn the 'join' method of the base class Abstract, so we need on to be defined in the localrepo. The ultimate goal here is to be able to stop relying for the 'localrepo' class to have a 'join' methods (there is above one hundred methods on 'localrepo'. This change make te 'repo' file cache have its own code so that we can prepare this change to the repostory class. explicite join

wix: add censor docs to installer script Spotted by Matt Harbison.

internals: add some brief documentation about censor

localrepo: rename proxycls to filteredrepo When debugging in a Python shell, the type of "repo" is "proxycls", which could confuse new people. In [1]: repo Out[1]: <mercurial.localrepo.proxycls at 0x7f65d4b976d0> Let's rename it to "filteredrepo" to make it clearer.

dirstate: track otherparent files same as nonnormal Calling dirstate.setparents() is expensive in a large repo because it iterates over every file in the dirstate. It does so to undo any merge state or otherparent state files. Merge state files are already covered by dirstate._nonnormalset, so we just need to track otherparent files in a similar manner to avoid the full iteration here. Fixing this shaves 20-25% off histedit in large repos. I tested this by adding temporary debug logic to verify that the old files processed in the loop matched the new files processed in the loop and running the test suite.

hgweb: use patch.diffhunks in webutil.diffs to simplify the algorithm Function patch.diffhunks yields items for a "block" (i.e. a file) as a whole so take advantage of this to simplify the algorithm and avoid parsing diff lines to determine whether we're starting a new "block" or not. Thus we drop to external block counter and rely on diffhunks iterations instead. We also take advantage of the fact that patch.diffhunks() yields *lines* of hunks (instead of a string) to avoid building a list that is ''.join-ed into a string that is then split. As lines in 'header' returned by patch.diffhunks() have no trailing new line, we need to insert it ourselves to match template expectations.

hgweb: start enumerate at 1 in webutil.diffs's inner function prettyprintlines

patch: add a diffhunks function yielding (diffheaders, hunks) trydiff function now yield (header, hunks) tuple that are processed by diffhunks(). Then diff() is a wrapper around diffhunks().

mdiff: let unidiff return (diffheader, hunks) This will be used to make it possible to filter diff hunks based on this range information. Now unidiff returns a 'hunks' generator that yield tuple (hunkrange, hunklines) coming from _unidiff() with 'newline at end of file' processing.

mdiff: extract a checknonewline inner function in unidiff()

mdiff: distinguish diff headers from hunks in unidiff() Let unidiff return the list of headers it produces (lines '--- <original>' and '+++ <new>') apart from diff hunks. In patch.diff(), we combine headers generated there (not specific to unified format) with those from unidiff(). By returning a list of header lines, we do not append new lines in datetag inner function of unidiff() so that all header lines are '\n'.join-ed in a similar way.

test: end printed diff "hunks" with an empty string in test-context.py So that the resulting diff is correct and does not include a spurious empty line between lines "diff --git a/foo b/foo" and "--- a/foo".

mdiff: let _unidiff yield hunks as (<range information>, <hunk lines>) Now _unidiff yields each hunk lines packed into a tuple with the "range information" `(s1, l1, s2, l2)` that is used to build the typical hunk header '@@ -s1,l1 +s2,l2 @@'. This will be used to make it possible to filter diff hunks based on this range information. The new "range information" is ignored in unidiff() (only caller of _unidiff) for now.

mdiff: turn the comment above _unidiff into a docstring

mdiff: compute newlines-splitted texts within _unidiff There is no reason to compute splitted texts l1, l2 in unidiff() before calling _unidiff as they are only used with the latter function.

dispatch: add pagination of two more help cases I missed these in the last round anf Yuya spotted them in review. Thanks!

help: avoid mutating passed-in `keep` list in `formattedhelp`

extensions: use [0:1] slice on config path instead of [0] This behaves the same in Python 2 and Python 3, even though the path is a bytes.

extensions: use inspect module instead of func_code.co_argcount Fixes the extsetup argspec check on Python 3.

treemanifest: add tests covering hg diff of partial trees Previously the hg files tests also covered the logic (i.e. treemanifest.matches) that governed how hg diff limited its diff. In a future patch we will be switching treemanifest.diff() to have a custom implementation, so let's go ahead and add equivalent test coverage for hg diff.

context: remove uses of manifest.matches This removes the uses of manifest.matches in context.py in favor of the new manifest.diff(match) api. This is part of removing manifest.matches since it is O(manifest). To drop the dependency on ctx._manifestmatches(s) we transfer responsibilty for creating status oriented manifests over to ctx._buildstatusmanifest(s). This already existed for workingctx, we just need to implement a simple version for basectx. The old _manifestmatches functionality is basically identical to the _buildstatusmanifest functionality (minus the matching part), so no behavior should be lost.

context: remove assumptions about manifest creation during _buildstatus Previously we called self.manifest() in some cases to preload the first manifest. This relied on the assumption that the later _manifestmatches() call did not duplicate any work the original self.manifest() call did. Let's remove that assumption, since it bit me during my refactors of this area and is easy to remove.

context: move _manifest from committablectx to workingctx committablectx had a _manifest implementation that was only used by the derived workingctx class. The other derived versions, like memctx and metadataonlyctx, define their own _manifest functions. Let's move the function down to workingctx, and let's break it into two parts, the _manifest part that reads from self._status, and the part that actually builds the new manifest. This separation will let us reuse the builder code in a future patch to answer _buildstatus with varying status inputs, since workingctx has special behavior for _buildstatus that the other ctx's don't have.

status: handle more node indicators in buildstatus There are several different node markers that indicate different working copy states. The context._buildstatus function was only handling one of them, and this patch makes it handle all of them (falling back to file content comparisons when in one of these states). This affects a future patch where we get rid of context._manifestmatches as part of getting rid of manifest.matches(). context._manifestmatches is currently hacky in that it uses the newnodeid for all added and modified files, which is why the current newnodeid check is sufficient. When we get rid of this function and use the normal manifest.diff function, we start to see the other indicators in the nodes, so they need to be handled or else the tests fail.

merge: remove uses of manifest.matches This gets rid of the manifest.matches calls in merge.py in favor of the new api. This is part of getting rid of manifest.matches since it is O(manifest).

copies: remove use of manifest.matches Convert the existing use of manifest.matches to use the new api. This is part of getting rid of manifest.matches, since it is O(manifest).

manifest: add match argument to diff and filesnotin As part of removing manifest.matches (since it is O(manifest)), let's start by adding match arguments to diff and filesnotin. As we'll see in later patches, these are the only flows that actually use matchers, so by moving the matching into the actual functions, other manifest implementations can make more efficient algorithsm. For instance, this will allow treemanifest diff's to only iterate over the files that are different AND meet the match criteria. No consumers are changed in this patches, but the code is fairly easy to verify visually. Future patches will convert consumers to use it. One test was affected because it did not use the kwargs version of the clean parameter.

fileset: drop false function signatures from revs() and status() docs They should be inserted by @predicate helper.

py3: factor out bytechr() function I also changed xrange(127) to range(127) as the number is relatively small.

vfs: use 'vfs' module directly in 'test-clone-uncompressed' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'test-filecache' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'test-parseindex' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'test-lock' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'contrib/undumprevlog' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'hgext.largefile' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'hgext.convert' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'hgext.transplant' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'hgext.shelve' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'hgext.mq' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.unionrepo' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.statichttprepo' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.bundlerepo' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.debugcommand' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.simplemerge' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.cmdutil' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.subrepo' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.archival' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.store' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.patch' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.repair' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: use 'vfs' module directly in 'mercurial.localrepo' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

chg: forward user-defined signals SIGUSR1 and SIGUSR2 are reserved for user-defined behaviors. They may be redefined by an hg extension [1], but cannot be easily redefined for chg. Since the default behavior (kill) is not that useful for chg, let's forward them to hg, hoping it got redefined there and could be more useful. [1] https://bitbucket.org/facebook/hg-experimental/commits/e7c883a465

chg: document why we send SIGHUP and SIGINT to process group This makes the code more consistent - other signals are documented.

tests: make test-shelve.t timing-independent It was sometimes taking 2s for me (not the "1s" the test expected).

tests: make run-tests.py run on Python 3 again

rebase: move actual rebase into a single transaction Previously, rebasing would open several transaction over the course of rebasing several commits. Opening a transaction can have notable overhead (like copying the dirstate) which can add up when rebasing many commits. This patch adds a single large transaction around the actual commit rebase operation, with a catch for intervention which serializes the current state if we need to drop back to the terminal for user intervention. Amazingly, almost all the tests seem to pass. On large repos with large working copies, this can speed up rebasing 7 commits by 25%. I'd expect the percentage to be a bit larger for rebasing even more commits. There are minor test changes because we're rolling back the entire transaction during unexpected exceptions instead of just stopping mid-rebase, so there's no more backup bundle. It also leave an unknown file in the working copy, since our clean up 'hg update' doesn't delete unknown files.

rebase: allow aborting if last-message.txt is missing Previously, if .hg/rebasestate existed but .hg/last-message.txt was missing, 'hg rebase --abort' would say there's no rebase in progress but 'hg checkout foo' would say 'abort: rebase in progress'. It turns out loading the collapse message will throw a "no rebase in progress" error if the file doesn't exist, even though .hg/rebasestate obviously indicates a rebase is in progress. The fix is to only throw an exception if we're trying to --continue, and to just eat the issues if we're doing --abort. This issue is exposed by us writing the rebase state earlier in the process. This will be used by later patches to ensure the user can appropriately 'hg rebase --abort' if there's a crash before the first the first commit has finished rebasing. Tests cover all of this. The only negative affect is we now require a hg rebase --abort in a very specific exception case, as shown in the test.

rebase: add storestatus support for transactions This let's the status writing logic support transactions. This will be useful in a later patch where we add a transaction around the entire rebase.

rebase: move storestatus onto rebaseruntime The rebaseruntime class already has the restorestatus function, so let's make it own the store status function too. This get's rid of a lot of unnecessary argument passing and will make a future patch cleaner that refactors storestatus to support transactions.

rebase: clear updatestate during rebase --abort in more cases Previously, rebase --abort would only call update if you were on a node that had already been rebased. This meant that if the rebase failed during the rebase of the first commit, the working copy would be left dirty (with a .hg/updatestate file) and rebase --abort would not have update to clean it up. The fix is to also perform an update if you're still on the target node or on the original working copy node (since the working copy may be dirty, we still need to do the update). We don't want to perform an update in all cases though because of issue4009. A subsequent patch makes this case much more common, since it causes the entire rebase transaction to rollback during unexpected exceptions. This causes the existing test-rebase-abort.t to cover this case.

repoview: convert attribute names to unicodes on Python 3 In Python 3, the attribute names must be strings i.e. unicodes.

parsers: alias long to int on Python 3

store: slice over a bytestring to get characters instead of ascii values On Python 2, >>> a = b'abc' >>> a[1] 'b' Whereas on python 3, >>> a = b'abc' >>> a[1] 98 >>> a[1:2] b'b' This does not change behaviour on python 2.

vfs: use 'vfs' module directly in 'mercurial.hg' Now that the 'vfs' classes moved in their own module, lets use the new module directly. We update code iteratively to help with possible bisect needs in the future.

vfs: extract 'vfs' class and related code to a new 'vfs' module (API) The 'scmutil' is growing large (1500+ lines) and 2/5 of it is related to vfs. We extract the 'vfs' related code in its own module get both module back to a better scale and clearer contents. We keep all the references available in 'scmutil' for now as many reference needs to be updated.

vfs: replace 'scmutil.opener' usage with 'scmutil.vfs' The 'vfs' class is the first class citizen for years. We remove all usages of the older API. This will let us remove the old API eventually.

fsmonitor: remove use of repo.opener This has been deprecated, so we need to switch to the appropriate vfs apis.

help: align description of 'base rev' with reality [issue5488] The text about revlogs seems to be wrong about -1 being used to indicate the start of a delta chain. Attempt to correct this.

help: fix internals.changegroups Add information about tree manifests, copy edit the text and fix up a few ambiguities. The document also contains a few additional fixes from Siddharth Agarwal <sid0@fb.com>, who used it to build a parser for changegroups in Rust.

share: drop 'relshared' requirement as well

share: fix typo to drop 'shared' requirement on unshare This must be a typo and it seems correct to drop the requirement since the repo is no longer a shared repository.

similar: compare between actual file contents for exact identity Before this patch, similarity detection logic (for addremove and automv) depends entirely on SHA-1 digesting. But this causes incorrect rename detection, if: - removing file A and adding file B occur at same committing, and - SHA-1 hash values of file A and B are same This may prevent security experts from managing sample files for SHAttered issue in Mercurial repository, for example. https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html https://shattered.it/ Hash collision itself isn't so serious for core repository functionality of Mercurial, described by mpm as below, though. https://www.mercurial-scm.org/wiki/mpm/SHA1 This patch compares between actual file contents after hash comparison for exact identity. Even after this patch, SHA-1 is still used, because it is reasonable enough to quickly detect existence of "(almost) same" file. - replacing SHA-1 causes decreasing performance, and - replacement of it has ambiguity, yet Getting content of removed file (= rfctx.data()) at each exact comparison should be cheap enough, even though getting content of added one costs much. ======= ============== ===================== file fctx data() reads from ======= ============== ===================== removed filectx in-memory revlog data added workingfilectx storage ======= ============== =====================

localrepo: handle rename with hardlinks properly In "aftertrans", we rename "journal.*" to "undo.*". We expect "journal.*" files to disappear after renaming. However, if "journal.foo" and "undo.foo" refer to a same file (hardlink), rename may be a no-op, leaving both files on disk, according to Linux manpage [1]: If oldpath and newpath are existing hard links referring to the same file, then rename() does nothing, and returns a suc‐ cess status. The POSIX specification [2] is not very clear about what to do. To be safe, remove "undo.*" before the rename so "journal.*" cannot be left on disk. [1]: http://man7.org/linux/man-pages/man2/rename.2.html [2]: http://pubs.opengroup.org/onlinepubs/9699919799/

dirstate: avoid unnecessary load+dump during backup Previously, dirstate.savebackup unconditionally dumps the dirstate map to disk. It may require loading dirstate first to be able to dump it. Those operations could be expensive if the dirstate is big, and could be avoided if we know the dirstate file is up-to-date. This patch avoids the read and write if the dirstate is clean. In that case, we just do a plain copy without any serialization. This should make commands which use transactions but do not touch dirstate faster. For example, "hg bookmark -r REV NAME".

dirstate: try to use hardlink to backup dirstate This should be more efficient once util.copyfile has real hardlink support.

dirstate: track updated files to improve write time Previously, dirstate.write() would iterate over the entire dirstate to find any entries that needed to be marked 'lookup' (i.e. if they have the same timestamp as now). This was O(working copy) and slow in large repos. It was most visible when rebasing or histediting multiple commits, since it gets executed once per commit, even if the entire rebase/histedit is wrapped in a transaction. The fix is to track which files have been editted, and only check those to see if they need to be marked as 'lookup'. This saves 25% on histedit times in very large repositories. I tested this by adding temporary debug logic to verify that the old files processed in the loop matched the new files processed in the loop and running the test suite.

hook: give exthooks tags for blocking time The ui.system autogenerated tag isn't really useful - as they're named, let's use the name the user gave us.

filemerge: tag merge tool for blocked times Merge tools can take a while - let's ensure that they're appropriately tagged

hgk: set a blocked tag when the user invokes view

transplant: set a blockedtag when invoking external filter

config: set blockedtag when invoking configuration edit