filectx: add a rawflags method The new method returns the low-level revlog flag. We already have "rawdata" so a "rawflags" makes sense. Both "rawflags" and "rawdata" will be used in a later patch.

filectx: move size to basefilectx See previous patch for context - avoid code duplication.

filectx: make renamed a property cache See previous patch for context - mainly to avoid code duplication.

filectx: make flags a property cache Make basefilectx._flags a property cache, so basefilectx.flags() could be reasonably reused. This avoids code duplication between memfilectx and a class added in a later patch.

commandserver: move printbanner logic to bindsocket bindsocket now handles listen automatically. "printbanner" seems to be just a part of "bindsocket". This simplifies the interface a bit.

commandserver: move "listen" responsibility from service to handler This enables chg to replace a server socket in an atomic way: 1. bind to a temp address 2. listen 3. rename Currently 3 happens before 2 so a client may see the socket file but fails to connect to it.

hghave: remove py27+ capability It is now unused. And we require Python 2.7+ now so this check is not necessary.

tests: remove test targeting Python 2.6 We just removed support for Python 2.7. This test is dead since it only ran on <2.7.

setup: drop support for Python 2.6 (BC) Per discussion on the mailing list and elsewhere, we've decided that Python 2.6 is too old to continue supporting. We keep accumulating hacks/fixes/workarounds for 2.6 and this is taking time away from more important work. So with this patch, we officially drop support for Python 2.6 and require Python 2.7 to run Mercurial.

perf: move revlog construction and length calculation out of benchmark We don't need to measure the time it takes to open the revlog or calculate its length. This is more consistent with what other perf* functions do. While I was here, I also renamed the revlog variable from "r" to "rl" - again in the name of consistency.

perf: clear revlog caches on every iteration cmdutil.openrevlog() may return a cached revlog instance. This /may/ be a recent "regression" due to refactoring of the manifest API. I'm not sure. Either way, this perf command was broken for at least manifests because subsequent invocations of the perf function would get cache hits from previous invocations, invalidating results. In the extreme case, testing the last revision in the revlog resulted in near-instantanous execution of subsequent runs (since the fulltext is cached). A time of ~1us would be reported in this case.

perf: don't convert rev to node before calling revlog.revision()

revlog: rename _chunkraw to _getsegmentforrevs() This completes our rename of internal revlog methods to distinguish between low-level raw revlog data "segments" and higher-level, per-revision "chunks." perf.py has been updated to consult both names so it will work against older Mercurial versions.

perf: store reference to revlog._chunkraw in a local variable To prepare for renaming revlog._chunkraw, we stuff a reference to this metho in a local variable. This does 2 things. First, it moves the attribute lookup outside of a loop, which more accurately measures the time of the code being invoked. Second, it allows us to alias to different methods depending on their presence (perf.py needs to support running against old Mercurial versions). Removing an attribute lookup from a tigh loop appears to shift the numbers slightly with mozilla-central: $ hg perfrevlogchunks -c ! read ! wall 0.354789 comb 0.340000 user 0.330000 sys 0.010000 (best of 28) ! wall 0.335932 comb 0.330000 user 0.290000 sys 0.040000 (best of 30) ! read w/ reused fd ! wall 0.342326 comb 0.340000 user 0.320000 sys 0.020000 (best of 29) ! wall 0.332857 comb 0.340000 user 0.290000 sys 0.050000 (best of 30) ! read batch ! wall 0.023623 comb 0.020000 user 0.000000 sys 0.020000 (best of 124) ! wall 0.023666 comb 0.020000 user 0.000000 sys 0.020000 (best of 125) ! read batch w/ reused fd ! wall 0.023828 comb 0.020000 user 0.000000 sys 0.020000 (best of 124) ! wall 0.023556 comb 0.020000 user 0.000000 sys 0.020000 (best of 126)

revlog: rename internal functions containing "chunk" to use "segment" Currently, "chunk" is overloaded in revlog terminology to mean multiple things. One of them refers to a segment of raw data from the revlog. This commit renames various methods only used within revlog.py to have "segment" in their name instead of "chunk." While I was here, I also made the names more descriptive. e.g. "_loadchunk()" becomes "_readsegment()" because it actually does I/O.

fsmonitor: do not nuke dirstate filecache In the future, chg may prefill repo's dirstate filecache so it's valuable and should be kept. Previously we drop both filecache and property cache for dirstate during fsmonitor reposetup, this patch changes it to only drop property cache but keep the filecache.

perf: move gettimer() call This is more consistent with other perf* functions.

perf: don't clobber startrev variable Previously, the "startrev" argument would be ignored due to "startrev = 0" in the benchmark function. This meant that `hg perfrevlog` always started at revision 0. Rename the local variable to "beginrev" so the variable does the right thing.

bundle: add optional 'tagsfnodecache' data to on disk bundle (issue5543) This should help performance when unbundling.

bundle2: move tagsfnodecache generation in a generic function This will help us reusing the logic for `hg bundle`.

bundle: introduce an higher level function to write bundle on disk The current function ('writebundle') is focussing on getting an existing changegroup to disk. It is no easy ways to includes more part in the generated bundle2. So we introduce a slightly higher level function that is fed the 'outgoing' object (that defines the bundled spec) and the bundlespec parameters (to control the changegroup generation and inclusion of other parts). This is creating the third logic dedicated to create a consistent bundle2 (the other 2 are the push code and the getbundle code). We should probably reconcile them at some points but they all takes different types of input. So we need to introduce an intermediate "object" that each different input could be converted to. Such unified "bundle2 specification" could be fed to some unified code. We start by having the `hg bundle` related code on its own to helps defines its specific needs first. Once the common and specific parts of each logic will be known we can start unification.

bundle: handle compression earlier We can also handle that part before starting any generation.

bundle: check changegroup version earlier We can check if we know how to bundle this changegroup version before actually starting to generate the changegroup.

bundle: check lack of revs to bundle before generating the changegroup We already have the information so we can check it earlier.

extdiff: copy back files to the working directory if the size changed In theory, it should be enough to pay attention only to the modification time when detecting if a snapshotted working directory file changed. In practice, BeyondCompare preserves all file attributes when syncing files at the directory level. (If you open the file and sync individual hunks, then mtime does change, and everything was being copied back as desired.) I'm not sure how many other synchronization tools would trigger this issue, but it's annoyingly inconsistent (if a single file is diffed, it isn't snapshotted, so the same BeyondCompare file sync operation _is_ visible, because wdir() is updated in place. I filed a bug with them, and they stated it is on their wish list, but won't be fixed in the near term. This isn't a complete fix (there is still the case of the size not changing), but this seems like a trivial enough change to fix most of the problem. I suppose we could fool around with making files in the other snapshot readonly, and copy back if we see the readonly bit copied. That seems pretty hacky though, and only works if the external tool copies all attributes.

test-extdiff: enable a previously failing test on Windows

test-extdiff: narrow the range of an '#if execbit' block Now that output can be conditionalized, the few `chmod +x` specific outputs can be conditionalized, and the rest of the tests run as normal. Disable one test that is failing on Windows for now.

test-extdiff: deduplicate tests

test-extdiff: fill in a missing Windows test

policy: eliminate ".pure." from module name only if marked as dual So we can switch cext/pure modules to new layout one by one.

policy: add "cext" package which will host CPython extension modules I'm going to restructure cext/pure modules and get rid of our hgimporter hack. C extension modules will be moved to cext/ directory so old and new compiled modules can coexist in development tree. This is necessary to run 'hg bisect' without recompiling. New extension modules will be loaded by an importer function: base85 = policy.importmod('base85') # select pure.base85 or cext.base85 This will also allow us to split cffi from pure modules, which is currently difficult because pure modules can't be imported by name.

policy: mark all string literals as sysstr or bytes The policy module won't be imported early in future, which means string literals will be processed by our Python 3 loader.

debuginstall: check C extensions only if they are loadable per policy This check is useless in pure installation and I want to make it directly import C extension modules.

osutil: proxy through util (and platform) modules (API) See the previous commit for why. Marked as API change since osutil.listdir() seems widely used in third-party extensions. The win32mbcs extension is updated to wrap both util. and windows. aliases.

win32mbcs: wrap underlying pycompat.bytestr to use checkwinfilename safely win32mbcs wraps some functions, to prevent them from unintentionally treating backslash (0x5c), which is used as the second or later byte of multi bytes characters by problematic encodings, as a path component delimiter on Windows platform. This wrapping assumes that wrapped functions can safely accept unicode string arguments. Unfortunately, d1937bdcee8c broke this assumption by introducing pycompat.bytestr() into util.checkwinfilename() for py3 support. After that, wrapped checkwinfilename() always fails for non-ASCII filename at pycompat.bytestr() invocation. This patch wraps underlying pycompat.bytestr() function to use util.checkwinfilename() safely. To avoid similar regression in the future, another patch series will add smoke testing on default branch.

hghave: prefill more version of Mercurial The previous code was unable to go above version 4.0.

graft: fix graft across merges of duplicates of grafted changes Graft used findmissingrevs to find the candidates for graft duplicates in the destination. That function operates with the constraint: 1. N is an ancestor of some node in 'heads' 2. N is not an ancestor of any node in 'common' For our purpose, we do however have to work correctly in cases where the graft set has multiple roots or where merges between graft ranges are skipped. The only changesets we can be sure doesn't have ancestors that are grafts of any changeset in the graftset, are the ones that are common ancestors of *all* changesets in the graftset. We thus need: 2. N is not an ancestor of all nodes in 'common' This change will graft more correctly, but it will also in some cases make graft slower by making it search through a bigger and unnecessary large sets of changes to find duplicates. In the general case of grafting individual or linear sets, we do the same amount of work as before.

graft: test coverage of grafts and how merges can break duplicate detection This demonstrates unfortunate behaviour: extending the graft range cause the graft to behave differently. When the graft range includes a merge, we fail to detect duplicates that are ancestors of the merge.

mpatch: proxy through mdiff module See the previous commit for why.

bdiff: proxy through mdiff module See the previous commit for why. mdiff seems a good place to host bdiff functions. bdiff.bdiff was already aliased as textdiff, so we use it.

base85: proxy through util module I'm going to replace hgimporter with a simpler import function, so we can access to pure/cext modules by name: # util.py base85 = policy.importmod('base85') # select pure.base85 or cext.base85 # cffi/base85.py from ..pure.base85 import * # may re-export pure.base85 functions This means we'll have to use policy.importmod() function in place of the standard import statement, but we wouldn't want to write it every place where C extension modules are used. So this patch makes util host base85 functions.

mdiff: move re-exports to top This style seems more common in our codebase.

test-commit-interactive-curses: remove unused import of parsers

churn: use the non-deprecated template option in the examples

strip: make tree stripping O(changes) instead of O(repo) The old tree stripping logic iterated over every tree revlog in the repo looking for commits that had revs to be stripped. That's very inefficient in large repos. Instead, let's look at what files are touched by the strip and only inspect those revlogs. I don't have actual perf numbers, since internally we don't use a true treemanifest, but simply iterating over hundreds of thousands of revlogs takes many, many seconds, so this should help tremendously when stripping only a few commits.

strip: move tree strip logic to it's own function This will allow external extensions to modify tree strip behavior more precisely.

manifest: remove unused property _oldmanifest The last use seems to have gone away in 7c7d845f8b64 (manifest: make manifestlog use it's own cache, 2016-11-10).

sslutil: reference fingerprints config option properly (issue5559) The config option is "host:fingerprints" not "host.fingerprints". This warning message is bad and misleads users.

py3: convert key to str to make kwargs.pop work in mq The keys are passed here and there as unicodes and our transformer make things bytes. Due to that, mq was not poped and this results in error on Py3. Here we abuse r'' to make that str on Python 3.

py3: convert kwargs' keys to str before passing in cmdutil.getcommiteditor

diff: add a fast path to avoid loading binary contents When diffing binary contents, with certain configs, we can show "Binary file <name> has changed" without actual content. That allows a fast path where we could avoid providing actual binary contents. Note: in that case we still need to test if two contents are the same, that's done by using "filectx.cmp", which could have its own fast path.

diff: correct binary testing logic This seems to be more correct given the table drawn in the previous patch. Namely, "losedatafn" and "opts.git" are removed, "not opts.text" is added. - losedatafn: diff output (binary) should not be affected by "losedatafn" - opts.git: binary testing is helpful for detecting a fast path in the next path. the fast path can also be used if opts.git is False - opts.text: if it's set, we should treat the content as non-binary

diff: draw a table about binary diff behaviors The table should make it easier to reason about future changes.

diff: use fctx.size() to test empty fctx.size() could have a fast path that does not require loading content.

diff: use fctx.isbinary() to test binary The end goal is to avoid calling fctx.data() when unnecessary. For example, if diff.nobinary=1 and files are binary, the expected behavior is to print "Binary file has changed". That could avoid reading fctx.data() sometimes. This is mainly to enable an external LFS extension to skip expensive binary file loading sometimes (read: most of the time with diff.nobinary=1 and diff.text=0), without any behavior changes to mercurial (i.e. whether a file is LFS or not does not change any behavior, LFS could be 100% transparent to users).

pycompat: extract helper to raise exception with traceback It uses "raise excobj, None, tb" form which I think is simpler and more useful than "raise exctype, args, tb".

largefiles: make sure debugstate command is populated before wrapping Copied the hack from 869d660b8669, which seemed the simplest workaround. Perhaps debugcommands.py should have its own commands table.

check-code: ignore re-exports of os.environ in encoding.py These are valid uses of os.environ.

check-code: exclude demandimport.py and policy.py from Python 3 checks These modules can't depend on pycompat.py, which means we have to write Py3 hacks in them.

check-code: rewrite py3 exclusion pattern with negative lookahead I want to add more patterns, but negative lookbehind requires patterns of the same length so not useful.

cleanup: remove useless re-raises of KeyboardInterrupt KeyboardInterrupt is no longer a subclass of Exception since Python 2.5. https://docs.python.org/2/whatsnew/2.5.html#pep-352-exceptions-as-new-style-classes

make: drop deprecated rule to process temporary copy of pure modules Pure modules never be copied to mercurial/ since 511a4384b033.

help: describe about choice of :prompt as a fallback merge tool explicitly "merge-tools" help topic has described that the merge of the file fails if no tool is found to merge binary or symlink, since c77f6276c9e7 (or Mercurial 1.7), which based on (already removed) MergeProgram wiki page. But even at that revision, and of course now, merge of the file doesn't fail automatically for binary/symlink. ":prompt" (or equivalent logic) is used, if there is no appropriate tool configuration for binary/symlink.

wix: only one KeyPath is allowed per Component

dirstate: optimize walk() by using match.visitdir() We already have the logic for restricting directory walks in match.visitdir() that we use for treemanifests. We should take advantage of it when walking the working copy as well. This speeds up "hg st -I rootfilesin:." on the Firefox repo from 0.587s to 0.305s on warm disk (and much more on cold disk). More time is spent reading the dirstate than walking the working copy after. I tried to find scenarios where calling match.visitdir() would be a noticeable overhead, but I couldn't find any. I encourage the reader to try for themselves, since this is performance-critical code.

match: optimize visitdir() for patterns matching only root directory Because _rootsanddirs() returns a list of directories to visit recursively and a list of directories to visit non-recursively. For patterns such as 'rootfilesin:foo/bar', we clearly need to visit the directory foo/bar, but we also need to visit its parents. The method therefore uses util.dirs() to find the parent directories of 'foo/bar'. That method does not include the root directory, but since we obviously need to visit the root directory, we always added '.' to the set of directories to visit non-recursively. The visitdir() method had special handling to consider set(['.']) to mean that no includes had been specified and would thus visit all directories. However, when the pattern is 'rootfilesin:.', set(['.']) is actually the real set of directories to visit and the special handling of that set meant that all directories got visited instead of just the root directory. The fix is simple: add '.' to the set of parent directories in _rootsanddirs() and stop treating set(['.']) specially. This makes hg files -r . -I rootfilesin:. in a treemanifest version of the Firefox repo go from 1.5s to 0.26s on warm disk (and a *much* bigger improvement on cold disk). Note that the -I is necessary for no good reason. We just haven't optimized visitdir() for regular (non-include, non-exclude) patterns yet.

rebase: don't update state dict same way for each root The update statement does not depend on anything in the loop, so just move it before the loop and do it once. There are no cases where update would happen 0 times before (and 1 now); the function returns early in all such cases.

forget: access status fields by name, not index

demandimport: add urwid.command_map to ignore list The useful pudb debugger can be used with Mercurial, but its import of urwid fails when demandimport is enabled. Add urwid.command_map to the ignore list so pudb can be used with hg without disabling all of demandimport.

outgoing: run on filtered repo outgoing has been using an unfiltered repo since fe67107094fd (discovery: outgoing pass unfiltered repo to findcommonincoming (issue3776), 2013-01-28). If I'm reading code and history correctly, it should be safe to run _outgoing() on a filtered repo since c5456b64eb07 (discovery: run discovery on filtered repository, 2015-01-07). By running _outgoing() on a filtered repo, we can also remove the workaround there for ignoring filtered revisions.

manifest: remove check for non-contexts in _dirmancache It looks like the _dirmancache has contained only manifest contexts since d79c141fdf41 (manifest: remove usages of manifest.read, 2016-11-10).

bundle: factor the 'getchangegroup' out The call in the two branches is identical, so we can just issue it outside of the conditional.

bundle: avoid reset of the 'outgoing' variable We have a cleaner way to achieve the same effect. Not resetting the variable will help us to simplify the code.

changegroup: deprecate 'getlocalchangroup' (API) We have 'getchangegroup' with a shorter name for the exactly same feature. Now that all users are gone we can formally deprecate it.

tests: directly 'getchangegroup' It is identical to 'getlocalchangegroup' with a shorter name.

exchange: directly 'getchangegroup' It is identical to 'getlocalchangegroup' with a shorter name.

commands: directly 'getchangegroup' It is identical to 'getlocalchangegroup' with a shorter name.

changegroup: deduplicate 'getlocalchangegroup' The two functions 'getlocalchangegroup' and 'getchangegroup' have been strictly identical for multiple years ('getchangegroup' had a deprecated docstring) We'll drop one of them (getlocalchangegroup, since it has the longest name). However, we needs to migrate all users of the dropped one to the new one before we can deprecate it. In the mean time we drop one of the duplicated definition and the outdated docstring.

py3: add test to show 'hg log -Tjson' works

py3: add test to show 'hg log -G' works

py3: rename test-check-py3-commands.t to test-py3-commands.t test-check-*.t is a set of tests which tests certain coding style checks. So this test was wrongly named, thanks to marmoute for pointing this out.

py3: use list of bytes rather than bytestring while extending bytes into lists Python 2: >>> ls = [] >>> ls.extend(b'abc') >>> ls ['a', 'b', 'c'] Python 3: >>> ls = [] >>> ls.extend(b'abc') >>> ls [97, 98, 99] So to make sure things work fine in Py3, this patch does: >>> ls = [] >>> ls.extend([b'a', b'b', b'c']) >>> ls [b'a', b'b', b'c'] After this `hg log -G` works!

py3: use pycompat.byteskwargs to converts kwargs to bytes baseformatter._item must contain both keys and values in bytes. So to make sure that, we convert the opts back to bytes.

py3: make adefaults keys str to be compatible with getattr getattr passes a str value of the attribute to be looked and keys in adefaults dict are bytes which resulted in AttributeError. This patch abuses r'' to make the keys str.

py3: abuse r'' to access keys in keyword arguments

py3: use pycompat.bytechr instead of chr

py3: use %d to format integers into bytestrings

py3: use pycompat.bytestr instead of bytes

py3: slice over bytes to prevent getting ascii values

py3: use encoding.unitolocal instead of .encode(encoding.encoding)

rebase: use matcher to optimize manifestmerge The old merge code would call manifestmerge and calculate the complete diff between the source to the destination. In many cases, like rebase, the vast majority of differences between the source and destination are irrelevant because they are differences between the destination and the common ancestor only, and therefore don't affect the merge. Since most actions are 'keep', all the effort to compute them is wasted. Instead, let's compute the difference between the source and the common ancestor and only perform the diff of those files against the merge destination. When using treemanifest, this lets us avoid loading almost the entire tree when rebasing from a very old ancestor. This speeds up rebase of an old stack of 27 commits by 20x. In mozilla-central, without treemanifest, when rebasing a commit from default~100000 to default, this speeds up the manifestmerge step from 2.6s to 1.2s. However, the additional diff adds an overhead to all manifestmerge calls, especially for flat manifests. When rebasing a commit from default~1 to default it appears to add 100ms in mozilla-central. While we could put this optimization behind a flag, I think the fact that it makes merge O(number of changes being applied) instead of O(number of changes between X and Y) justifies it.

changegroup: delete unused 'bundlecaps' argument (API)

localrepo: reuse exchange.bundle2requested() It seems like localrepo.getbundle() is trying to do the same thing, so let's just call the method. That way we get the same condition as there (matching any "HG2" prefix, not only "HG20").

py3: use raw strings while accessing class.__dict__ The keys of class.__dict__ are unicodes on Python 3.

py3: handle opts correctly for `hg add` opts in add command were passed again to cmdutil.add() as kwargs so we need to convert them again to str. Intstead we convert them to bytes when passing scmutil.match(). Opts handling is also corrected for all the functions which are called from cmdutil.add().

py3: handle opts correctly for rollback dryrun and force are just check for None, the value is not used. So its better to leave opts as unicodes as that wont harm us.