bundle2: move function building obsmarker-part in the bundle2 module We move it next to similar part building functions. We will need it for the "writenewbundle" logic. This will allow us to easily include obsmarkers in on-disk bundle, a necessary step before having `hg strip` also operate on markers. (Yes, the bundle2 module was already too large, but there any many interdependencies between its components so it is non-trivial to split, this is a quest for another adventure.)

policy: remove unused policynoc and policynocffi constants

cffi: remove superfluous "if True" blocks

cffi: split modules from pure The copyright lines are updated per change history. cffi/osutil.py isn't tested since I have no access to OS X machine right now, sorry.

policy: extend API version checks for cffi This is just a stub for future extension. I could add a version constant to CFFI modules by putting it to both ffi.set_source() and ffi.cdef(), but that doesn't seem right. So for now, cffi modules will be explicitly unversioned (i.e. version constant must be undefined or set to None.) We can revisit it later when we need to consider CFFI support more seriously.

filterpyflakes: allow reexporting pure symbols from cffi modules cffi modules will do 'from ..pure.<module> import *'.

import-checker: guess names of C extension modules Since extension modules aren't included in the list of source files, they need to be populated somehow. Otherwise the import from cext/cffi would be treated as a global one.

import-checker: convert localmods to a set of module names This makes it easy to add a source-less module name to the set.

import-checker: allow importing symbols from pure modules This allows us to re-export pure functions from cffi modules: # mercurial/cffi/base85.py from ..pure.base85 import *

cffi: put compiled modules into mercurial.cffi package Don't pollute the top-level namespace.

cffi: rename build scripts This frees up cffi package for modules to be split from pure.

match: remove support for includes from patternmatcher Includes (and excludes) are now delegated to the includematcher.

match: simplify includematcher a bit The "include" we have in symbols is redundant and the double negative in visitdir() can be removed.

match: remove support for non-include patterns from includematcher The includematcher will always get at least one include pattern and will never get any non-include patterns, so we can remove most of the code in it. This patch does mostly straight-forward deletions of code. We will clean up further later.

match: split up main matcher into patternmatcher and includematcher At this point the includematcher is an exact copy of the main matcher class. We will specialize and simplify both classes in the following patches. This initial unmodified copy is just to make the differences clearer. We also rename the main matcher to "patternmatcher" for consistency. I may eventually merge this new includematcher back into the main matcher, but I think doing it this way makes the intermediate steps clearer regardless.

match: remove support for exact matching from main matcher class Exact matching is now handled by the exactmatcher class. We can safely remove _files from the __repr__() implementation, because even though the field is set, the patternspat field is enough for the representation to be unambiguous (which was not the case when the matcher could handle exact matches).

match: handle exact matching using new exactmatcher

merge: use intersectmatchers() in "m2-vs-ma optimization" It doesn't seem like this can actually happen, but seems like cleaner anyway.

match: handle includes using new intersectionmatcher

match: move entire uipath() implementation to basematcher Even though most matchers will always want to use the relative path in uipath(), when we add support for intersecting matcher, we will want to control which form to use for any kind of matcher without knowing the type (see next patch), so we need the implementation on the base class. Also rename the attribute from "pathrestricted" to "relativeuipath" since there actually are cases where we match everything but still use relative paths (like when the user runs "hg files .." from inside mercurial/).

local-clone: also copy tags related caches This caches provide a large speedup for some repositories. Keeping it around is valuable.

local-clone: also copy revs-branch-cache files This cache provides a large speedup for some repositories. Keeping it around is valuable.

local-clone: extract the listing of caches to copy Right now, the clone only copies the branchmap caches. There are multiple other valuable caches that we should copy and extensions might add their own. So we add a function to list the cache files to copy from the repository. The repository argument is unused but extensions will want it.

local-clone: extract the closure copying caches Closures often get on the way. They are not much value in having that as a closure so I'm extracting it at the module level.

test: add isolated prune case (to test-obsolete-bundle-strip.t) This adds a test where the prune marker is not related to any other obsmarkers.

test-obsolete-bundle-strip: add a complex split and fold case This is a more complex case that checks the logic used when split and fold gets into play.

test-obsolete-bundle-strip: add cases with prune on missing revs Same as the previously added case, but the prune is no longer known locally. This will mostly matter for the strip testing. Introducing the test early will help clarify patches related to strip.

obsolete: fix relevant-obsmarkers computation on pruned changeset The markers pruning a node was not directly considered relevant for the pruned node, only to its parents. This went unnoticed during obsmarkers exchange because all ancestors of the pruned node would be included in the computation. This still affects obsmarkers exchange a bit since "inline" prune markers would be ignored (see second test case). This went unnoticed, because in such case, we always push another obsolescence markers for that node. We add explicit tests covering this case. (The set of relevant changeset is use in the obsmarkers discovery protocol used in the evolve experimental extension, the impact will be handled on the extension side).

test: add a test file for relevant obsmarkers and its usage The logic around obsmarkers "relevant" to a set of revs have a couple of test around in other places but no systematic testing. In addition, all the current testing focus on the exchange case (we looks at relevant markers for '::heads'). For bundles, we'll need something a bit different. We'll no longer have set of revision going down to the repository roots. So we'll have to test these cases too. In addition, stripping obsmarkers will introduce new logic around obsmarkers that will need testing too. So a new test file make sense here. We start with a simple tests, more advanced cases are coming in the next changesets. The extra testing catch a minor bug (later in the series).

annotate: add a new experimental --skip option to skip revs This option is most useful for mechanical code modifications, especially ones that retain the same number of lines.

annotate: add core algorithm to skip a rev The core algorithm is inspired by git hyper-blame, implemented at https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/master/git_hyper_blame.py. The heuristic is as documented in the comments.

annotate: make pair take all parents to pair against In upcoming patches we'll need to be aware of all parents at the same time. This also exposes a potential bug: if a line can be annotated with both parents of a merge commit, it'll always be annotated with p2, not p1. I'm not sure if that's what we want, but at least the code makes it clear now.

annotate: move pair function to top level We'll want to make this more complicated and have unit tests for it in upcoming patches.

bookmarks: fix check of hash-like name to not abort by ambiguous identifier 'mark in repo' may raise LookupError. I set it to not be warned since bookmark names shorter than 4 chars aren't checked and short names are likely to be ambiguous.

localrepo: document that __contains__() may raise LookupError

hidden: drop outdated comment about "dynamic" performance This comment is now irrelevant since we have a faster algorithm and no cache.

hidden: unify the static and dynamic blocker logic We no longer have cache and they both work the same way. Unifying the logic simplify the code and reduce the amount of set copies.

hidden: drop the hidden cache logic The improvement in time complexitty and the speed-up in computation is large enough that the has little use now. Its update time can even gets in the way. So we drop it. This will allow us to unify the static/dynamic blockers logic in the next changeset.

hidden: simplify the computation of consistency blocker For a couple of years, we now have precomputed set for all mutable phases. We can use this set restrict our search and quickly detect non-hideable children of hideable changesets. This speeds up the hidden computation. See docstring of the new function for details. This new version reuses the '_domainancestors' function to keep the computation of revealed changeset in O(len(visible)) Below are perfvolatilesets timing from two Mozilla repositories with different contents. hidden cache is disabled while obtaining them. 1) Mozilla repository with: * 400667 changesets * 35 hidden changesets (first rev-268334) * 288 visible drafts * 1 unstable changeset Before: ! visible ! wall 0.001744 comb 0.000000 user 0.000000 sys 0.000000 (best of 1563) After: ! visible ! wall 0.000742 comb 0.000000 user 0.000000 sys 0.000000 (best of 3755) The timing above include the computation of obsolete changeset: ! obsolete ! wall 0.000396 comb 0.000000 user 0.000000 sys 0.000000 (best of 6816) So adjusted time give 1.3ms before versus 0.3ms after. A 4x speedup. 2) Mozilla repository with: * 405645 changesets * 4312 hidden changesets (first rev-326004) * 264 visible drafts * 1 unstable changeset Before: ! visible ! wall 0.025476 comb 0.030000 user 0.030000 sys 0.000000 (best of 111) After ! visible ! wall 0.007703 comb 0.010000 user 0.010000 sys 0.000000 (best of 358) The timing above include the computation of obsolete changeset: ! obsolete ! wall 0.006408 comb 0.010000 user 0.010000 sys 0.000000 (best of 404) So adjusted time give 19ms before versus 1.3ms after. A 17x speedup.

hidden: use _domainancestors to compute revs revealed by dynamic blocker The complexity of computing the revealed changesets is now 'O(revealed)'. This massively speeds up the computation on large repository. Moving it to the millisecond range. Below are timing from two Mozilla repositories with different contents: 1) mozilla repository with: * 400667 changesets * 35 hidden changesets (first rev-268334) * 288 visible drafts * obsolete working copy (dynamicblockers), Before: ! visible ! wall 0.030247 comb 0.030000 user 0.030000 sys 0.000000 (best of 100) After: ! visible ! wall 0.000585 comb 0.000000 user 0.000000 sys 0.000000 (best of 4221) The timing above include the computation of obsolete changeset: ! obsolete ! wall 0.000396 comb 0.000000 user 0.000000 sys 0.000000 (best of 6816) So adjusted time give 30ms before versus 0.2ms after. A 150x speedup. 2) mozilla repository with: * 405645 changesets * 4312 hidden changesets (first rev-326004) * 264 visible drafts * obsolete working copy (dynamicblockers), Before: ! visible ! wall 0.168658 comb 0.170000 user 0.170000 sys 0.000000 (best of 48) After ! visible ! wall 0.008612 comb 0.010000 user 0.010000 sys 0.000000 (best of 325) The timing above include the computation of obsolete changeset: ! obsolete ! wall 0.006408 comb 0.010000 user 0.010000 sys 0.000000 (best of 404) So adjusted time give 160ms before versus 2ms after. A 75x speedup.

hidden: add a function returning ancestors of revs within a domain See documentation for details. This will be used to improve the hidden computation algorithm. See new changesets for usage.

tests: hint how to run slow tests when rejecting

zsh_completion: install as _hg not hg The contrib/zsh_completion file itself says to name it _hg. With a name like `hg`, if the user has a line like `autoload ${^fpath}/*(N-.:t)` in their zshrc, it will create a shell function named `hg` that will hide the actual hg command and make hg unusable. Separately from that though, the underscore prefix makes it actually work. The zsh man page states: The convention for autoloaded functions used in completion is that they start with an underscore This does not seem to just be a "convention", though. With the ill-advised line removed from my zshrc and the file named `/usr/local/share/zsh/site-functions/hg` (without the underscore), these completions did not seem to get loaded and the ones from the zsh installation were loaded instead. If I renamed them to be `/usr/local/share/zsh/site-functions/_hg`, however, they were loaded. I manually tested the above statement by starting a new zsh instance with the file in `/usr/local/share/zsh/site-functions` with the following names: - As `hg`, `which _hg_labels` did not show anything - As `_hg`, `which _hg_labels` showed the expected function.

osx: include chg by default

osx: override default exclude filter for pkgbuild To quote `man 1 pkgbuild`: --filter filter-expression By default, --root will include the entire contents of the given root-path in the package payload, except for any .svn or CVS directories, and any .DS_Store files. You can override these default filters by specifying one or more --filter options. Each filter-expression is an re_format(7) ``extended'' expression: any path in the root which matches any of the given expressions will be excluded from the pack- age payload. (Note that specifying even one --filter inhibits the default filters, so you must respecify the default fil- ters if you still want them to be used.) It turns out the default filter these days *also* includes .git and .hg. Notice how that filter expression is a regular expression? That (presumably unintentionally) prevents a file named "chg" or "_hg" from getting included in the distribution. Many many thanks to spectral@ for trying to include a _hg file which led us to figure this bug out. Bug filed with Apple for this as rdar://problem/32437369, mentioning both the gap in documentation and the wrong defaults.

osx: update Mac packaging tests for bdiff.so install location change

strip: use the 'writenewbundle' function to get bundle on disk This will ensure the backup bundle use the best available logic (eg: includes relevant caches so that we loose less of them on strip.)

perfphases: add 'perfphases' command This commands benchmark the time spend computing the data in a repository.

match: remove support for excludes from matcher class The support is now provided by differencematcher() and still available via the match() function.

match: handle excludes using new differencematcher As I've said on earlier patches, I'm hoping to use more composition of simpler matchers instead of the single complex matcher we currently have. This extracts a first new matcher that composes two other matchers. It matches if the first matcher matches but the second does not. As such, we can use it for excludes, which this patch also does. We'll remove the now-unncessary code for excludes in the next patch.

match: override matchfn() the usual way in subdirmatcher

match: make matchfn a method on the class This makes it easier to override in subclasses, so they don't have to assign the attribute with a lambda.

util: raise ParseError when parsing dates (BC) a7dce526c462 refactored util.parsedate in order to raise ValueError instead of Abort for using with ui.configwith. It causes several problems, putting arbitrary bytes in ValueError can cause issues with Python 3. Moreover, we added a function to convert ValueError exceptions back to Abort. A better approach would be to make parsedate raises ParseError, removing the convert function and update configwith to also catch ParseError. The side-effect is that error message when giving an invalid date in CLI change from: abort: invalid date: 'foo bar' to: hg: parse error: invalid date: 'foo bar' I'm not sure if it's an acceptable change, I found personally the error message more clear but more verbose too.

match: fix visitdir for roots of includes I'm hoping to rewrite the matcher so excludes are handled by composition of one matcher with another matcher where the second matcher has only includes. For that to work, we need to make visitdir() to return 'all' for directory 'foo' for a '-I foo' matcher.

match: make subdirmatcher extend basematcher This makes the subdirmatcher not depend on the main matcher, giving us more freedom to modify that (specifically, it will lose it _always field in a while).

match: make basematcher._files a @propertycache This will make it easier to override in subclasses (otherwise the function @propertycache object will be replaced by the super-constructor call)..

match: extract base class for matchers We will soon start splitting up the current matcher class into more specialized classes, so we'll want a base class for all the things that don't vary much between different matchers.

debugwalk: also print matcher representation This will make the effect of coming patches clearer.

transaction: run _writejournal unfiltered The function use the length of the repository, something affected by filtering. It seems better to use the unfiltered length here. Credit for finding this goes to Durham Goode.

bookmarks: warn about bookmark names that unambiguously resolve to a node (BC) I just burned myself on this today because I left out the -r in my `hg bookmark` command, which then left me confused because I didn't notice the bookmark I created in the wrong place that was silently shadowing the revision I was trying to check out. Let's warn the user. This patch only enforces the check on bookmark names 4 characters long or longer. We can tweak that if we'd like, I selected that since that's the fewest characters shortest will use in the templater output. A previous version of this patch rejected such bookmarks. It was proposed during review (and I agree) that the behavior change for a bookmark named "cafe" or similar as history accumulated was a little too weird, but that the warning definitely has merit.

pycompat: try __bytes__() to convert object to bytestr It should be better than using __str__() unconditionally.

ui: fix ui.configdate for invalid dates a7dce526c462 introduced util._parsedate with the aim to be used in ui.configdate but ui.configdate was using util.parsedate instead. It have the impact of raising an AbortError in case of an invalid date instead of a ConfigError exception. Fix ui.configdate to use the right function and add a test for invalid dates. Thanks to Yuya for the catch!

demandimport: stop overriding __getattribute__() Proxy __dict__ and __doc__ explicitly instead. I'm not sure which is less evil, but this seems slightly simpler than hooking all attribute accesses.

demandimport: look for 'mod' suffix as alternative name for module reference It's widely used in our codebase.

demandimport: insert empty line per method _demandmod class is getting bigger, and I don't want to put more things in a dense form.

demandimport: strictly compare identity of proxy object This looks better, and __eq__() may be overridden in an undesired way.

match: use ProgrammingError where appropriate

revlog: raise error.WdirUnsupported from revlog.node() if wdirrev is passed When we try to run, 'hg debugrevspec 'branch(wdir())'', it throws an index error and blows up. Lets raise the WdirUnsupported if wdir() is passed so that we can catch that later.

revset: add support for ancestors(wdir()) This is a part of extending support for wdir() predicate.

revset: add support for using ~ operator on wdir() predicate We catch the WdirUnsupported exception and handles the wdir() case here.

revset: add support for p2(wdir()) to get second parent of working directory This adds support for finding the second parent of working directory using the p2 predicate.

revset: use try-except instead of if-else because of perf For wdir(), we now raises an exception which will be raised when wdir() will be passed, so catching that exception is better checking for wdir() using if-else.

revset: remove redundant condition and change to else from elif Before going to this piece of code, we already check whether n is in [0, 1, 2], so dropping the redundant condition.

tests: add tests for predicates and operators which works with wdir() This one is cluttering up the test file though.

revset: add support for "wdir()^n" This patch catches the WdirUnsupported exception raised, and adds support for wdir^n which will give us the nth parent of the working directory.

revset: add parentheses after wdir since its a function wdir was shown in help rather than wdir().

cmdutil: avoid redefining write() function in export if possible Doing less work inside the loop just feels better to me.

cmdutil: extract closure that performs the actual export formatting This simplifies things a little by making the actual act of turning a revision into patch data a single function. After this, adding formatter support to `hg export` should be much simpler.

cmdutil: use a generator expression instead of a list comprehension

cmdutil: rename template param to export to fntemplate It's actually a template for the filename, not a formatter template.

cmdutil: comprehensively document the interface of export I want to make some improvements here, but in order to make future patches easier to review I want to document the current state of the world.

tests: hide warning from test-xdg.t The test-xdg.t test uses the system hgrc because it unsets HGRCPATH. If the system has an extension enabled that doesn't work with the development version of Mercurial it prints an 'extension failed to load' warning. Let's just hide that by piping stderr to /dev/null.

policy: define C module versions individually This allows us to bump the version for a single module without changing all of them.

repoview: rename '_getdynamicblockers' to 'revealedrevs' (API) Recent mailing list discussion made me realised we could clarify these. We make the function "public" to encourage extensions to wrap it and we use a more explicit name that mirror "hideablerevs".

repoview: move '_getdynamicblock' next to 'hideablerevs' There are the two functions that extensions should use to augment the hidding logic. It seem better to have them together at the top of the file.

python3: allow hgloader to work with lazy loaders Don't clobber the loader returned from find_spec. This brings `hg version` down from 0.27 seconds to 0.17.

init: turn on demandimport for Python 3.6 and above This uses the new demandimport implementation for Python 3 introduced in previous patches. This doesn't yet enhance performance because it isn't integrated with the custom source file loader we use on Python 3. We'll integrate the two in upcoming patches.

demandimport: add python 3 implementation This implementation uses the new importlib finder/loader functionality available in Python 3.5 and up. # no-check-commit

demandimport: move ignore list to __init__.py We're going to use the same ignore list for Python 3.

check: check modules in hgdemandimport A few places only check modules in mercurial and hgext. Add hgdemandimport to the list in those places.

demandimport: move to separate package In Python 3, demand loading is per-package. Keeping demandimport in the mercurial package would disable demand loading for any modules in mercurial.

import-checker: add a way to directly import certain symbols We'll use this for the 'demandimport' symbol in an upcoming patch.

check-code: allow skipping hasattr check in py3-only code hasattr is safe in Python 3, and in an upcoming patch we can't use util.safehasattr.

profiling: allow loading profiling extension before everything else 6d642ecf1a89 makes profiler start early without loading extensions. That makes it impossible for an extension to add customized profilers. This patch adds a special case: if a profiler is not found but an extension with the same name could be loaded, load that extension first, and expect it to have a "profile" contextmanager method. This allows customized profilers and extension setup time is still profiled.

extensions: allow loading a whitelisted subset of extensions This feature will be used by the next patch.

match: catch attempts to create case-insenstive exact matchers Exact matchers are only created internally (as opposed to from user input) based on a set of files that the caller collected before, so they should always match the list exactly (i.e. case-sensitively).