merge with stable

localrepo: reinstate localrepo.parents with a deprecation warning The function was dropped in 3fe8cb40c9c5. This API drop brokes three of my extensions including some critical to my workflow like tortoisehg. Lets mark this API for death and give people time to fix their code.

bookmark: deprecate 'bmstore.write' method This function does not collaborate with the transaction and must disappear. As we have likely a lot of third party users, we make it deprecated to let them some time to upgrade their code. Thanks goes to Laurent Charignon for cleanup the last remains of the 'write' method.

ui: add a 'deprecwarn' helper to issue deprecation warnings As discussed on the list, we are adding an official way to keep old API around for a short time in order to help third party developer to catch up. The deprecated API will issue developer warning (issued by default during test runs) to warn extensions authors that they need to upgrade their code without instantaneously breaking tool chains and normal users. The version is passed as an explicit argument so that developer think about it and a potential future script can automatically check for it. This is not build as a decorator because accessing the 'ui' instance will likely be different each time. The message is also free form because deprecated API are replaced in a variety of ways. I'm not super happy about the final rendering of that message, but this is a developer oriented warning and I would like to move forward.

ui: add a 'stacklevel' argument to 'develwarn' This allows helper functions (like deprecation warning) to prepare a devel warning for higher up in the stack. The argument is named after the one in the Python's 'warning,warn' function.

import-checker: tell which symbol causes "direct symbol import" This would be sometimes useful to understand why import-checker.py complains about it.

import-checker: allow absolute imports of sub modules from local packages Before this patch, import-checker.py didn't know if a name in ImportFrom statement are module or not. Therefore, it complained the following example did "direct symbol import from mercurial". # hgext/foo.py from mercurial import hg This patch reuses the dict of local modules to filter out sub-module names.

manifest: use 't' for tree manifest flag We currently use 'd' to indicate that a manifest entry is a directory. Let's switch to 't', since that's not a valid hex digit and therefore easier to spot in the raw manifest data. This will break any existing repos with tree manifests, but it's still an experimental feature and there are probably only a few test repos in existence with 'd' flags.

test: update the docstring of 'test-devel-warnings.t' extension We are testing more than just locks now.

setup.py: don't rewrite @LIBDIR@ when creating wheels This is necessary to produce wheels that install properly. More details are captured in an in-line comment. After this patch, produced wheels can be installed via `pip install` and appear to "just work," including on Windows.

setup.py: attempt to build and install hg.exe on Windows Currently, packaging Mercurial on Windows will produce a Scripts\hg Python script and a Scripts\hg.bat batch script. The py2exe distribution contains a hg.exe which loads a Python interpretter and invokes the "hg" Python script. Running a exe directly has benefits over batch scripts because batch scripts do things like muck around with command arguments. This patch implements a custom "build_scripts" command which attempts to build hg.exe on Windows. If hg.exe is built, it is marked as a "script" file and installed into the Scripts\ directory on Windows. Since hg.exe is redundant and better than hg.bat, if hg.exe is built, hg.bat is not installed. Since some environments don't support compiling C programs, we treat hg.exe as optional and catch failures building it. This is not ideal. However, I reckon most Windows users will not be installing Mercurial from source: they will get it from the MSI installer or via `pip install Mercurial`, which will download a wheel that has hg.exe in it. So, I don't think this is a big deal.

merge.graft: add option to keep second parent Currently merge.graft re-writes the dirstate so only a single parent is kept. For some cases, like evolving a merge commit, this behaviour is not desired. More specifically, this is needed to fix issue4389.

ui: support declaring path push urls as sub-options Power users often want to apply per-path configuration options. For example, they may want to declare an alternate URL for push operations or declare a revset of revisions to push when `hg push` is used (as opposed to attempting to push all revisions by default). This patch establishes the use of sub-options (config options with ":" in the name) to declare additional behavior for paths. New sub-options are declared by using the new ``@ui.pathsuboption`` decorator. This decorator serves multiple purposes: * Declaring which sub-options are registered * Declaring how a sub-option maps to an attribute on ``path`` instances (this is needed to `hg paths` can render sub-options and values properly) * Validation and normalization of config options to attribute values * Allows extensions to declare new sub-options without monkeypatching * Allows extensions to overwrite built-in behavior for sub-option handling As convenient as the new option registration decorator is, extensions (and even core functionality) may still need an additional hook point to perform finalization of path instances. For example, they may wish to validate that multiple options/attributes aren't conflicting with each other. This hook point could be added later, if needed. To prove this new functionality works, we implement the "pushurl" path sub-option. This option declares the URL that `hg push` should use by default. We require that "pushurl" is an actual URL. This requirement might be controversial and could be dropped if there is opposition. However, objectors should read the complicated code in ui.path.__init__ and commands.push for resolving non-URL values before making a judgement. We also don't allow #fragment in the URLs. I intend to introduce a ":pushrev" (or similar) option to define a revset to control which revisions are pushed when "-r <rev>" isn't passed into `hg push`. This is much more powerful than #fragment and I don't think #fragment is useful enough to continue supporting. The [paths] section of the "config" help page has been updated significantly. `hg paths` has been taught to display path sub-options. The docs mention that "default-push" is now deprecated. However, there are several references to it that need to be cleaned up. A large part of this is converting more consumers to the new paths API. This will happen naturally as more path sub-options are added and more and more components need to access them.

ui: pass ui instance to path.__init__ It will be used in a subsequent patch.

ui: store pushloc as separate attribute The magic @property is going to interfere with the ability to print path sub-options. We only access it in one location and it is trivial to in-line, so do that.

commands: add debugdeltachain command We have debug commands for displaying overall revlog statistics (debugrevlog) and for dumping a revlog index (debugindex). As part of investigating various aspects of revlog behavior and performance, I found it important to have an understanding of how revlog delta chains behave in practice. This patch implements a "debugdeltachain" command. For each revision in a revlog, it dumps information about the delta chain. Which delta chain it is part of, length of the delta chain, distance since base revision, info about base revision, size of the delta chain, etc. The generic formatting facility is used, which means we can templatize output and get machine readable output like JSON. This command has already uncovered some weird history in mozilla-central I didn't know about. So I think it's valuable.

histedit: pick an appropriate base changeset by default (BC) Previously, `hg histedit` required a revision argument specifying which revision to use as the base for the current histedit operation. There was an undocumented and experimental "histedit.defaultrev" option that supported defining a single revision to be used if no argument is passed. Mercurial knows what changesets can be edited. And in most scenarios, people want to edit this history of everything on the current head that is rewritable. Making histedit do this by default and not require an explicit argument or additional configuration is a major usability win and will enable more people to use histedit. This patch changes the behavior of the experimental and undocumented "histedit.defaultrev" config option to select an appropriate base revision by default. Comprehensive tests exercising the edge cases in the new, somewhat complicated default revset have been added. Surprisingly, no tests broke. I guess we were never testing the behavior with no ANCESTOR argument (it used to fail with "abort: histedit requires exactly one ancestor revision"). The new behavior is much more user friendly. The functionality for choosing the default base revision has been moved to destutil.py, where it can easily be modified by extensions.

rebase: remove extra "if" from check of collapsing named branches

rebase: drop redundant functions to keep branch and graft source explicitly All entries in extra dict are propagated by default since 88fde8db5307.

color: drop useless override of ui.popbuffer() Because labels are applied at write() time since 717b75ae5bb0, colorui.popbuffer() is useless and it doesn't update _bufferapplylabels correctly. Removing it should fix the problem.

repoview: bypass changelog method to computed cache key Getting the data necessary for the cache key using the changelog/revlog method adds a significant overhead. Given how simple the underlying implementation is and often this code is ran, it makes sense to violate layering and directly compute the data. Testing `hg log` on Mozilla-central, this reduce the time spent on changelog cache validation by an extra half: before: 12.2s of 69s after: 6.1s of 62s Total speed up from this patch and it's parent is 3x (With stupid python profiler overhead) The global speedup without profiler overhead is still there, Before: 51s After: 39s (-23%)

repoview: stop recomputing cached key in all case As explained in the comment, we were computing the key of the cache value every time because of some obscure MQ test failure. I've dropped that code and ran the test again that failure is gone. I assume some transaction cleanup got rid of it. So we are dropping that code. This provide a significant speedup. Testing `hg log` on Mozilla-central this reduce the time spent on changelog cache validation by a third: before: 19.5s of 80s after: 12.2s of 69s (With stupid python profiler overhead)

commands.debugindexdot: use cmdutil.openrevlog() This pattern is used for all the other debug* commands that operate on revlogs. debugindexdot is an outlier. Make it conform.

commands: unify argument handling for revlog debug commands The same 3 options are used in a few locations and I'm about to add another. Might as well consolidate the pattern.

commit: fix rest syntax of examples This fixes the formatting of help/commit page and silence test-gendoc.t.

ui: optionally ignore sub-options from configitems() For convenience.

ui: add method to return option and all sub-options Apparently ":" has been blessed as a generic separator for options and sub-options. We formalize this by introducing an API for obtaining an option and all its sub-options. This will be used in a subsequent patch for declaring sub-options for [paths].

revlog: don't consider nullrev when choosing delta base In the most complex case, we try using the incoming delta base, then we try both parents, and then we try the previous revlog entry. If none of these result in a good delta, we natually use the null revision as base. However, we sometimes consider the nullrev before we have exhausted our other options. Specifically, when both parents are null, we use the nullrev as delta base if it produces a good delta (according to _isgooddelta()), and we fail to try the previous revlog entry as delta base. After 20a9226bdc8a (addrevision: use general delta when the incoming base delta is bad, 2015-12-01), it can also happen for non-merge commits when the incoming delta is not good. The Firefox repo (from many months back) shrinks a tiny bit with this patch: from 1.855GB to 1.830GB (1.4%). The hg repo itself shrinks even less: by less than 0.1%. There may be repos that get larger instead. This undoes the unexplained test change in 20a9226bdc8a.

revlog: make calls to _isgooddelta() consistent We always want to call _isgooddelta() before accepting a delta. We mostly call the function right after building the delta, but not always. Instead, we have an extra call at the end of the big code block. Let's make it consistent, so we call _isgooddelta() right after builddelta() and exactly once per delta. That also lets us rely on "delta is None" to mean we didn't find a good delta.

revlog: clarify which revision is added to 'tested' when using cached delta The tested delta revisions are added to the 'tested' set. These are the same revisions we pass to builddelta(). However, in one case, we add builddelta(rev)[3] to the set intead of adding 'rev' itself. In that particular case, that element is the same as the function's input revision (because self._generaldelta is true), so the effect is the same. Still, let's just add the function's input revision to avoid confusing future readers.

revlog: remove unused variable 'chainlen'

commit: adjust the quoting in the examples to be Windows friendly We should probably avoid strong quotes around command line args in the examples, since cmd.exe doesn't recognize them, and it will surprise a user who cargo cults them. I don't see a way to make a rule for this, since strong quoting is OK inside command line args, like within revsets.

wireproto: config options to disable bundle1 bundle2 is the new and preferred wire protocol format. For various reasons, server operators may wish to force clients to use it. One reason is performance. If a repository is stored in generaldelta, the server must recompute deltas in order to produce the bundle1 changegroup. This can be extremely expensive. For mozilla-central, bundle generation typically takes a few minutes. However, generating a non-gd bundle from a generaldelta encoded mozilla-central requires over 30 minutes of CPU! If a large repository like mozilla-central were encoded in generaldelta and non-gd clients connected, they could easily flood a server by cloning. This patch gives server operators config knobs to control whether bundle1 is allowed for push and pull operations. The default is to support legacy bundle1 clients, making this patch backwards compatible.

dockerlib: short form for non-unique uid/gid for CentOS 5 compat (issue4977) CentOS 5 does not support '--non-unique', but does support the short '-o'.

exchange: standalone function to determine if bundle2 is requested This will be used in a subsequent patch.

wireproto: add docstring for wirepeer

commit: add some help examples (issue4963) Includes documenting --date now, which resolves the above issue.

changegroup: restate file linknode callback using generator expressions I think this is slightly clearer, and it nicely avoids an extra nested function.

changegroup: clean up file lookup function One case is basically degenerate, so just extract it and make the function clearer.

changegroup: remove one special case from lookupmflinknode In the fastpathlinkrev case, lookupmflinknode was a very complicated way of saying mfs.__getitem__, so let's just get that case out of our way so it's easier to understand what's going on.

changegroup: drop 'if True' that made the previous change clearer

changegroup: avoid iterating the whole manifest The old code gathered the list of all files that changed anywhere in history and then gathered changed file nodes by walking the entirety of each manifest to be sent in order to gather changed file nodes. That's going to be unfortunate for narrowhg, and it's already inefficient for medium-to-large repositories. Timings for bundle --all on my hg repo, tested with hgperf: Before: ! wall 23.442445 comb 23.440000 user 23.250000 sys 0.190000 (best of 3) After: ! wall 20.272187 comb 20.270000 user 20.190000 sys 0.080000 (best of 3)

posix: work around "posix" systems without os.link available (issue4974) Some platforms (see bug, notably a terminal emulator on Android) ship with os.link removed from Python to try and cater to other tools that expect os.link to exist iff hardlinks are supported on that platform. As a workaround for this madness, include a fallback path for when we're on a "posix" platform but lack os.link.

pathutil: use temporary variables instead of complicated wrapping The one-lining did not help readability, we get rid of it.

context: use a the nofsauditor when matching file in history (issue4749) Before this change, asking for file from history (eg: 'hg cat -r 42 foo/bar') could fail because of the current content of the working copy (eg: current "foo" being a symlink). As the working copy state have no influence on the content of the history, we can safely skip these checks. The working copy context class have a different 'match' implementation. That implementation still use the repo.auditor will still catch symlink traversal. I've audited all stuff calling "match" and they all go through a ctx in a sensible way. The most unclear case was diff which still seemed okay. You raised my paranoid level today and I double checked through tests. They behave properly. The odds of someone using the wrong (matching with a changectx for operation that will eventually touch the file system) is non-zero because you are never sure of what people will do. But I dunno if we can fight against that. So I would not commit to "never" for "at this level" and "in the future" if someone write especially bad code. However, as a last defense, the vfs itself is running path auditor in all cases outside of .hg/. So I think anything passing the 'matcher' for buggy reason would growl at the vfs layer.

localrepo: add a second auditor without file system check Auditors keeps a cache of audited paths. Therefore we cannot use the same auditor for working copy and history operation. We create a new one without file system check for this purposes.

pathauditor: add a way to skip file system check We need to be able to skip it when looking at data within the history. Doing them in all cases leads to buggy behavior like issue4749.

pathauditor: move file system specific check in their own function This will make it easy to disable that part when not relevant (eg: auditing filename for operation in history)