wix: remove enum and future packages These were cargo culted from the THG installer code. I'm not sure what needs them in THG land. But the official MSIs certainly do not - at least not as direct dependencies. .. bc:: The Windows MSI installers no longer include the enum and future Python packages. Differential Revision: https://phab.mercurial-scm.org/D6101

wix: remove pywin32 This dependency was for ancient Mercurial versions. We recently removed it from the Inno Setup installers. So let's remove it from the WiX installers as well. .. bc:: The Windows MSI installers no longer include the pywin32 Python package. Differential Revision: https://phab.mercurial-scm.org/D6100

wix: remove sphinx and dependencies Sphinx was cargo culted into our install environment as part of emulating TortoiseHG's behavior. THG seems to install Sphinx in order to generate THG specific documentation. We don't appear to need Sphinx or any of its dependencies in the official WiX installers. So remove it. This shaves ~1MB off the size of the MSI installers. .. bc:: The Windows MSI installers no longer include the Python sphinx package and its various dependencies. Differential Revision: https://phab.mercurial-scm.org/D6099

wix: functionality to automate building WiX installers Like we did for Inno Setup, we want to make it easier to produce WiX installers. This commit does that. We introduce a new hgpackaging.wix module for performing all the high-level tasks required to produce WiX installers. This required miscellaneous enhancements to existing code in hgpackaging, including support for signing binaries. A new build.py script for calling into the module APIs has been created. It behaves very similarly to the Inno Setup build.py script. Unlike Inno Setup, we didn't have code in the repo previously to generate WiX installers. It appears that all existing automation for building WiX installers lives in the https://bitbucket.org/tortoisehg/thg-winbuild repository - most notably in its setup.py file. My strategy for inventing the code in this commit was to step through the code in that repo's setup.py and observe what it was doing. Despite the length of setup.py in that repository, the actual amount of steps required to produce a WiX installer is actually quite low. It consists of a basic py2exe build plus invocations of candle.exe and light.exe to produce the MSI. One rabbit hole that gave me fits was locating the Visual Studio 9 C Runtime merge modules. These merge modules are only present on your system if you have a full Visual Studio 2008 installation. Fortunately, I have a copy of Visual Studio 2008 and was able to install all the required updates. I then uploaded these merge modules to a personal repository on GitHub. That is where the added code references them from. We probably don't need to ship the merge modules. But that is for another day. The installs from the MSIs produced with the new automation differ from the last official MSI in the following ways: * Our HTML manual pages have UNIX line endings instead of Windows. * We ship modules in the mercurial.pure package. It appears the upstream packaging code is not including this package due to omission (they supply an explicit list of packages that has drifted out of sync with our setup.py). * We do not ship various distutils.* modules. This is because virtualenvs have a custom distutils/__init__.py that automagically imports distutils from its original location and py2exe gets confused by this. We don't use distutils in core Mercurial and don't provide a usable python.exe, so this omission should be acceptable. * The version of the enum package is different and we ship an enum.pyc instead of an enum/__init__.py. * The version of the docutils package is different and we ship a different set of files. * The version of Sphinx is drastically newer and we ship a number of files the old version did not. (I'm not sure why we ship Sphinx - I think it is a side-effect of the way the THG code was installing dependencies.) * We ship the idna package (dependent of requests which is a dependency of newer versions of Sphinx). * The version of imagesize is different and we ship an imagesize.pyc instead of an imagesize/__init__.pyc. * The version of the jinja2 package is different and the sets of files differs. * We ship the packaging package, which is a dependency for Sphinx. * The version of the pygments package is different and the sets of files differs. * We ship the requests package, which is a dependency for Sphinx. * We ship the snowballstemmer package, which is a dependency for Sphinx. * We ship the urllib3 package, which is a dependency for requests, which is a dependency for Sphinx. * We ship a newer version of the futures package, which includes a handful of extra modules that match Python 3 module names. # no-check-commit because foo_bar naming Differential Revision: https://phab.mercurial-scm.org/D6097

wix: move contrib/wix to contrib/packaging/wix We're trying to consolidate all our packaging code into contrib/packaging. Let's move the WiX files there. Differential Revision: https://phab.mercurial-scm.org/D6096

wix: remove hg.cmd This file is not referenced anywhere AFAICT. Differential Revision: https://phab.mercurial-scm.org/D6095

setup: include hgext3rd package in py2exe builds This is a core Mercurial package and we should always ship it. Differential Revision: https://phab.mercurial-scm.org/D6094

setup: properly install build_hgextindex for py2exe builds Because the hgbuild class has a private copy of build.sub_commands, modifying build.sub_commands from this code effectively resulted in a no-op. Registering the sub-command on hgbuild actually results in the sub-command running when building Mercurial. Differential Revision: https://phab.mercurial-scm.org/D6093

setup: configure py2exe config via environment variables The Inno Setup and WiX installers ship a different set of packages with py2exe builds. And there are multiple WiX installer variants (e.g. TortoiseHG). Since there are multiple variants of py2exe configs and they can be defined by entities not in our repository, let's provide a mechanism for setup.py to supplement behavior via environment variables. This is slighly less hacky than a setup.cfg file IMO since the caller doesn't need to worry about mutating global state of the source directory. Differential Revision: https://phab.mercurial-scm.org/D6092

packaging: extract py2exe functionality to own module py2exe builds are shared between Inno Setup and WIX. We'll want the logic for performing py2exe builds to be reusable across the code for both installers. This commit extracts the py2exe-specific functionality into its own module. There's definitely room to customize things further. This will be done in future commits, as necessary. (I'm not even sure what customizations WIX will require yet. Presumably a lot.) Differential Revision: https://phab.mercurial-scm.org/D6091

packaging: extract python exe info to own function This is generic functionality. We'll need it for WIX. As part of the port, we expose the full version and return the data as a dict. Differential Revision: https://phab.mercurial-scm.org/D6090

packaging: don't use temporary directory We were no longer doing anything with it after extracting virtualenv and py2exe to the build directory. Differential Revision: https://phab.mercurial-scm.org/D6089

packaging: extract virtualenv and py2exe to build directory The build directory is essentially a cache. We can extract the virtualenv and py2exe package sources to this directory. Differential Revision: https://phab.mercurial-scm.org/D6088

packaging: move Inno Setup core logic into a module Aspects of building the Inno Setup and WIX installers are shared. It will make sense for them to share code. Plus, having code in a reusable library (as opposed to a standalone script) is just a better approach. This commit moves the core logic to build the Inno Setup installer into the hgpackaging package. inno/build.py is now a simple frontend script that calls into a module to do the bulk of the work. As part of this change, I also found a typo in build() where it was referencing "iscc" instead of "iscc_exe." Because "iscc" was in the global scope via the only caller, things just happened to work before. Another benefit of always using functions and not putting global code for __main__ in the same file as library code. Differential Revision: https://phab.mercurial-scm.org/D6087

packaging: move find_vc_runtime_files() into hgpackaging.util In preparation for moving the bulk of the Inno Setup code into hgpackaging. Differential Revision: https://phab.mercurial-scm.org/D6086

packaging: move DOWNLOADS dict to hgpackaging.downloads We'll want to keep state in sync between multiple packaging tools. It makes sense to share a central data structure defining downloads. We also change the function to return the downloads entry so callers don't have to access the global DOWNLOADS in the new location. Differential Revision: https://phab.mercurial-scm.org/D6085

packaging: split downloading code into own module As we will introduce more code to support packaging, it will be useful to have download code in its own module. Differential Revision: https://phab.mercurial-scm.org/D6084

packaging: establish hgpackaging package Previously, contrib/packaging behaved as a root to a package directory and we had a "packagingutil" module. As I work more on packaging code, we'll want to have more code shared between different packaging tools. I think it makes sense to have a single package containing multiple modules than multiple top-level modules. This commit establishes an "hgpackaging" package by moving the existing packagingutil code to it. Differential Revision: https://phab.mercurial-scm.org/D6083

py3: use % instead of .format() on a bytestring Differential Revision: https://phab.mercurial-scm.org/D6112

py3: use r'' for group name arguments to MatchObjects in phabricator.py MatchObject group names are strings, not bytes. Using bytes in py3 leads to an IndexError. Differential Revision: https://phab.mercurial-scm.org/D6111

py3: use %d instead of %s when formatting an int into a byte string Differential Revision: https://phab.mercurial-scm.org/D6110

py3: only pass unicode to json.dumps in writediffproperties Differential Revision: https://phab.mercurial-scm.org/D6109

py3: fix a few "dict keys as str instead of bytes" issues in phabricator.py Differential Revision: https://phab.mercurial-scm.org/D6108

py3: convert URL to str before passing it to request Differential Revision: https://phab.mercurial-scm.org/D6106

py3: convert indexes into bytes when enumerating lists in urlencodenested Otherwise it'll try to insert them them into a %s slot in a b'' later and fail. Differential Revision: https://phab.mercurial-scm.org/D6105

py3: don't try and format a bare dict into a byte string in callconduit Differential Revision: https://phab.mercurial-scm.org/D6104

py3: use fsencode for vcr recording paths and strings for custom_patches args This fixes phabricator.py's vcrcommand under py3 Differential Revision: https://phab.mercurial-scm.org/D6102

phabricator: convert conduit response JSON unicode to bytes inside callconduit Previously the byte conversion was happening piecemeal in callers, and in the case of createdifferentialrevision not at all, leading to UnicodeEncodeErrors when trying to phabsend a commit with a description containing characters not representable in ascii. (issue6040) Remove all the scattered encoding.unitolocal calls and perform it once, inside callconduit, on the entire response dict recursively before returning it, in keeping with the strategy of converting at the earliest opportunity. Convert all keys used on returned object to bytes. Modify the phabsend tests to test this by adding a € to the commit message of alpha. Differential Revision: https://phab.mercurial-scm.org/D6044

transaction: include txnname in the hookargs dictionary There is no reason to not include the txnname alongside the txnid in all case. The python hooks already have them, so aligning the the shell hooks seems it could be useful in the future. (I don't have a strong opinion about this, we can also decide to never align the python and shell hooks and this and I'll drop this patch).

discovery-helper: reflect argument value in the name of the results It is common to create multiple pairs of repositories using different argument values. Recording the argument value in the results names has two main advantages: * It is easy to remember the value used to create a pair, * It get simpler to create multiple pair at the same time from the same source. Previously, running: `./discovery-helper.sh pypy 50 10` would create a `pypy-left` and `pypy-right` repository. Now it will create `pypy-50h-10d-left` and `pypy-50h-10d-right`.

discovery-helper: echo the stripped revsets early Having them printed early make it easy for a user to just grab the generated revset and directly uses them

contrib: move the `discovery-helper.sh` script in`perf-utils` directory The script appeared before the directory. However the directory exists for this kind of script. We move it there.

tests: stabilize test-split.t for Windows It looks like there will be additional problems beyond this trivial fix, but this should make the bot green again.

tests: stabilize test-share.t on Windows PYTHON was not getting mangled for MSYS style paths, and remote was spitting out remote: 'C' is not recognized as an internal or external command, remote: operable program or batch file. (once -q was removed). Additionally, this should fix a failure with py3 because of spaces in the path.

split: add tests which demonstrate the issue5864 Differential Revision: https://phab.mercurial-scm.org/D6057

discovery: clarify why the caching of children is valid Yuya Nishihara pointed out that the code looks wrong without this clarification. (And, unsurprisingly, Yuya is right)

tests: clarify test setup test in test-uncommit.t I assume the "hg uncommit b" is there to prove that the working copy is dirty before we try "hg uncommit --allow-dirty-working-copy b". It seems clearer to put that check just before we run the actual test. Differential Revision: https://phab.mercurial-scm.org/D6078

tests: fix a stale reference to experimental.uncommitondirtywdir These tests no longer test the config option, they test the command line flag. Differential Revision: https://phab.mercurial-scm.org/D6077

discovery: explicitly use `undecided` for the children mapping Recent performance achievements makes the assumption that the `undecided` set is used for sampling. That assumption is always true in practice. We stop pretending that taking anything else would make sense and we directly use the `undecided` set from the object. This provides a more honest API.

discovery: cache the children mapping used during each discovery During discovery, the `undecided` set keep shrinking. Therefore, the map computed for an iteration N will be valid for iteration N+1. Instead of computing the same data over and over we cache it the first time. Our private pathological case speed up from about 7.5 seconds to about 6.3 seconds. (starting from over 70s at the start of the full series)

discovery: move children computation in its own method This clarifies the main logic and starts to pave the way to some caching.

discovery: simplify the building of the children mapping Since we only care about the revisions inside the set we are sampling, we can use simpler code (and probably sightly faster).

discovery: simply walk the undecided revs when building the children mapping The sampling only care about revisions in the undecided set, so building children relationship within this set is sufficient. The set of undecided changesets can be much smaller than the full span from its smallest item to the tip of the repository. This restriction can significantly speed up operations in some cases. For example, on our private pathological case, this speeds things up from about 53 seconds to about 7.5 seconds.

discovery: use a lower level but faster way to retrieve parents We already know that no revision in the undecided set are filtered, so we can skip multiple checks and directly access lower level data. In a private pathological case, this improves the timing from about 70 seconds to about 50 seconds. There are other actions to be taken to improve that case, however this gives an idea of the general overhead.

discovery: avoid computing identical sets of heads twice The very same set of heads is computed in the previous statement, it seems more efficient to just copy that result.

discovery: moved sampling functions inside discovery object In this patch, we transform the sampling functions into methods of the `partialdiscovery` class in the Python case. This will provide multiple benefit. For example we can keep some cache from one sampling to another. In addition this will help the Oxidation work as all graph traversal related logic will be contained in a single object.

discovery: rename `srvheads` to `knownsrvheads` The `srvheads` variable only contains the known set of remove heads. Renaming the variable make it clearer.

verify: small refactoring and documentation in `_verifymanifest` Small changes to make this area of code clearer.

verify: document the `_verifymanifest` method

verify: document `_verifychangelog` We document the method input, output and checks.

verify: rename the `checklog` to `_checkrevlog` The method is for internal use only. In addition we make the method name explicitly contains `revlog` to make it clearer it is checking higher level revlog properties.

verify: document the `checklog` method Let us add details about what the function is expected to do.

revlog: add some documentation to the `checksize` method I had to look at it, so I figured I would leave some documentation for the next person seeking information.

verify: make `checkentry` a private method This method is for internal use only.

verify: document the `checkentry` method This method checks various core propertes of a revision. We document inputs, outputs and the checks performed.

verify: add some inline documentation to the top level `verify` method The goal is to clarify each section goal.

verify: explicitly return 0 if no error are encountered Relying on the fact None is treated as 0 by other logics seems smarter than we should be.

verify: minimal documentation for `verifier.verify` I expect the docstring to grow over time, so lets start small.

verify: make the `exc` method private The method is for internal use only.

verify: document the `exc` method Simple method, get simple documentation.

verify: make `err` a private method The method is for internal use only.

verify: document the `err` method Simple method get simple documentation.

verify: make the `warn` method private The method is for internal use only. Let us make that fact clearer.

verify: document the `warn` method This is the first bit of an effort to document and augment the verify code.

uncommit: add flag --allow-dirty-working-copy This adds a flag `--allow-dirty-working-copy` as an alias for the experimental config option `experimental.uncommitondirtydir`. Differential Revision: https://phab.mercurial-scm.org/D6069

uncommit: don't allow dirty working copy with PATH (issue5977) On a dirty PATH, uncommit was working without even setting the config `experimental.uncommitondirtydir` to `True`. Ideally, it should abort as it does for a dirty dir. This patch makes uncommit to require the config option `experimental.uncommitondirtydir` on a dirty PATH. Original patch to evolve extension authored by Dan Villiom Podlaski Christiansen: https://bitbucket.org/octobus/evolve-devel/pull-requests/8/bug-5977-uncommit-dirtiness/diff Differential Revision: https://phab.mercurial-scm.org/D5940

cleanup: remove two bogus test names from python3 list I suspect one of these was a typo from the start, the other appears to have become a .t test at some point. Differential Revision: https://phab.mercurial-scm.org/D6076

revlog: preserve `_lazydelta` attribute in `revlog.clone` The attribute was introduce in 688fc33e105d, Yuya Nishihara pointed out that this preservation was missing. This changeset fixes the preservation and make sure we set the attribute according the modes.

localrepo: explicit `_lock` arguments in `lock` The argument for `_lock` are non-trivial, having them passed explicitly makes thing clearer in my opinion. This is a Gratuitous change, I expect it will save me (and others) time in the future.

inno: remove w9xpopen.exe w9xpopen.exe is a utility program shipped with Python <3.4 (https://bugs.python.org/issue14470 tracked its removal). The program was used by subprocess to wrap invoked processes on Windows 95 and 98 or when command.com was used in order to work around a redirect bug. The workaround is only used on ancient Windows versions - versions that we shouldn't see in 2019. While Python 2.7's subprocess module still references w9xpopen.exe, not shipping it shouldn't matter unless we're running an ancient version of Windows. Python will raise an exception if w9xpopen.exe can't be found. It's highly unlikely anyone is using current Mercurial releases on these ancient Windows versions. So remove w9xpopen.exe from the Inno installer. .. bc:: The 32-bit Windows Inno installers no longer distribute w9xpopen.exe. This should only impact people running Mercurial on Windows 95, 98, or ME. Differential Revision: https://phab.mercurial-scm.org/D6068

inno: stop shipping pywin32 Ancient versions of Mercurial relied on pywin32 and I suspect that's why we have this dependency. We also ship the "keyring" package, which has a dependency on "pywin32-ctypes" (providing the "win32ctypes" package). This is a stripped down version of pywin32 that doesn't have as many dependencies. Since we don't have a dependency on pywin32 and since pywin32 is a bit annoying to package, let's get rid of it. With this change, py2exe no longers picks up DLL dependencies on various UCRT DLLs (because we no longer have a .pyd file beloning to pywin32 which was pulling them in). So, we were able to remove code in support of the UCRT DLLs. .. bc:: The Windows Inno installers no longer ship the pywin32 package. This package was being bundled for historical reasons. Mercurial stopped using pywin32 several years ago and the disappearance of this package should not have any meaningful impact. Differential Revision: https://phab.mercurial-scm.org/D6067

inno: script to automate building Inno installer The official Inno installer build process is poorly documented. And attempting to reproduce behavior of the installer uploaded to www.mercurial-scm.org has revealed a number of unexpected behaviors. This commit attempts to improve the state of reproducibility of the Inno installer by introducing a Python script to largely automate the building of the installer. The new script (which must be run from an environment with the Visual C++ environment configured) takes care of producing an Inno installer. When run from a fresh Mercurial source checkout with all the proper system dependencies (the VC++ toolchain, Windows 10 SDK, and Inno tools) installed, it "just works." The script takes care of downloading all the Python dependencies in a secure manner and manages the build environment for you. You don't need any additional config files: just launch the script, pointing it at an existing Python and ISCC binary and it takes care of the rest. The produced installer creates a Mercurial installation with a handful of differences from the existing 4.9 installers (produced by someone else): * add_path.exe is missing (this was removed a few changesets ago) * The set of api-ms-win-core-* DLLs is different (I suspect this is due to me using a different UCRT / Windows version). * kernelbase.dll and msasn1.dll are missing. * There are a different set of .pyc files for dulwich, keyring, and pygments due to us using the latest versions of each. * We include Tcl/Tk DLLs and .pyc files (I'm not sure why these are missing from the existing installers). * We include the urllib3 and win32ctypes packages (which are dependencies of dulwich and pywin32, respectively). I'm not sure why these aren't present in the existing installers. * We include a different set of files for the distutils package. I'm not sure why. But it should be harmless. * We include the docutils package (it is getting picked up as a dependency somehow). I think this is fine. * We include a copy of argparse.pyc. I'm not sure why this was missing from existing installers. * We don't have a copy of sqlite3/dump.pyc. I'm not sure why. The SQLite C extension code only imports this module when conn.iterdump() is called. It should be safe to omit. * We include files in the email.test and test packages. The set of files is small and their presence should be harmless. The new script and support code is written in Python 3 because it is brand new and independent code and I don't believe new Python projects should be using Python 2 in 2019 if they have a choice about it. The readme.txt file has been renamed to readme.rst and overhauled to reflect the existence of build.py. Differential Revision: https://phab.mercurial-scm.org/D6066

setup: exclude some internal UCRT files When attempting to build the Inno installer locally, I was getting several file not found errors when py2exe was crawling DLL dependencies. The missing DLLs appear to be "internal" DLLs used by the Universal C Runtime (UCRT). In many cases, the missing DLLs don't appear to exist on my system at all! Some of the DLLs have version numbers that appear to be N+1 of what the existing version number is. Maybe the "public" UCRT DLLs are probing for version N+1 at load time and py2exe is picking these up? Who knows. This commit adds the non-public UCRT DLLs as found by py2exe on my system to the excluded DLLs set. After this change, I'm able to produce an Inno installer with an appropriate set of DLLs. Differential Revision: https://phab.mercurial-scm.org/D6065

setup: include additional packages in py2exe distribution I'm attempting to reproduce the Inno installers on my local machine. As part of auditing differences between installer output, I noticed that the existing Inno installers include various 3rd party packages. There is no mention of this in the build instructions nor on the wiki. This must be something that is done by the installer producer. This commit teaches setup.py to include these 3rd party packages in py2exe's library. After this change, I am able to produce Inno installers that have a nearly identical set of Python modules. It's worth noting that pywin32 is included even though it probably shouldn't be. But including it is necessary in order to achieve parity with existing Inno installers. Differential Revision: https://phab.mercurial-scm.org/D6064

setup: define build_doc command Currently, various processes for packaging Mercurial state to manually invoke `make -C doc` in order to generate the documentation. This Makefile merely invokes `gendoc.py` and `runrst` to produce man pages and HTML pages. Not all environments may have the ability to easily run Makefiles. Windows is notably in this set. This commit ports the man page and HTML generation logic from doc/Makefile to setup.py. We introduce a new build_doc command which generates documentation by calling gendoc.py and runrst. The documentation can now be built via pure Python by running `python setup.py build_doc`. We don't implement dependency tracking because IMO it is more effort than it is worth. We could potentially remove the duplicated functionality in doc/Makefile. But I'm not sure what all is depending on it. So I plan to keep it around. # no-check-commit because forced foo_bar function names Differential Revision: https://phab.mercurial-scm.org/D6063

inno: remove references to pywin32 According to the commit message for 0c35bb01a1195, pywin32 was removed in Mercurial 1.8! Differential Revision: https://phab.mercurial-scm.org/D6062

inno: remove cacerts.pem from documentation The inno distribution no longer includes cacert.pem as of ca1ad8ef38be22 (April 2015). The docs were never updated. Differential Revision: https://phab.mercurial-scm.org/D6061

inno: replace add_path.exe with a Pascal script While attempting to build the Inno installer, I was unable to find a copy of add_path.exe from the source site previously listed in the docs. Some quick Googling revealed that achieving this functionality in native Pascal scripts seems to be preferred these days. This commit vendors "Modify Path" (fetched from https://www.legroom.net/software/modpath) and plugs it into our Inno config file per its instructions. The existing Inno installer appears to only modify PATH for the current user (as opposed to at the system level). I've maintained this behavior with Modify Path. Although it would be trivial to change or add checkboxes to control the behavior. I'll leave this as a follow-up. Differential Revision: https://phab.mercurial-scm.org/D6060

inno: move inno installer files to contrib/packaging/inno Let's isolate the inno installer files to their own directory so the separation between things is clearer. This required adjusting a few relative paths and references to the old directory. Differential Revision: https://phab.mercurial-scm.org/D6059

py3: add a new passing test found by buildbot Differential Revision: https://phab.mercurial-scm.org/D6073

tests: make test-status.t compatible with test-check-module-imports.t Otherwise the later fails on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6072

py3: convert KEY_PRESSED value to bytes in crecord.py This was a str before. Differential Revision: https://phab.mercurial-scm.org/D6071

hghave: skip emacs tests on 24.3 and earlier Turns out with-eval-after-load is new in 24.4. Differential Revision: https://phab.mercurial-scm.org/D6074

py3: use bytes instead of str in isinstance() We use bytes everywhere. Differential Revision: https://phab.mercurial-scm.org/D6070

server: allow customizing the default repo filter hgweb has the (undocument) configuration option web.view that allows restricting visible revisions to immutable. This is useful for serving the same storage as publishing and non-publishing repo. Add the new server.view option to serve the same purpose by changing the default behavior of `getdispatchrepo`. Drop the hard-coded 'served' filter in the batch handler of v1 of the wire proto, this is a left-over from the days before `getdispatchrepo` existed. Differential Revision: https://phab.mercurial-scm.org/D5946

py3: convert filtername to str if it's None I have not called pycompat.bytestr() and rather converted the value there because I am starting to get concerned about the function call overhead of all this bytes to str or vice versa convert functions. Differential Revision: https://phab.mercurial-scm.org/D6042

walkfilerevs: rename filerevgen() to filerevs() since it's not a generator Differential Revision: https://phab.mercurial-scm.org/D6053

global: use raw string for setlocale() argument Otherwise Python 2 will coerce a unicode to str, which fails on HGUNICODEPEDANTRY=1. Differential Revision: https://phab.mercurial-scm.org/D6052

encoding: use raw strings for encoding arguments This prevents the internals of Python from coercing a unicode to str on Python 2 and makes tests run with HGUNICODEPEDANTRY=1 a lot happier. Differential Revision: https://phab.mercurial-scm.org/D6051

revsetlang: use sysbytes() instead of blind encode() Otherwise we will call str.encode() on Python 2, which is wrong. sysbytes() does encode('utf-8') on Python 3. But the source is guaranteed ASCII, so it shouldn't matter. With this change, `hg` now runs with `HGUNICODEPEDANTRY=1` set. However, several tests are failing. Differential Revision: https://phab.mercurial-scm.org/D6050

global: use raw strings for namedtuple() Otherwise Python 2 will attempt to coerce unicode to str, which we don't want. Differential Revision: https://phab.mercurial-scm.org/D6049

attr: don't attempt to .encode() a str on Python 2 Otherwise it coerces automatically. Differential Revision: https://phab.mercurial-scm.org/D6048

procutil: use a raw string for module name Otherwise Python 2 will coerce unicode to str. Differential Revision: https://phab.mercurial-scm.org/D6047

global: use raw strings for __slots__ Otherwise Python 2 will coerce unicode to str at module load time. We don't like automatic string coercions. Differential Revision: https://phab.mercurial-scm.org/D6046

absorb: run cleanupnodes() within transaction also when not using obsmarkers scmutil.cleanupnodes() schedules stripping to be done after the current transaction, so we can safely run it within the transaction. This also means that the phase will be updated within the transaction, which I believe means that the initial (possibly incorrect) phase will not visible. Differential Revision: https://phab.mercurial-scm.org/D6037

py3: pass a str to getpass.getpass() Otherwise I see `password: b''` on the prompt. # skip-blame because just prefix Differential Revision: https://phab.mercurial-scm.org/D6041

py3: port things from chgserver.py I have installed mercurial on my personal laptop using Python 3.7. I also have hg aliased to chg. Before this commit, `hg version` didn't work. After this patch, things are better and now chg works. Differential Revision: https://phab.mercurial-scm.org/D6040

py3: make sure return value of posix.groupname() is bytes Differential Revision: https://phab.mercurial-scm.org/D6039

contrib: make check-code.py check code fragments embedded in test scripts

contrib: add line offset information to file check function of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. This information will be useful to show correct line number in an actual file for errors detected in code fragments embedded in *.t test scripts.

contrib: change return value of file checking function of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. After this patch, caller of _checkfiledata() can count number of errors in each code fragments of an actual file. This will be useful to share --per-file limitation by all embedded code fragments.

contrib: factor out actual error check for file data of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. Newly added _checkfiledata() will be useful to apply checks on code fragments embedded in *.t test scripts.

contrib: refactor preparation logic for patterns of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. Before this patch, preparation logic in _preparepats() of check-code.py is not reusable. It can handle only module global list "checks". This patch splits preparation logic into small internal functions, and add the loop to invoke them, in order to increase reusability of the logic. "c[-2]" is equivalent to "c[3]" for "checks". This patch uses the former, because it will be more reusable for subsequent patch than the latter.

contrib: split pypats list in check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. Ideally, all patterns in "pypats" before this patch should be applied on not only normal *.py files but also code fragments embedded in *.t test scripts. But fixing test scripts for some patterns requires many changes, and has less profit than effort. Therefore, this patch splits pypats list into two below: - commonpypats, which are applied on all (= including code fragments embedded in *.t test scripts) *.py files - pypats, which are applied only on normal *.py

tests: remove useless line wrapping in a code fragment embedded in test script This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. This revision avoids error of check-code.py below. Use () to wrap long lines in Python, not \

storage: introduce a `revlog.reuse-external-delta` config This option goes a bit further and provides a way to get the same behavior as the `re-delta-all` optimisation from `hg debugupgraderepo`. The effect of the option is a bit hard to test as we do not have multiple diff algorithm at hand. However, we at least make sure the code path run.

storage: introduce a `revlog.reuse-external-delta-parent` config As pointed in c59987ab96b4, we had no simple way to get a client to not blindly reuse the delta parent from a bundle. Instead one had to rely on a side effect of the deprecated `format.generaldelta` configuration. We introduce an explicit `revlog.reuse-external-delta-parent` configuration option (default to True) to control this behavior. If the option is not set, `format.generaldelta` still control this behavior. To test the new option, we convert a couple of place where `generaldelta` have been used for its side effects.

merge with stable

py3: convert return values of inspect.getabsfile() to bytes It's weird that python docs for inspect does not mention getabsfile(). Differential Revision: https://phab.mercurial-scm.org/D6033

py3: add two new passing tests found by buildbot Differential Revision: https://phab.mercurial-scm.org/D6032

py3: make contrib/debugshell.py work with Python 3 I changed default mercurial installation of my personal laptop to one installed with python 3.7. debugshell is one of the extension which I have enabled and it was failing. This patch makes debugshell works with Python 3. I found that chg does not work with python 3. Differential Revision: https://phab.mercurial-scm.org/D6031

absorb: let scmutil.cleanupnodes() take care of setting phase Differential Revision: https://phab.mercurial-scm.org/D6036

absorb: use scmutil.cleanupnodes() also when obsmarkers are disabled A side-effect of this is the new warning in the test case, but the warning is valid, so I don't think that's a problem. Differential Revision: https://phab.mercurial-scm.org/D6035

absorb: use scmutil.cleanupnodes() so operation gets set The operation is useful for e.g. `hg obslog` output. Differential Revision: https://phab.mercurial-scm.org/D6034

test: follow-up on 1c4d6ab2ecb8, stabilize test-remotefilelog-bgprefetch.t In 1c4d6ab2ecb8, I've added some sleeps after about half of the calls to `hg debugwaitonrepack` but it seems the other calls are also flaky as catched by our CI: https://ci.octobus.net/job/MercurialPy2/299/console. Add sleeps after all calls in order to be safe.

tests: add more wildcards to test-extdiff.t This test has been flakey recently on Windows, but it appears to be the same scenario as 88d4477ac4f6.

branchmap: prevent reading the file twice through different iterators Otherwise, test-static-http.t breaks. Differential Revision: https://phab.mercurial-scm.org/D6028

remotefilelog: fix format str, blobsize isn't always a #, can be "(missing)" Differential Revision: https://phab.mercurial-scm.org/D6025

remotefilelog: do not specify an explicit version for repack Differential Revision: https://phab.mercurial-scm.org/D6024

test: generate the `test-sparse-revlog` artifact when slow-test is allowed The `test-sparse-revlog.t` logic requires a large bundle to be generated. This bundle can be reused from one run to the next but its initial generation is slow. With this patch, if the bundle is missing and slow tests are permitted, the bundle will be generated during the test run. This should ensure that CI run this test.

runtest: also update slow test timeout during `#if` clauses For a `#if slow` test to be useful, we need the test timeout to be increased. Without this, the slow section would likely be killed before it finish.

runtest: move slow timeout process earlier in the `_hghave` method Before this changesets, early returns might prevent this logic to apply. It seem safer to run it sooner.

runtest: extract the logic that update timeout for slow tests We want to use the "slow" test logic not only for "#require" clauses but also in "#if" clauses. For this to be useful we need the logic to bump the timeout in at least two spots. The first step it to factor it out.

tests: increase timeout for slow test Test case `test-sparse-revlog.t` need some artifact (a bundle) build before it can run. The artifact is expensive to build, but can be reused from one run to the other. We are about to update that test to make the artifact building automatic if `--allow-slow-tests` is passed. However, we need a bump the timeout a bit to make sure the artifact building as time to finish. We could maybe teach run-tests.py how to directly handle such artifacts. However since there is only one of them for now, this seems premature. There are also some room to speed up the bundle creation for test-sparse-revlog.t

merge with stable

rewriting: add an option for rewrite commands to use the archived phase Using the archived phase for cleanup provide the same effect than stripping, but in a faster, append-only way. We keep the feature experimental for now until it gets a bit more testing.

strip: introduce a soft strip option This is the first user-accessible way to use the archived phase introduced in 4.8. This implements a feature discussed during the Stockholm sprint, using the archived phase for hiding changesets. The archived phase behaves exactly as stripping: changesets are no longer visible, but pulling/unbundling them will make then reappear. The only notable difference is that unlike hard stripping, soft stripping does not affect obsmarkers. The next changeset will make use of the archived phase for history rewriting command. However, having a way to manually trigger the feature first seems a necessary step before exposing users to this phase; there is a way to un-archived changesets (unbundling), so there must be a way to archive them again. Adding a flag to strip is a good way to provide access to the feature without taking a too big risk on the final UI we want. The flag is experimental so it won't be exposed by default. Using the archived phase is faster and less traumatic for the repository than actually stripping changesets.

branchcache: move loading of branch names and nodes into it's own function This will help me in implementing lazy loading of the branchcache in upcoming patches. Differential Revision: https://phab.mercurial-scm.org/D6023

rebase: add missing dashes in help text

test: stabilize bundle generation for test-sparse-revlog.t To reduce the instability in the bundle binary content, we force it to contains delta against p1 in all cases. In the previous changeset, we already stabilized the processing of the bundle. So we don't see any output change in the test itself.

test: don't trust delta bases from the bundle in test-sparse-revlog.t The point of the test is to check the strategy sparse-revlog uses to pick delta base. If we trust the bases used in the bundle, we no longer fully test this logic. In order to force this computation we have to use the side effect of a legacy format configuration `format.generaldelta`. The lack of a more official way to do so will be fixed in a later changeset.

test: update test-sparse-revlog.t output This test is skipped unless a large artefact is pre-build. It seems like nobody ran it in a while. Changeset 3764330f76a6 changed the expected output but nobody noticed. This changeset focus on the first and simpler step: putting the expected output back to what one would get by running this test. However this test changes highlight a couple of deeper issues: 1) Even if the revision content did not changed, a change in the delta contained in the bundle affected the delta stored in the final revlog, changing the test result. Since we are testing the delta computation strategy with sparse, we should not blindly reuse the delta-base from the bundled delta. 2) A change in the format of the repository used to generate the bundle changed the delta stored in the bundle. We should get a more stable output to avoid future instabilities of this test. 3) The test is it not run by CI or developer. We'll try to address all this issues in the coming changesets.

bundle2: don't send "shared" requirement when cloning from a share Differential Revision: https://phab.mercurial-scm.org/D5985

tests: demonstrate failure when cloning from a share via bundle2 Differential Revision: https://phab.mercurial-scm.org/D5984

mq: disable qrecord during histedit (issue5981) qrecord during histedit may lead to deadlock-like situations. qpop will throw an error on called during histedit even after qrecord-ing those changes. This patch makes qrecord to abort on histedit. Differential Revision: https://phab.mercurial-scm.org/D5997

committablectx: move status-related methods closer together The modified()/added()/removed()/deleted() clearly belong very close to status(). I separated them in committablectx by the new p[12]copies() methods. This brings the close again. Sorry about the churn. Differential Revision: https://phab.mercurial-scm.org/D5996

tests: add test for hg-test-mode emacs code This is just coverage for the compilation-mode support, but that was enough of a hassle that I wanted to have it covered somehow. Test methodology is _extremely_ cargo-culted from the test for compilation-mode in emacs, so I still have no idea what I'm doing. Differential Revision: https://phab.mercurial-scm.org/D6003

hghave: add check for GNU emacs Differential Revision: https://phab.mercurial-scm.org/D6002

contrib: also linkify tracebacks in compilation output when using hg-test-mode Differential Revision: https://phab.mercurial-scm.org/D6001

contrib: add compilation-mode linking for our test output These regular expressions will cause compilation-mode buffers in emacs to link to source when there are check-code errors in the output of a .t test. In the true tradition of this file, I also have no idea what I'm doing. Differential Revision: https://phab.mercurial-scm.org/D6000

diff: make sure we output stat even when --git is not passed (issue4037) (BC) Before this patch, `hg diff --stat` will give an empty output. It will not show the stat information. I debugged and found that the underlying code does not return the diff header and due to that, other code paths fails to parse that as a diff. I looked into why we don't return diff headers in quiet mode and found the behavior is from 8f8bb77d560e70bcc95577e4dfa877df18d876ab which does not have any mention about why it is done. We also show the diff headers in git, so I think it's fine showing diff header in normal diff in quiet mode. Differential Revision: https://phab.mercurial-scm.org/D6007

tests: add test to demonstrate issue4037 `hg diff --stat -q --config diff.git=0` does not output anything whereas it should print the stat. This is a quiet old bug dating to 2013 and looking at code I think it exists it since 2005 or when --stat was introduced. The next patch will fix the bug. Differential Revision: https://phab.mercurial-scm.org/D6006

grep: reuse getrenamedfn() from scmutil My motivation is to reduce uses of filectx.renamed(). Reusing scmutil.getrenamedfn() also means that we get some caching of copy information per file and revision. I don't think that matters for `hg grep` (I doubt it speeds up significantly, and I doubt it wastes significant memory), but I'm not sure. Differential Revision: https://phab.mercurial-scm.org/D6022

templatekw: move getrenamedfn() to scmutil (API) The function is already used by `hg log` (for following renames, not for templates), so it seems it does not belong in templatekw. Differential Revision: https://phab.mercurial-scm.org/D6021

mq: get copy source from context object instead of from filelog This removes dependence on filelog.renamed(). Differential Revision: https://phab.mercurial-scm.org/D6020

mq: slightly modernize by using context object Context objects have existed since mid-2006. Differential Revision: https://phab.mercurial-scm.org/D6019

absorb: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6018

fix: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6017

memfilefromctx: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6016

largefiles: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6015

commit: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6014

tests: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6013

templatekw: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6012

convert: migrate to new method for getting copy source Differential Revision: https://phab.mercurial-scm.org/D6011

context: move equivalent renamed() implementations to superclass Now that renamed() in workingfilectx and in overlayfilectx are written in terms of copysource(), they are functionally identical and can be reused. Differential Revision: https://phab.mercurial-scm.org/D6010

context: rewrite renamed() in terms of new copysource() where appropriate It's only the filectx for committed files that have the copy source's nodeid easily available; workingfilectx's and overlayfilectx's renamed() simply find the nodeid by looking in the changeset's p1's manifest. Differential Revision: https://phab.mercurial-scm.org/D6009

context: add specialized way of getting copy source file only I'm working on support for storing copy metadata in the changeset instead of the filelog. I don't intend to include the file nodeid there, but most callers don't need that anyway. This patch introduces a method similar to ctx.renamed(), but the new method returns only the source filename, not the nodeid. Subsequent patches will move callers over to this new method. Differential Revision: https://phab.mercurial-scm.org/D6008

changegroup: move non-pruning of non-ellipsis manifests to _prunemanifests() Google has an extension that overrides _prunemanifests() and removes nodes that we fetch using another mechanism. That broke when _prunemanifests() no longer got called. It works again if we move the check for "not self._ellipses" inside _prunemanifests(). Differential Revision: https://phab.mercurial-scm.org/D6004

context: use includematcher when checking dir/file conflicts This is for performance; patternmatcher can't easily interpret its results to make visitchildrenset be the "optimal" set of paths to inspect, but includematcher can. Since there aren't any special patterns being used here, I believe that the two matchers are equivalent. Differential Revision: https://phab.mercurial-scm.org/D5999

tests: change the paths slightly in test-rebase-inmemory.t c/c was a little difficult to understand (and verify that it was the *correct* 'c/' that was being talked about), and it's useful to have multiple directories to prove that we are able to detect this even if there's no files (just a subdirectory) in the immediate directory that's conflicting. Differential Revision: https://phab.mercurial-scm.org/D5998

rust-cpython: using rustext.dagop.headrevs in revlog As with the previous oxidation series, revlog plays the role of the factory, either using its parents function, or passing the index. We include below results of revsetbenchmarks.py taken on the PyPy repository on those of contrib/all-revsets.tx that involve `heads()`. In most of the cases, this seems to be either neutral or an improvement. In the cases where it's actually a bit slower, we suspect that differences in `heads()` performance is actually burried in variance on the incoming revset (probably several orders of magnitude slower). The precheck for filtered revisions of parent changeset has a significative performance benefit, too. Result by revset ================ Revision: 0) 0c7b353ce100; rust-cpython: binding for headrevs() 1) Parent of this changeset; changelog: prefilter in headrevs() 2) This changeset revset #0: heads(commonancestors(last(head(), 2))) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.001379 0.001361 0.001381 0.001410 0.001393 0.001372 0.001414 0.001387 0.001411 0.001429 0.001415 1) 0.001351 0.001373 0.001383 0.001392 0.001401 0.001385 0.001405 0.001406 0.001385 0.001424 0.001399 2) 0.001365 0.001362 0.001375 0.001393 0.001370 0.001365 0.001413 0.001386 0.001377 0.001415 0.001411 revset #1: heads(commonancestors(head())) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.047578 0.048578 0.047764 0.048065 0.047289 0.047305 0.047729 0.047370 0.047611 0.048005 0.047755 1) 0.048072 0.047471 0.048351 0.048193 0.048380 0.047968 0.047683 0.047355 0.048587 0.047044 0.048299 2) 0.047124 0.046699 0.046896 0.047250 0.046920 0.047379 0.046855 0.047753 0.047289 0.047219 0.046991 revset #2: heads(all()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.037654 0.037814 0.037149 0.037457 0.037609 0.037053 0.036825 0.037054 0.037739 0.036816 0.037604 1) 0.021845 58% 0.022172 58% 0.022148 59% 0.022059 58% 0.022261 59% 0.022246 60% 0.021691 58% 0.021967 59% 0.022156 58% 0.021820 59% 0.023141 61% 2) 0.014459 66% 0.014470 65% 0.014420 65% 0.014413 65% 0.014421 64% 0.014492 65% 0.014512 66% 0.014579 66% 0.014500 65% 0.014501 66% 0.014537 62% revset #3: heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.003696 0.003681 0.003719 0.003746 0.003725 0.003750 0.003692 0.003747 0.003712 0.003754 0.003763 1) 0.002131 57% 0.002142 58% 0.002147 57% 0.002203 58% 0.002143 57% 0.002208 58% 0.002158 58% 0.002182 58% 0.002169 58% 0.002209 58% 0.002201 58% 2) 0.001490 69% 0.001524 71% 0.001515 70% 0.001528 69% 0.001531 71% 0.001520 68% 0.001549 71% 0.001542 70% 0.001560 71% 0.001559 70% 0.001544 70% revset #4: (-5000:-1000) and heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.003832 0.003816 0.003747 0.003814 0.003749 0.003894 0.003784 0.003796 0.003915 0.003829 0.003795 1) 0.002282 59% 0.002208 57% 0.002220 59% 0.002240 58% 0.002210 58% 0.002276 58% 0.002250 59% 0.002250 59% 0.002311 59% 0.002230 58% 0.002241 59% 2) 0.001658 72% 0.001662 75% 0.001568 70% 0.001599 71% 0.001588 71% 0.001696 74% 0.001615 71% 0.001593 70% 0.001710 73% 0.001622 72% 0.001616 72% revset #5: heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.826449 7.563260 7.581034 7.688493 7.634001 7.777860 7.768228 8.026097 7.767422 7.565254 7.938643 1) 7.750766 7.562555 7.660426 7.574089 7.492220 7.438582 7.562015 7.530635 93% 7.636343 7.636712 7.645113 2) 7.617941 7.519601 7.584922 7.507653 7.547440 7.524436 7.575291 7.883991 7.792142 7.709622 7.868595 revset #6: heads(matching(tip, "author")) and -10000:-1 plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.744489 7.728684 7.734065 7.928513 7.875949 7.883727 7.815492 7.791335 7.784793 7.761218 7.815731 1) 7.808956 7.480446 7.618759 7.920270 7.676343 7.803613 7.770210 7.713100 7.584420 7.767335 7.825140 2) 7.519987 7.938748 106% 7.805328 7.694162 7.750129 7.714229 7.603825 7.580734 7.555291 7.524207 7.504580 revset #7: (-10000:-1) and heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.909321 7.694357 7.666021 7.538686 7.771821 7.876217 7.852103 7.812727 7.545919 7.788860 7.764585 1) 7.749232 7.683715 7.968393 7.895257 7.764160 8.314884 105% 7.921697 7.882613 7.867209 7.684707 7.544501 2) 7.824903 7.784605 7.727846 7.566613 7.581994 7.539205 90% 7.555316 7.535572 7.581786 7.901795 7.662832

changelog: prefilter in headrevs() In case where headrevs() is called on some revisions, we perform the check that aren't filtered in advance, and switch revlog to use its unchecked form. This allows to work with alternative implementations that don't have knowledge of the filtering system, such as the Rust one.

tests: fixed test too dependent on actual exception wording On one of the machines I use to run the tests prior to submission, the default Python is 2.7.5, with the following wording: must be encoded string without NULL bytes, not str This third form (and possible future ones) are motivation to use a wider catching regexp.

contrib: enforce wrapping too-long lines with () instead of \ This is the style I prefer, and an anecdotal exploration of styles recommended in style guides etc. Further, to quote pep8: > The preferred way of wrapping long lines is by using Python's implied > line continuation inside parentheses, brackets and braces. Long lines > can be broken over multiple lines by wrapping expressions in > parentheses. These should be used in preference to using a backslash > for line continuation. So I think this is a virtuous change. Differential Revision: https://phab.mercurial-scm.org/D5995

cleanup: prefer nested context managers to \-continuations I'd prefer Python accept a tuple of context managers, but alas it isn't meant to be. This will have to suffice. Differential Revision: https://phab.mercurial-scm.org/D5994

cleanup: use () to wrap long lines instead of \ This is a little less brittle, and often helps indentation. In a surprising number of cases the entire cleanup was deleting the \, as the expression was *already* parenthesized in a workable way. Differential Revision: https://phab.mercurial-scm.org/D5993

tests: use () instead of \ to wrap lines This should auto-format more consistently, and is slightly more typical Python. Differential Revision: https://phab.mercurial-scm.org/D5992

merge with stable

copies: do copy tracing based on ctx.p[12]copies() if configured This adds an option to do copy tracing in a changeset-optimized way. If the metadata is stored in filelogs, this is obviously going to be suboptimal. The point is that it provides a way of transitioning to changeset-stored metadata. Some of the tests behave a little differently, but they all seem resonable to me. The config option may very well be renamed later when it's clearer what options we want and how they will behave. When the test suite is run with --extra-config-opt to use the new copy tracing, all tests pass, besides test-copies.t (which fails in the same way as you can see in this patch). `hg debugpathcopies 4.0 4.8` reports 82 copies. With this option enabled, the only difference is this: -mercurial/pure/bdiff.py -> mercurial/cffi/bdiff.py +setup_bdiff_cffi.py -> mercurial/cffi/bdiff.py I believe that happened because it was renamed in different ways on different sides of a merge and the new algorithm arbitrarily prefers copies that happened on p1. The runtime is about 0.85 seconds with the old copy tracing and 5.7 seconds with the new copy tracing. That's kind of slow, but actually better than I had expected. Differential Revision: https://phab.mercurial-scm.org/D5991

context: introduce p[12]copies() methods and debugp[12]copies commands As mentioned earlier, I'm working on support for storing copy metadata in the changeset instead of in the filelog. In order to transition a repo from storing metadata in filelogs to storing it in the changeset, I'm going to provide a config option for reading the metadata from the changeset, but falling back to getting it from the filelog if it's not in the changeset. In this compatiblity mode, the changeset-optmized algorithms will be used. We will then need to convert the filelog copy metadata to look like that provided by changeset copy metadata. This patch introduces methods that do just that. By having these methods here, we can start writing changeset-optimized algorithms that should work already before we add any support for storing the metadata in the changesets. This commit also includes new debugp[12]copies commands and exercises them in test-copies.t. Differential Revision: https://phab.mercurial-scm.org/D5990

copies: filter out copies when target is not in destination manifest When chaining a series of commits that copied a file with a series that removed the destination file, we would still include the copy in the result. Similar to the previous patch, I have checked that `hg status --copies` is not affected by this bug, but I wouldn't be surprised if some commands are. Differential Revision: https://phab.mercurial-scm.org/D5989

copies: make _backwardrenames() filter out copies by destination As shown by the test case, _backwardrenames() doesn't filter by the matcher. It doesn't show up in `hg status --copies` because that only prints files changed between the two commits. I wouldn't be surprised if some other command that replies on pathcopies() was broken before this patch, but I haven't bothered to check other commands. Differential Revision: https://phab.mercurial-scm.org/D5988

copies: respect narrowmatcher in "parent -> working dir" case I don't know when this case happens and we don't seem to have tests for it, but let's fix it anyway since I happened to notice it. Differential Revision: https://phab.mercurial-scm.org/D5987

tests: add tests of pathcopies() I'm working on support for storing copy metadata in the changeset instead of in the filelog. When storing it in the changeset, it will obviously be efficient to get the copy metadata for all files in a single changeset, but it will be more expensive to get the copy metadata all revisions of a single file. Some algorithms will then need to be optimized differently. The first method I'm going to rewrite is pathcopies(). This commit adds many tests for pathcopies(), so we can run the tests with both old and new versions of the code, as well as with metadata stored in filelog or in changeset (later). They use the debugpathcopies command I recently added (with no tests when it was added). They show a few bugs and few cases of slightly weird behavior. I'll fix the bugs in the next few commits. Differential Revision: https://phab.mercurial-scm.org/D5986

uncommit: add config option to keep commit by default We have a Google-internal extension that keeps track of "review units" (like Phabricator reviews, or Gerrit's Change-Id). This information is stored outside of the commit. It is updated with rewrites. Every now and then we get reports from users who are confused because `hg uncommit` lost track of their review. Keeping the empty commit by default would reduce this confusion. It may also cause confusion about the empty commit. This patch adds a config option that lets us easily test both behaviors on our users. Differential Revision: https://phab.mercurial-scm.org/D5970

debugpathcopies: sort output so it's deterministic Differential Revision: https://phab.mercurial-scm.org/D5983

subrepo: use root-repo-relative path from `hg files` with ui.relative-paths=no The fix is to pass in a "subuipathfn" as we do everywhere else. Differential Revision: https://phab.mercurial-scm.org/D5978

subrepo: demonstrate broken `hg files` with ui.relative-paths=no Differential Revision: https://phab.mercurial-scm.org/D5977

py3: whitelist 5 new passing tests caught by buildbot Thanks to indygreg and durin42 recent patches. Differential Revision: https://phab.mercurial-scm.org/D5979

uncommit: inform user if the commit is empty after uncommit Differential Revision: https://phab.mercurial-scm.org/D5969

test: stabilize test-run-tests.t output We have reached a point where the duration in JSON reports of `test-run-tests.t` were greater or equal than 10 seconds, which doesn't match anymore the regex. For example here: https://ci.octobus.net/blue/organizations/jenkins/MercurialPy2/detail/MercurialPy2/276/pipeline ``` "diff": "", ? (re) - "end": "\s*[\d\.]{4,5}", ? (re) + "end": "10.040", "result": "skip", ? (re) ``` Instead of accepting more characters, I changed the regex to accept any number of digits before the `.` then 3 or 4 digits after. This way the regex is more precise (only one `.` is authorized and we can ensure that the precision doesn't change). Differential Revision: https://phab.mercurial-scm.org/D5966

test: stabilize test-remotefilelog-bgprefetch.t flaky output When running the test suite with multiple processes, we often get flaky outputs, like here: https://ci.octobus.net/job/MercurialPy2/274/console ``` - $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx - $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack - $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx - $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack + $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0 + $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/ef95c5376f34698742fe34f315fd82136f8f68c0 + $TESTTMP/hgcache/master/39/5df8f7c51f007019cb30201c49e884b46b92fa/69a1b67522704ec122181c0890bd16e9d3e7516a + $TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca + $TESTTMP/hgcache/master/af/f024fe4ab0fece4091de044c58c9ae4233383a/bb6ccd5dceaa5e9dc220e0dad65e051b94f69a2c ``` Add some sleeps after `debugwaitonrepack` calls as it seems it needs some extra time to cleanup. Differential Revision: https://phab.mercurial-scm.org/D5960

merge with stable

color: change color of grep.rev label (BC) GNU grep uses green for line numbers as we do, but I sometimes get a bit confused when I do `hg grep --diff -n` and get both line numbers and revisions and don't know which one is which. A different colour can help.

color: give colours to the grep.inserted and grep.deleted labels I find the "bold" makes it stand out a bit more when the green is next to the line number. Note that these labels are applied before the grep.change label, which is now disabled. To get the old colour, users can restore [color] grep.change = green dim to their hgrc.

grep: give different labels to + and - symbols I find it more useful to give different colours to plus and minus, but it's difficult to do so if the default output uses the same label for both. The following augments the names of columns with some extra labels, akin to the diff.inserted and diff.deleted labels for the diff command. This is done by adding an extra label field to the columns tuples.

tests: bulk changes to avoid whitespace errors of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. This revision avoids "whitespace" errors of check-code.py below: - missing whitespace after "," - missing whitespace in expression - no whitespace around "=" for named parameters - wrong whitespace around "="

tests: replace "naked except clause" to avoid check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: bulk changes to avoid "omit superfluous pass" check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: replace imported module to avoid check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: bulk change to avoid "unneeded trailing ',' ..." error of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. "trailing ','" at the end of command option list in the code fragment of test-bundle2-format.t is still left as it is, in order to follow the style of command definitions in Mercurial core.

tests: define class in modern style to avoid check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: use NO_CHECK_EOF as heredoc limit mark to omit checking code fragments This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. "primes.py" embedded in test-highlight.t causes an error of check-commit.py below: don't use .next(), use next(...) But changing embedded primes.py is painful, because it is committed in test script, and affects hash IDs. On the other hand, primes.py itself is never executed in test script. Therefore, this patch uses NO_CHECK_EOF as heredoc limit mark in order to omit any checking on this code fragments.

tests: conditionalize msys path mangling in test-bundle.t This broke in 252cc56c9ff6 when the variables were printed one per line. The only reason I can think of is that MSYS knew the former string couldn't be a list of paths, but thinks this form is.

tests: glob over quoting differences in test-blackbox.t for Windows

chistedit: ensure a locale is set My paternal surname was showing incorrectly without this fix.

templatekw: make negrev return empty for wdir() and nullrev I considered just returning the same output that {rev} returns here, but {rev} also returns essentially gibberish: either an INT_MAX-kind of variable for wdir() or -1 for null. Since these are numbers that are intended to be used for calculations, and since the numbers for wdir() and -1 are not really very helpful for calculation (and worse, when used as a revision number -1 is equal to unhidden tip), I figured the most reasonable thing to do here is to just return nothing for negrev. This could potentially break scripts that are expecting to parse a nonempty integer out of a {negrev}, but that seems like a very remote concern at this juncture.

test-sqlitestore: run sqlite3 with no init file These tests were failing for me because I have a ~/.sqliterc that alters the default output. Tests should ignore any init file.

tests: add a missing abort exit code on Windows in test-audit-subrepo.t

subrepo: add test for Windows relative-ish path with drive letter Matt Harbison pointed out that Windows had some weird path syntax. Fortunately it's rejected appropriately by pathauditor, so we're safe. Let's test the behavior as we have a special handling for Windows drive letters. This patch includes a basic example. Maybe we'll need to extend the test case further, but writing such tests on Linux isn't easy.

copies: return True instead of filename as it is expected to return boolean As the function documentation says this function should True if we hit a match and False otherwise. And I see that we are not using that returned filename anywhere. Differential Revision: https://phab.mercurial-scm.org/D5964

narrow: fix command name in error messsage Differential Revision: https://phab.mercurial-scm.org/D5982

templatekw: add a {negrev} keyword Revision numbers are getting much maligned for two reasons: they are too long in large repos and users get confused by their local-only nature. It just occurred to me that negative revision numbers avoid both of those problems. Since negative revision numbers change whenever the repo changes, it's much more obvious that they are a local-only convenience. Additionally, for the recent commits that we usually care about the most, negative revision numbers are always near zero. This commit adds a negrev templatekw to more easily expose negative revision numbers. It's not easy to reliably produce this output with existing keywords due to hidden commits while at the same time ensuring good performance.

tests: correct the remaining fallout from recent path style changes on Windows Per @martinvonz, `ui.slash` set by the test runner is now capable of playing a more active role.[1] I verified that both of these work by setting `ui.slash` to False, but these changes seem cleaner. The problem with check-perf-code.py was that the proper imports were not being whitelisted due to '\' vs '/'. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-February/128701.html

templates: adding a config() function for template customization This allows templates to be written such that users can customize them easily, or that they can be customized based on other configuration of the system. For enterprise deployments, we often have complex template aliases, and right now the only way individual users can customize those is by replacing the whole template alias (which means they won't get company-wide updates to it anymore, plus most users don't want to have to get a complex template right). With this change, they can just set a config option which feeds into our templates for common changes (e.g. whether to limit commit descriptions to the width of their terminal or not). To work around the issue of having to register the config options, I declared a dedicated section [templateconfig] for these options to be dynamically declared. They can still reference any other config option that's registered elsewhere. I only did string, bool and int at this time - list and date would add other complications with parsing the default so I'll leave that as an exercise to the reader :) Differential Revision: https://phab.mercurial-scm.org/D5959

changegroup: don't try to prune manifest nodes if not ellipses In non-ellipses case, the number of manifest nodes can be very big, and finding whether one of them can be pruned or not is very costly. For each node, we try to find the rev and the linkrev, which is expensive. Sending bit more manifest nodes seems better and much faster here. On our internal repository, this saves around 5 seconds on `hg tracked --addinclude <some_path>` on a narrow repo with ellipses disabled. Differential Revision: https://phab.mercurial-scm.org/D5782

branchmap: improve doc about BranchMapCache class It was confusing that we are having two different branchcache and BranchMapCache classes. The doc in BranchMapCache class was not much helpful to understand the difference. This patch improves the doc there. Differential Revision: https://phab.mercurial-scm.org/D5933

rust: itering less on MissingAncestors.bases for max() Instead of iterating on the whole `self.bases` each time to find its max, we keep the latter in a separate member attribute and keep it up to date in `add_bases()` On a perfdiscovery done on PyPy, with repos prepared with `contrib/discovery-helper.sh 50 100`, this gives a slight improvement (around 0.5% on wall time, but 10% on CPU) before: ! wall 0.172801 comb 0.180000 user 0.180000 sys 0.000000 (median of 541) after: ! wall 0.171798 comb 0.160000 user 0.160000 sys 0.000000 (median of 551) (perf command run time upped because of bigger variability during this test). Differential Revision: https://phab.mercurial-scm.org/D5945

rust: stop putting NULL_REVISION in MissingAncestors.bases As noted in initial review of MissingAncestors, adding NULL_REVISION in constructor in case the given `bases` is empty wasn't really useful, yet it's been kept for identity with the Python implementation Differential Revision: https://phab.mercurial-scm.org/D5944

rust: less set lookups in MissingAncestors using the return values of HashSet::remove(), we can factor pairs of `contains()/remove()` into a single `remove()`. On a perfdiscovery run done on the PyPy repository, prepared with contrib/discovery-helper.sh 50 100, I do get a modest improvement with this (mean of medians of three runs is better by 2%) Sample readings, before this change: ! wall 0.175609 comb 0.180000 user 0.180000 sys 0.000000 (median of 58) With this change: ! wall 0.171662 comb 0.180000 user 0.170000 sys 0.010000 (median of 60) Differential Revision: https://phab.mercurial-scm.org/D5943

rust: less set lookups in AncestorsIterator This uses the boolean return of `HashSet::insert()` to factor pairs of contains()/insert() into a single insert() On the mozilla-central repository (450k changesets), I get about a bit more than 10% better medians in perfancestors (taking the mean of three runs) Best run for parent changeset: ! wall 0.106474 comb 0.110000 user 0.110000 sys 0.000000 (median of 93) Best run for this changeset: ! wall 0.093191 comb 0.090000 user 0.090000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D5942

obsutil: don't assume leftctx and rightctx repo as same Backed out changeset 520514af2d93. hgsubversion can pass leftctx and rightctx which are instances of two different repositories. This was making tests fail on hgsubversion with 4.9. The two different instances are: (Pdb) p rightctx.repo() <filteredrepo:served <hgsubversion.svnrepo.svnlocalrepo object at 0x7fe29d296d10>> (Pdb) p leftctx.repo() <filteredrepo:visible <hgsubversion.svnrepo.svnlocalrepo object at 0x7fe29d494590>> Differential Revision: https://phab.mercurial-scm.org/D5968

tests: add more wildcards to test-extdiff.t The diff tool (which is `echo`) runs in the background and output order can therefore be non-deterministic. We need to glob over the file names to account for this. Differential Revision: https://phab.mercurial-scm.org/D5976

url: don't pass strict argument on Python 3 The argument was removed in Python 3.4. Differential Revision: https://phab.mercurial-scm.org/D5975

url: always use str for proxy configuration Previously, proxies didn't work on Python 3 for various reasons. First, the keys to the "proxies" dict are fed into a `setattr(self, "%s_open", ...)` call and passing bytestrings results in setting an oddly named attribute due to the b'' in %s formatting. This resulted in "http_open" and "https_open" not being properly overridden and proxies not being used. Second, the standard library was expecting proxy URLs to be str. And various operations (including our custom code in url.py) would fail to account for the str/bytes mismatch. This commit normalizes everything to str and adjusts our proxy code in url.py to account for the presence of str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5952

py3: port tinyproxy.py to work with Python 3 There were various str/bytes mismatches in the code. This caused the proxy server to misbehave at run-time. The manifestation was typically premature socket disconnect from the perspective of the client. Differential Revision: https://phab.mercurial-scm.org/D5951

url: always access req._tunnel_host The getattr() was there to handle Python versions before 2.6, which lacked this attribute. We /might/ be able to further delete some code here. However, the behavior here is extremely hard to follow because large parts of this code duplicate code from the Python standard library and it is difficult to understand what is actually needed. Differential Revision: https://phab.mercurial-scm.org/D5950

tests: double escape \ in test-import-eol.t and test-mq-eol.t The shell eats the \\. Differential Revision: https://phab.mercurial-scm.org/D5974

tests: use raw string in test-impexp-branch.t On first glance, the escaping of \s seems correct. However, the shell eats the escape and we're left with '\s` in the written file. Let's use a raw string so we don't have to double escape. Differential Revision: https://phab.mercurial-scm.org/D5973

phabricator: make user searches case-insensitive User names in conduit are case insensitive, but when looking for "FOO" it would return "foo" instead and we'd think the user didn't exist. So lower case both the query and the response when comparing them. Differential Revision: https://phab.mercurial-scm.org/D5934

exchange: raise error.Abort instead of ValueError Raising ValueError results in an uncaught exception and a traceback being printed. In the context of servers, it can result in an HTTP 500 and an exception being logged in the error log. I don't think this is proper behavior. The bundle2 code paths have a mechanism for translating an error.Abort into an error message reported to the clients. I think we should use that instead. This commit replaces some ValueError with Abort so that servers can error more gracefully. Differential Revision: https://phab.mercurial-scm.org/D5972

tests: remove -q from test-lfs-serve.t This will make it easier to observe a behavior change in the next commit. Differential Revision: https://phab.mercurial-scm.org/D5971

chistedit: use magenta for current line as in crecord (issue6071) It was inconsistent in the UI to have different way to show the current line.

chistedit: improve proper username in histedit curses interface in changeset section (issue6072) Differential Revision: https://phab.mercurial-scm.org/D5967

crecord: remove obsolete version check An internal function shouldn't be checking compatibility with Mercurial versions.

histedit: remove "chistedit" mention from interface "chisted" is internal jargon. The end user should not need to be aware that it's different from histedit.

revset: improve documentation on expectsize() This is a follow-up patch to D5813. It improves the documentation of `expectsize(set, size)`. Differential Revision: https://phab.mercurial-scm.org/D5953

scmutil: fix a comment that doesn't match the code Differential Revision: https://phab.mercurial-scm.org/D5956

remotefilelog: remove strkwargs() The previous commit activated previously unused code paths on Python 3 and revealed that we were calling strkwargs() on a dict that already had str keys. The only caller of _forceprefetch() is _adjustlinknode() a few lines up and the static commonlogkwargs dict features str keys. Differential Revision: https://phab.mercurial-scm.org/D5958

remotefilelog: use raw strings when looking for variable names Keys in self.__dict__ and vars() are always str. So we need to use raw strings to ensure lookups work on Python 3. # skip-blame just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5957

rust-cpython: binding for headrevs() This uses the core `dagops::retain_heads` to give a Rust implementation to `mercurial.dagop.headrevs`. Testing happens for now from `test-rust-ancestors.py` (for quick and minimal change), but it'd made more sense to put the binary index data elsewhere and to create a new test python module

rust-cpython: moved py_set() utility to conversion module We're still hoping to get rid of it eventually, but we're going to need it from outside the `ancestors` module before that.

rust: translated random test of missingancestors This is a Rust implementation of the random DAG generator and related incrementalmissingancestors tests against a naive brute force implementation. It is provided as an integration test, so that it won't run by default if any unit test fails. In case of a failed example, all needed information for reproduction is included in the panic message, (this is how `test_remove_ancestors_from_case1()` has been generated), as well as the random seed. The whole test is rerunnable by passing the random seed in the TEST_RANDOM_SEED environment variable. The other parameters (numbers of iterations) can be passed in the TEST_MISSING_ANCESTORS environment variable. An alternative would have been to expose to Python MissingAncestors<VecGraphs> but that would have meant pollution of the release build used from Python, whereas we do it in this changeset within the tests submodule Differential Revision: https://phab.mercurial-scm.org/D5417

debugpathcopies: fix typo in synopsis Differential Revision: https://phab.mercurial-scm.org/D5949

debugrename: don't require at least one path I don't see a reason that it needs to require a path. Most commands match everything when no paths are given, but here you have to do something like `hg debugrename -r . .` (from the repo root) to match everything. Differential Revision: https://phab.mercurial-scm.org/D5948

revlog: use iterbytestr() Otherwise we iterate over integers in Python 3 and the character compare fails. Differential Revision: https://phab.mercurial-scm.org/D5947

commit: respect ui.relative-paths The command usually doesn't print any paths, but there are some warnings and verbose messages that includes paths. Differential Revision: https://phab.mercurial-scm.org/D5939

resolve: slightly simplify join expression by joining with empty strings Differential Revision: https://phab.mercurial-scm.org/D5938

resolve: respect ui.relative-paths also for warning messages I guess this should have been part of 72a9aacff645 (resolve: respect ui.relative-paths, 2019-01-29). Differential Revision: https://phab.mercurial-scm.org/D5937

addremove: respect ui.relative-paths I previously changed these code paths while trying to not change any behavior to avoid inconsistencies between them in the intermediate commits. They're now all ready to be switched over to respecting ui.relative-paths. Differential Revision: https://phab.mercurial-scm.org/D5936