runtests: add an IPv6 command line flag Now we have all IPv6 related issues fixed, add a command line flag so people could actually run tests with IPv6.

runtests: always set web.ipv6 Previously, we only set web.ipv6 if IPv6 is used, but not on the IPv4 case. Since we already have set web.address, it makes sense to move "web.ipv6" out from "extra config options".

runtests: set web.address to localhost Previously, "hg serve" will listen on "", which is not clear which interface it will actually listen on - it could listen on all interfaces (ex. 0.0.0.0 on IPv4). The run-tests.py script only checks "localhost" for available ports. So let's make it the same for "hg serve" by explicitly setting "web.address" to "localhost". This resolves some IPv6 EADDRINUSE errors.

tests: use LOCALIP This patch replaces hardcoded 127.0.0.1 with $LOCALIP in all tests. Till now, the IPv6 series should make tests pass on common IPv6 systems where the local device has the address "::1" and the hostname "localhost" resolves to "::1".

dummyssh: use LOCALIP This patch replaces hard-coded 127.0.0.1 with $LOCALIP in dummyssh.

runtests: export LOCALIP Previously, tests hard-code local IP address as "127.0.0.1". That won't work for IPv6. This patch exports the $LOCALIP environment variable, which is set to "::1" if we decide to use IPv6.

tinyproxy: use IPv6 if HGIPV6 is set to 1 This patch makes tinyproxy.py work in IPv6 mode if HGIPV6 is set to 1. This will make test-http-proxy.t pass on IPv6 machines.

dumbhttp: use IPv6 if HGIPV6 is set to 1 This will fix flaky tests using dumbhttp. The patch was tested on gcc112.fsffrance.org using the following command: ./run-tests.py -j 40 --runs-per-test 120 test-bundle2-remote-changegroup.t

runtests: export HGIPV6 to hint test scripts whether to use IPv6 Previously, run-tests.py only exports HGPORT, and scripts in tests do not know if IPv6 should be used. And that breaks scripts like dumbhttp.py which always uses IPv4. This patch makes run-tests.py export HGIPV6, which can help test scripts like dumbhttp.py and tinyproxy.py to decide whether to use IPv6 or not.

runtests: prefer IPv4 to IPv6 To make IPv6 work, there are multiple areas that need to fix. Before they all get fixed, use IPv4 by default. This should fix tests caused on IPv6 systems.

doc: correct example concerning "hg purge" alias in man page "hgrc.5" The "hg purge" alias as currently described in "hgrc.5" issues a possibly confusing error message like rm: missing operand Try 'rm --help' for more information. if no files are to be purged at all. This patch slightly modifies the example by adding a "-f" option to the "rm" command.

tests: prove that ignore works

annotate: migrate to modern pager API

ui: add a debug print right before we start the pager This makes it easier to figure out why a command is getting paginated.

ui: respect historic pager.attend-$COMMAND=no I'm on the fence about this behavior, but the user's intent was pretty specific and it's not expensive to support this case.

tests: clean up a bunch of pager testing that is about to be invalidated All this attend logic and potential bugs just no longer make sense to test.

ui: add ignore-single-command functionality This closes the last feature gap other than the attend list from the extension. For now, I'm leaving the attend list in the extension, because I'm unsure it has merit in a world where commands have been updated to take advantage of the modern API.

ui: introduce neverpager() call I'm about to add direct paging support to some commands, and as a result we need a way to communicate from the higher layers of dispatch that paging is explicitly disabled.

pager: move more behavior into core This moves the global flag and the --pager=yes logic into core. Only functionality change is that users now always get a --pager flag and can enable the pager via the flag without the extension active. Moving the flag into core exposes a defect in the ro localization, which will have to be corrected later.

pager: move pager-initiating code into core No functionality change. A previous version of this API had a category argument on ui.pager(). As I migrated the commands in core, I couldn't come up with good enough consistency in any categorization scheme so I just scrapped the whole idea. It may be worth revisiting in the future.

test-logtoprocess: use cat to wait for outputs Commands started by logtoprocess are running asynchronously. To be able to test the output, we need to block and wait for the output. The patch uses "| cat" to wait for such "asynchronous" outputs, to make the test more reliable. I have also written a short notice at the top, hopefully people would be aware of the pitfall when changing the test.

chgserver: move comments in config example "#" must be the first character of a line to mark the text as comments. So let's change the docstring.

localrepo: move extension loading to a separate method The stateful chg plan [1] requires a special repo object, where ideally all side effects caused by loading the repo object could be reverted by just dropping (gabbage collect) the loaded repo object. Currently, that is impossible because repo.__init__ calls "extensions.loadall", which may have unpredictable side-effects that cannot be reverted by dropping the repo object. This patch moves "extensions.loadall" to a separate method, so chg could subclass localrepository and make extensions loading a no-op. [1]: mercurial-scm.org/pipermail/mercurial-devel/2017-February/092547.html

py3: convert the mode argument of os.fdopen to unicodes Couple of these from the earlier series got lost while rebasing. So this patch converts them again.

runtests: unindent an "if True" block The block was left to make review easier. This patch unindents it.

runtests: set web.ipv6 if we use IPv6 As explained by the previous patch, we need to set "web.ipv6=True" if we decide to use IPv6. Otherwise "hg serve" will still try to listen on IPv4. This patch makes it so by appending web.ipv6 to "extra configs". This patch was tested in a Linux system with IPv6, by the following steps: 1. Change hgweb/server.py temporarily to write a file if IPv6HTTPServer.__init__ is called. 2. run-tests.py -l --keep-tmpdir test-serve.t 3. Check the generated .hgrc, make sure it sets web.ipv6=1. 4. Check the log file to make sure IPv6HTTPServer.__init__ is called.

runtests: checkportisavailable should only check one family As explained by the previous patch, checkportisavailable() should only check the preferred family - either IPv4 or IPv6, not both. This patch makes it so.

runtests: add a function to test if IPv6 is available Previously, checkportisavailable returns True if the port is free either on IPv4 or IPv6, but the hg server only uses IPv4 by default. That leads to issues when IPv4 port is not free but the IPv6 one is. To address that, run-tests should stick with either IPv4 or IPv6. This patch adds a function similar to checkportisavailable to test if IPv6 is available, and assigns the result to a variable. The new function was tested in a Linux system script with the following steps: 1. Run "ip addr del ::1/128 dev lo" to delete lo's IPv6 address, Confirm checkipv6available() returns False. 2. Run "ip addr add ::1/128 dev lo" to add back lo's IPv6 address. Confirm checkipv6available() returns True. 3. Start a web server taking the 8000 port. Confirm checkipv6available(8000) is still True.

histedit: log the time taken to read in the commands list If we're being fed an external command list from stdin (histedit --commands -), then the time spent reading stdin is outside our control. Log it.

extdiff: log time spent in external diff program We can't fix the time external diff programs take to run. Log that duration for us to remove from any stats we gather

crecord: log blocked time waiting for curses input We want to know when we're blocked waiting for the user - log the time spent waiting in the curses keyboard handlers

ui: give editor() a tag of its own We know that calls to ui.editor() always block on the user's configured editor. Use a blocking tag that ensures that we don't see a huge variety of editor options in our logging.

ui: time calls to ui.system We want to know when we're blocked on ui.system, and why. Allow the user to supply a tag - otherwise we record on an unspecific tag derived from cmd.

ui: log time spent blocked on stdio We use a wrapper around Mercurial at Facebook that logs key statistics (like elpased time) to our standard performance tooling. This is less useful than it could be, because we currently can't tell when a command is slow because we need to fix Mercurial versus when a command is slow because the user isn't interacting quickly. Teach Mercurial to log the time it spends blocked, so that our tooling can pick it up and submit it with the elapsed time - we can then do the math in our tooling to see if Mercurial is slow, or if the user simply failed to interact. Combining this with the command duration log means that we can ensure that we concentrate performance efforts on the things that bite Facebook users. The perfwrite microbenchmark shifts from: Linux: ! wall 3.213560 comb 0.410000 user 0.350000 sys 0.060000 (best of 4) Mac: ! wall 0.342325 comb 0.180000 user 0.110000 sys 0.070000 (best of 20) before this change to: ! wall 3.478070 comb 0.500000 user 0.420000 sys 0.080000 (best of 3) Mac: ! wall 0.218112 comb 0.220000 user 0.150000 sys 0.070000 (best of 15) showing a small hit in comb time, but firmly in the noise on wall time.

contrib: add a write microbenchmark to perf.py I'm adding some performance logging to ui.write - this benchmark lets us confirm that the cost of that logging is acceptably low. At this point, the microbenchmark on Linux over SSH shows: ! wall 3.213560 comb 0.410000 user 0.350000 sys 0.060000 (best of 4) while on the Mac locally, it shows: ! wall 0.342325 comb 0.180000 user 0.110000 sys 0.070000 (best of 20)

ui: provide a mechanism to track and log blocked time We want to log the time Mercurial spends trapped in things outside programmatic control. Provide a mechanism to give us both command runtime and as many different sources of blocking as we deem useful.

mercurial: switch to util.timer for all interval timings util.timer is now the best available interval timer, at the expense of not having a known epoch. Let's use it whenever the epoch is irrelevant.

util: introduce timer() As documented for timeit.default_timer, there are better timers available for performance measures on some platforms. These timers don't have a set epoch, and thus are only useful for interval measurements, but have higher resolution, and thus get you a better measurement overall. Use the same selection logic as Python's timeit.default_timer. This is a platform clock on Python 2 and early Python 3, and time.perf_counter on Python 3.3 and later (where time.perf_counter is introduced as the best timer to use).

color: move the '_render_effects' function to the core module

color: move '_effect_str' function into the core module

color: move configstyles into the core module The extension is getting thinner as we speak!

color: rework conditional 'valideffect' Not very important, but the full conditional is not that hard to follow and having it unified make the function role a bit clearer in my opinion.

color: move 'valideffect' function into the core module

color: move '_terminfo_params' into the core 'color' module On step closer to have color in core.

color: move '_effect' mapping into core This is the second things we can move into core safely.

color: spread '_effect' values for readability We move to our "usual" one value per line style.

merge with stable

update: clarify that -C and -c are mutually exclusive This makes it clear in both the synopsis and in the verbose output that -C and -c are mutually exclusive. It also restructures the verbose output a little so it's better prepared for a third option (--merge). This patch also reorders the options to match the flag table.

update: move check for dirty wdir into hg.updatetotally() The function has a "check" parameter that's currently unused, and it makes sense to me to have it honor it. That way other callers than commands.update() could set it if they needed.

destutil: drop now-unused "check" parameter from destupdate()

destutil: remove duplicate check and leave it to merge.update() The check is done in merge.update() already and the next few patches will add more checks there. Some of the additional checks will need information about the merge that will not be available in destutil. Since commands.postincoming() catches UpdateAbort(), we need to change merge.update() to raise that more specific exception. This goes directly again 45b86dbabbda (destupdate: move the check related to the "clean" logic in the function, 2015-10-05), but it will simplify the next few patches, and we can always move it out again (preferably move, not copy) after if we still think it's better that way.

make: update .PHONY targets

debugcommands: move 'debugwireargs' in the new module

debugcommands: move 'debugwalk' in the new module

debugcommands: move 'debugtemplate' in the new module

debugcommands: move 'debugsuccessorssets' in the new module

debugcommands: move 'debugsub' in the new module

debugcommands: move 'debugstate' in the new module

debugcommands: move 'debugsetparents' in the new module

debugcommands: move 'debugrevspec' in the new module

debugcommands: move 'debugrevlog' in the new module

debugcommands: move 'debugrename' in the new module

debugcommands: move 'debugrebuildfncache' in the new module

debugcommands: move 'debugrebuilddirstate' in the new module

debugcommands: move 'debugpvec' in the new module

debugcommands: move 'debugpushkey' in the new module

ui: remove urllib2 from being imported early Before this change, urllib2 was brought in when constructing the ui object, and that added ~5ms on my Linux workstation to the hg startup time for every command, bringing the time for 'HGRCPATH=/dev/null hg root' from 46ms to 40ms. Now, we construct a single proxy object per initial ui creation (so that if the ui is copied they share the object), but defer the actual instantiation of it and the import of urllib2 until it's needed. # no-check-commit

tests: switch "this command isn't paged" example to id I'm about to enable pager support in summary.

tests: update test-i18n.t to not depend on the pager extension I'm about to mark that extension as deprecated, and that broke this test.

pager: add a test of --pager=no functionality I'm about to upend the pager universe, but I would like to not regress anything.

hg: allow usage of XDG_CONFIG_HOME/hg/hgrc Modern applications must use the following paths to store configuration files: - $XDG_CONFIG_HOME/hg/hgrc, - $HOME/.config/hg/hgrc (if XDG_CONFIG_HOME is unset or not absolute). For backward compatibility, ~/.hgrc is still created if no hgrc exist using hg config --edit. See https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

debugcommands: move 'debugpathcomplete' in the new module

debugcommands: move 'debugobsolete' in the new module

debugcommands: move 'debuglocks' in the new module

debugcommands: move 'debugnamecomplete' in the new module

debugcommands: move 'debugmergestate' in the new module

debugcommands: move 'debuglabelcomplete' in the new module

dispatch: start profiling earlier This makes it possible to profile extension loading and setup, which takes a substantial fraction of overall execution time for fast commands. (99% of this commit is simply changes of indentation to reflect the hoisting of the two calls to maybeprofile to a single one that happens earlier.) # no-check-commit

dispatch: move detection of profiling earlier during startup

ui: fix configwith doctest 4.2 cannot be expressed with IEEE floating point losslessly, and could cause test failure on some platform: File ".../mercurial/ui.py", line 414, in mercurial.ui.ui.configwith Failed example: u.configwith(float, s, 'float2') Expected: -4.2 Got: -4.2000000000000002 This patch fixes that by changing the number to 4.25, which can be expressed by the binary number 100.01.

test-bdiff: move import inside the function to avoid test failure test-check-module-imports.t fails on some systems where the path of home directories is different than sys.prefix and sys.exec_prefix. Importing silenttestrunner will help avoiding that failure.

profiling: add statprof support for Chrome trace viewer rendering We synthesize function call begin/end events from snapshots, and try (configurably) to eliminate "noisy" stack frames. Example invocation: hg --config profiling.output=$HOME/Desktop/clone.json \ --config profiling.statformat=chrome \ --profile clone https://www.mercurial-scm.org/repo/hg

statprof: allow rendering in the Chrome trace viewer format

statprof: add a path simplification function

ui: rewrite configint in terms of configwith

ui: add a configwith method This is a long-overdue generalization of the pattern in configint and configbool.

py3: convert the mode argument of os.fdopen to unicodes (2 of 2)

py3: convert the mode argument of os.fdopen to unicodes (1 of 2) os.fdopen() does not accepts bytes as its second argument which represent the mode in which the file is to be opened. This patch makes sure unicodes are passed in py3 by using pycompat.sysstr().

bugzilla: add a rest api backend (usable with bugzilla 5.0+) Add support for the bugzilla rest api documented at https://wiki.mozilla.org/Bugzilla:REST_API and at https://bugzilla.readthedocs.io/en/latest/ This backend has the following benefits: * It supports the bugzilla api keys so hgrc does not need to contain a user's bugzilla password * Works with Mercurial's "hostfingerprints" support making handling bugzilla instances with self-signed certs easier * Does not use xmlrpc ;-) Adds configuration item 'apikey' in [bugzilla] section. My major concern with these patches is if the approach to HTTP access is the right way for an extension and if hooking into request object and the overriding the get_method to perform PUT requests was a sensible approach. # no-check-commit

keepalive: honor urllib2 style get_method overrides In urllib2 docs and discussions it can be found that in order to make a request other than GET or POST the get_method of a Request object can be overriden. Make Mercurial's internal version of this honor the return value of get_method. Please see the followup patch to the bugzilla extension for the reason for this change. Marking RFC because I'm not entirely sure this is the right way make the HTTP requests.

lock: include Linux pid namespace identifier in prefix Previously, the lock only contains a hostname as an attempt to detect pid namespace difference. However, that's not enough on modern Linux - a single hostname could have different pid namespaces. That means if people run hg inside different PID namespaces with a same UTS namespae, the lock would be broken - an hg proccess in pid namespace A will think the lock having a "random" pid in pid namespace B is "dead" and remove it. This patch solves the above issue by appending an PID namespace identifier of the current process to the lock prefix ("hostname"). It depends on /proc being mounted properly. But I don't think there is a better way to get pid namespace identifier reliably.

lock: move lock._host calculation to a function This allows customization per platform. See the next patch for why.

debugcommands: move 'debugknown' in the new module

debugcommands: extract debuginstall in the debugcommands module

dispatch: load debugcommand before extension Multiple extension will manipulate commands on load, we need the debug command to be loaded before that point.

merge with stable

bundle2: fix assertion that 'compression' hasn't been set `n.lower()` will return `compression`, not `Compression`.

wireproto: properly report server Abort during 'getbundle' Previously Abort raised during 'getbundle' call poorly reported (HTTP-500 for http, some scary messages for ssh). Abort error have been properly reported for "push" for a long time, there is not reason to be different for 'getbundle'. We properly catch such error and report them back the best way available. For bundle, we issue a valid bundle2 reply (as expected by the client) with an 'error:abort' part. With bundle1 we do as best as we can depending of http or ssh.

getbundle: cleanly handle remote abort during getbundle bundle2 allow the server to report error explicitly. This was initially implemented for push but there is not reason to not use it for pull too. This changeset add logic similar to the one in 'unbundle' to the client side of 'getbundle'. That logic make sure the error is properly reported as "remote". This will allow the server side of getbundle to send clean "Abort" message in the next changeset.

bundle1: fix bundle1-denied reporting for pull over ssh Changeset b288fb2724bf introduced a config option to have the server deny pull using bundle1. The original protocol has not really been design to allow that kind of error reporting so some hack was used. It turned the hack only works on HTTP and that ssh server hangs forever when this is used. After further digging, there is no way to report the error in a unified way. Using `ooberror` freeze ssh and raising 'Abort' makes HTTP return a HTTP-500 without further details. So with sadness we implement a version that dispatch according to the protocol used. Now the error is properly reported, but we still have ungraceful abort after that. The protocol do not allow anything better to happen using bundle1.

bundle-tests: operate from outside a repository We are about to add a test for ssh pull/cloning being denied because of bundle1 usage. For this, it is cleaner to not operate from the clone using http. So we update the test beforehand for clarity. This is more churns that what I'm happy to see on stable, but the rests of the series is worth it in my opinion.

bundle1: display server abort hint during unbundle The code was printing the abort message but not the hint. This is now fixed.

bundle1: fix bundle1-denied reporting for push over ssh Changeset b288fb2724bf introduced a config option to have the server deny push using bundle1. The original protocol has not really be design to allow such kind of error reporting so some hack was used. It turned the hack only works on HTTP and that ssh wire peer hangs forever when the same hack is used. After further digging, there is no way to report the error in a unified way. Using 'ooberror' freeze ssh and raising 'Abort' makes HTTP return a HTTP500 without further details. So with sadness we implement a version that dispatch according to the protocol used. We also add a test for pushing over ssh to make sure we won't regress in the future. That test show that the hint is missing, this is another bug fixed in the next changeset.

bundle2: keep hint close to the primary message when remote abort The remote hint message was ignored when reporting the remote error and passed to the local generic abort error. I think I might initially have tried to avoid reimplementing logic controlling the hint display depending of the verbosity level. However, first, there does not seems to have such verbosity related logic and second the resulting was wrong as the primary error and the hint were split apart. We now properly print the hint as remote output.

misc: update year in copyright lines This patch also makes some expected output lines in tests glob-ed for persistence of them. BTW, files below aren't yet changed in 2017, but this patch also updates copyright of them, because: - mercurial/help/hg.1.txt almost all of "man hg" output comes from online help of hg command, and is already changed in 2017 - mercurial/help/hgignore.5.txt - mercurial/help/hgrc.5 "copyright 2005-201X Matt Mackall" in them mentions about copyright of Mercurial itself

localrepo: avoid unnecessary sorting headrevs output already sorted, no need to sort it again.

localrepo: cache self.changelog in local variable Repeated self.changelog lookups can incur overhead. Let's cache it in the separate variable.

destutil: remove dead code about non-linear updates IIUC, the non-linear updates no longer happen by default since 6b1fc09c699a (update: change default destination to tipmost descendant (issue4673) (BC), 2016-02-02), and it was only if they happened by default that we used to error out, so there is no longer a need to handle this case.

update: fix typo/stale comment to match code The comment about "obsolete.background" seems to have been about obsolete.foreground ever since it was introduced in a59e575c6ff8 (update: allow dirty update to foreground (successors), 2013-04-16), so let's change it.

merge: remove unused handling of default destination in merge.update() As far as I can tell, all the callers of merge.update() have been migrated over to use destutil to find the default destination when the revision was not specified. So it's time to delete the code for handling a node value of None. Let's add an assertion that node is not None, so any extensions relying on it will not silently misbehave.

update: localize logic around which ancestor to use The merge code works by applying the changes between an ancestor and the working copy onto the destination. To achieve an update, it sets the ancestor to be the parent of the working copy. To achieve a clean update (update -C), it sets the ancestor to be the working copy itself (so there are no changes to carry over). The logic for this was spread out a bit. Let's move it all to one place so it's easier to follow the reason for it. Also add some documentation.

tests: add test for updating to null revision While working on merge.py, I realized that we don't (as far as I could tell) have any tests for updating to the null revision with a dirty working copy. This adds some simple tests for that.

import: mention "stdin" (abbreviated) and add example I actually didn't even think it was possible because I searched the help text for "stdin", and didn't even think of searching for "standard input". Let's mention the abbreviated form too to help others like me. (When importing from stdin, we actually print a message saying "applying patch from stdin".) This patch also adds an example showing how to import from stdin.

merge: print status message before launching external merge tool It seems somewhat common that people run into a merge conflict and don't notice the launched merge tool, and instead they think hg just hung. Let's print a message for each file that we launch a GUI merge tool for.

pager: exit cleanly on SIGPIPE (BC) Changeset aaa751585325 removes SIGPIPE handling completely. This is wrong, as it means that Mercurial does not exit when the pager does. Instead, raise SignalInterrupt when SIGPIPE happens with a pager attached, to trigger the normal exit path. This will cause "killed!" to be printed to stderr (hence the BC warning), but in the normal pager use case (where the pager gets both stderr and stdout), this message is lost as we only get SIGPIPE when the pager quits.

runtests: catch EPROTONOSUPPORT in checkportisavailable This is a follow-up of "runtests: check ports on IPv6 address". On some platforms, "socket.AF_INET6" exists while that does not necessarily mean the platform support IPv6 - when initializing a socket using "socket.socket", it could fail with EPROTONOSUPPORT. So treat that as "Port unavailable".

zstd: vendor python-zstandard 0.7.0 Commit 3054ae3a66112970a091d3939fee32c2d0c1a23e from https://github.com/indygreg/python-zstandard is imported without modifications (other than removing unwanted files). The vendored zstd library within has been upgraded from 1.1.2 to 1.1.3. This version introduced new APIs for threads, thread pools, multi-threaded compression, and a new dictionary builder (COVER). These features are not yet used by python-zstandard (or Mercurial for that matter). However, that will likely change in the next python-zstandard release (and I think there are opportunities for Mercurial to take advantage of the multi-threaded APIs). Relevant to Mercurial, the CFFI bindings are now fully implemented. This means zstd should "just work" with PyPy (although I haven't tried). The python-zstandard test suite also runs all tests against both the C extension and CFFI bindings to ensure feature parity. There is also a "decompress_content_dict_chain()" API. This was derived from discussions with Yann Collet on list about alternate ways of encoding delta chains. The change most relevant to Mercurial is a performance enhancement in the simple decompression API to reuse a data structure across operations. This makes decompression of multiple inputs significantly faster. (This scenario occurs when reading revlog delta chains, for example.) Using python-zstandard's bench.py to measure the performance difference... On changelog chunks in the mozilla-unified repo: decompress discrete decompress() reuse zctx 1.262243 wall; 1.260000 CPU; 1.260000 user; 0.000000 sys 170.43 MB/s (best of 3) 0.949106 wall; 0.950000 CPU; 0.950000 user; 0.000000 sys 226.66 MB/s (best of 4) decompress discrete dict decompress() reuse zctx 0.692170 wall; 0.690000 CPU; 0.690000 user; 0.000000 sys 310.80 MB/s (best of 5) 0.437088 wall; 0.440000 CPU; 0.440000 user; 0.000000 sys 492.17 MB/s (best of 7) On manifest chunks in the mozilla-unified repo: decompress discrete decompress() reuse zctx 1.367284 wall; 1.370000 CPU; 1.370000 user; 0.000000 sys 274.01 MB/s (best of 3) 1.086831 wall; 1.080000 CPU; 1.080000 user; 0.000000 sys 344.72 MB/s (best of 3) decompress discrete dict decompress() reuse zctx 0.993272 wall; 0.990000 CPU; 0.990000 user; 0.000000 sys 377.19 MB/s (best of 3) 0.678651 wall; 0.680000 CPU; 0.680000 user; 0.000000 sys 552.06 MB/s (best of 5) That should make reads on zstd revlogs a bit faster ;) # no-check-commit

tests: exclude python-zstandard from pyflakes analysis

py3: fix the way we produce bytes list in store.py bytes(range(127)) does not produce a list whereas we need a list. This patch fixes that.

py3: convert os.__file__ to bytes os.__file__ returns unicode path on Python 3. We need to have bytespath. This patch uses pycompat.fsencode() to encode unicode path to bytes path.

commandserver: handle backlog before exiting Previously, when a chg server is exiting, it does not handle connected clients so clients may get ECONNRESET and crash: 1. client connect() # success 2. server shouldexit = True and exit 3. client recv() # ECONNRESET d7875bfbfccb makes this race condition easier to reproduce if a lot of short chg commands are started in parallel. This patch fixes the above issue by unlinking the socket path to stop queuing new connections and processing all pending connections before exit.

misc: replace domain of mercurial-devel ML address by mercurial-scm.org This patch also adds new check-code.py pattern to detect invalid usage of "mercurial-devel@selenic.com".

i18n: update Report-Msgid-Bugs-To property of *.po files This patch replaces domain of mercurial-devel ML address by mercurial-scm.org for "Report-Msgid-Bugs-To" property of each *.po files. This avoids releasing 4.1.1 with invalid "Report-Msgid-Bugs-To" in *.mo file, if corresponded *.po file isn't msgmerge-ed with recent hg.pot by translator. These *.po files aren't covered by check-code.py pattern newly added in subsequent patch, because it ignores them.

misc: replace domain of mercurial ML address by mercurial-scm.org This patch also adds new check-code.py pattern to detect invalid usage of "mercurial@selenic.com". Change for test-convert-tla.t is tested, but similar change for almost same test-convert-baz.t isn't yet tested actually, because I couldn't find out the way to get "GNU Arch baz client". AFAIK, buildbot skips test-convert-baz.t, too. Does anybody have appropriate environment for testing?

commandserver: prevent unlink socket twice This patch changes unixforkingservice so it only calls `self._servicehandler.unlinksocket(self.address)` at most once. This is needed by the next patch.

runtests: check ports on IPv6 address Previously, checkportisavailable only checks ports on the IPv4 address. This patch makes it check IPv6 as well. It'll be useful if "localhost" does not have an IPv4 address, or its IPv4 address does not exist somehow.

zeroconf: fail nicely on IPv6 only system zeroconf only knows how to deal with IPv4; I develop on a system where the only IPv4 address is 127.0.0.1. Teach zeroconf to ignore IPv6 addresses when looking for plausible IPv4 connectivity.

chg: verify XDG_RUNTIME_DIR According to the specification [1], $XDG_RUNTIME_DIR should be ignored unless: The directory MUST be owned by the user, and he MUST be the only one having read and write access to it. Its Unix access mode MUST be 0700. This patch adds a check and ignores it if it does not meet part of the criteria. [1]: https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

check-code: permit functools.reduce

perf: split obtaining chunks from decompression Previously, the code was similar to what revlog._chunks() was doing, which took a raw data segment and delta chain, obtained buffers for the raw revlog chunks within, and decompressed them. This commit splits the "get raw chunks" action from "decompress." The goal of this change is to more accurately measurely decompression performance. On a ~50k deltachain for a manifest in mozilla-central: ! full ! wall 0.430548 comb 0.440000 user 0.410000 sys 0.030000 (best of 24) ! deltachain ! wall 0.016053 comb 0.010000 user 0.010000 sys 0.000000 (best of 181) ! read ! wall 0.008078 comb 0.010000 user 0.000000 sys 0.010000 (best of 362) ! rawchunks ! wall 0.033785 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) ! decompress ! wall 0.327126 comb 0.320000 user 0.320000 sys 0.000000 (best of 31) ! patch ! wall 0.032391 comb 0.030000 user 0.030000 sys 0.000000 (best of 100) ! hash ! wall 0.012587 comb 0.010000 user 0.010000 sys 0.000000 (best of 233)

smartset: move set classes and related functions from revset module (API) These classes are pretty large and independent from revset computation. 2961 mercurial/revset.py 973 mercurial/smartset.py 3934 total revset.prettyformatset() is renamed to smartset.prettyformat(). Smartset classes are aliased since they are quite common in revset.py.

help: test if "hg help TOPIC" reference is valid It's quite easy to make a reference invalid by mistake.

help: uppercase command placeholder 'command' isn't a valid help topic but a placeholder text. Make it upper case to avoid confusion.

help: show section that couldn't be found For better error indication.

cmdutil: remove forwarding methods per deprecation policy

util: always force line buffered stdout when stdout is a tty (BC) pager replaced stdout with a line buffered version to work around glibc deciding on a buffering strategy on the first write to stdout. This is going to make my next patch hard, as replacing stdout will make tracking time spent blocked on it more challenging. Move the line buffering requirement to util.py, and remove it from pager. This means that the abuse of ui.formatted=True and pager set to cat or equivalent no longer results in a line-buffered output to a pipe, hence (BC), although I don't expect anyone to be affected

localrepo: avoid unnecessary conversion from node to rev changelog.heads() first calls headrevs then converts them to nodes. localrepo.heads() then sorts them using self.changelog.rev function and makes useless conversion back to revs. Instead let's call changelog.headrevs() from localrepo.heads(), sort the output and then convert to nodes. Because headrevs does not support start parameter this optimization only works if start is None.

debian: update copyright years

debian: update mailing list address

bundle2: implement a basic __repr__ for bundle2 part We display basic data as the part id and part type. This make debugging bundle2 related code friendlier.

bundle2: drop an outdated comment The function is no longer in "early" stage and have been used in production for years. We can probably drop that part of the docstring...

unbundle: swap conditional branches for clarity This is a small style update for clarity. The previous situation was: if foo: 50 lines else: 2 lines In such case I tend to invert these to get the simpler branch out of the way earlier: if not foo: 2 lines else: 50 lines This makes the conditional and various alternatives fit on the same screen, simpler to read overall.

unbundle: add a small comment to tag the bundle1 case as such This makes the code clearer to understand for someone new to it (or rusted)

unbundle: add a small comment to clarify the 'check_heads' call Bundle2 has its own mechanisms to check for heads (and other) changes, so push using bundle2 is relying on the "check:heads" bundle part of unbundle and the 'check_heads' call is not checking anything. We add a small comment to make this clearer.

pager: don't terminate with extreme prejudice on SIGPIPE (BC) The default SIGPIPE handler causes Mercurial to exit immediately, without running any Python cleanup code (except and finally blocks, atexit handlers etc). This creates problems if you want to do something at exit. If we need a different exit code for broken pipe from pager, then we should code that ourselves in Python; this appears to have been cargo-culted from the fork implementation of pager that's no longer used, where it was needed to stop Broken Pipe errors appearing on the user's terminal.

verify: replace _validpath() by matcher The verifier calls out to _validpath() to check if it should verify that path and the narrowhg extension overrides _validpath() to tell the verifier to skip that path. In treemanifest repos, the verifier calls the same method to check if it should visit a directory. However, the decision to visit a directory is different from the condition that it's a matching path, and narrowhg was working around it by returning True from its _validpath() override if *either* was true. Similar to how one can do "hg files -I foo/bar/ -X foo/" (making the include pointless), narrowhg can be configured to track the same paths. In that case match("foo/bar/baz") would be false, but match.visitdir("foo/bar/baz") turns out to be true, causing verify to fail. This may seem like a bug in visitdir(), but it's explicitly documented to be undefined for subdirectories of excluded directories. When using treemanifests, the walk would not descend into foo/, so verification would pass. However, when using flat manifests, there is no recursive directory walk and the file path "foo/bar/baz" would be passed to _validpath() without "foo/" (actually without the slash) being passed first. As explained above, _validpath() would return true for the file path and "hg verify" would fail. Replacing the _validpath() method by a matcher seems like the obvious fix. Narrowhg can then pass in its own matcher and not have to conflate the two matching functions (for dirs and files). I think it also makes the code clearer.

rebase: fix code comment to refer to right issue (4504, not 4505) The comment was introduced in 8a544fb645bb (rebase: ensure rebase revision remains visible (issue4504), 2015-01-27), which mentions the right issue in the description.

merge with stable

Added signature for changeset e1526da1e6d8

Added tag 4.1 for changeset e1526da1e6d8

merge with i18n

i18n-pt_BR: synchronized with dfc6663f97ca

merge: more safe detection of criss cross merge conflict between dm and r 41f6af50c0d8 introduced handling of a crash in this case. A review comment suggested that it was not entirely obvious that a 'dm' always would have a 'r' for the source file. To mitigate that risk, make the code more conservative and make less assumptions.

tests: correct (I think) command in test-largefiles-update When this test was introduced, it used the short-form of all the flags on this update invocation. I suspect, based on the "start with clean dirstates" comment and the fact that the no-exec branch of the #if guard leaves dirstate clean, that this should have been 'update -qCr' instead of 'update -qcr', but that a bug in largefiles --check handling left this problem unnoticed. I'll leave a breadcrumb further up about the current failure mode in the hopes that we can fix this some day. This was previously discussed in [0] but the trail in that thread goes cold after a few replies. Given that this is still a flaky test, that appears to only be passing by bad fortune, I think it's worth correcting the code of the test to make a correct assertion, and to keep track of the suspected bug with some other mechanism than an invalid test (if we had support for "expected failure" blocks this might be a worthwhile use of them?). 0: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089501.html

tests: expand flags to long form in test-largefiles-update.t I spent some time confused by this test. I'm pretty sure that this line intends to be cleaning the dirstate, not checking that it's clean before updating: the preceding #if block leaves the dirstate clean in the noexec case, and dirty in the exec case, so we can't expect consistent behavior across that platform variation. A subsequent patch will modify this command to use --clean instead of --check. I'll elaborate in that patch about the hypothetical bug here.

merge: fix crash on criss cross merge with dir move and delete (issue5020) Work around that 'dm' in the data model only can have one operation for the target file, but still can have multiple and conflicting operations on the source file where the other operation is a 'rm'. The move would thus fail with 'abort: No such file or directory'. In this case it is "obvious" that the file should be removed, either before or after moving it. We thus keep the 'rm' of the source file but drop the 'dm'. This is not a pretty fix but quite "obviously" safe (famous last words...) as it only touches a rare code path that used to crash. It is possible that it would be better to swap the files for 'dm' as suggested on https://bz.mercurial-scm.org/show_bug.cgi?id=5020#c13 but it is not entirely obvious that it not just would create conflicts on the other file. That can be revisited later.

tests: use 'f' in test-merge-criss-cross.t to prepare for recursive dumping Prepare for adding a test case with files in a directory.

util: make sortdict.keys() return a copy dict.keys() is documented to return a copy, so it's surprising that sortdict.keys() did not. I noticed this because we have an extension that calls readlocaltags(). That method tries to remove any tags that point to non-existent revisions (most likely stripped). However, since it's unintentionally working on the instance it's modifying, it sometimes fails to remove tags when there are multiple bad tags in a row. This was not caught because localrepo.tags() does an additional layer of filtering. sortdict is also used in other places, but I have not checked whether its keys() and/or __delitem__() methods are used there.

test-highlight: add normalization rule for Pygments 2.2 The test failed on Debian sid because of new class="vm".

tests: account for different newline behavior between Solaris and GNU grep GNU grep, when emitting a matching line that doesn't have a terminating newline, will add an extra newline. Solaris grep passes the original line through without the newline. This causes differences in test output when looking at the last line of the output of get-with-headers.py, which doesn't usually emit (and certainly doesn't guarantee) a terminating newline. Both grep implementations succeed in matching the requested pattern, though, so rely on specifying the full pattern on grep's commandline instead of expecting it in the output, and send the output to /dev/null.

tests: also allow "Protocol not supported" in test-http-proxy error I've seen this in a (misconfigured) FreeBSD jail which has ::1 as an entry for localhost, but IPv6 support is disabled in the jail. It took me months to figure out what was going on (and I only figured it out when tinyproxy.py got confused by similar IPv4-level misconfiguration of the localhost domain in /etc/hosts.) I don't feel strongly about this patch: on the one hand, it's papering over a host-level misconfiguration, but on the other it avoids some weird and hard to diagnose problems that can occur in weirdly restricted environments.

revset: prevent using outgoing() and remote() in hgweb session (BC) outgoing() and remote() may stall for long due to network I/O, which seems unsafe per definition, "whether a predicate is safe for DoS attack." But I'm not 100% sure about this. If our concern isn't elapsed time but CPU resource, these predicates are considered safe. Perhaps that would be up to the web/application server configuration? Anyway, outgoing() and remote() wouldn't be useful in hgweb, so I think it's okay to ban them.

tests: use an absolute path to get around '..' being invalid on a dead CWD Only FreeBSD seems to be this picky. Note that this explicit absolute-path `cd` exposes a defect in the test, in that we end up still inside the cwd-vanish repository, but that's not a regression in this change. Since we're in a code freeze, I'm doing the smallest thing possible to try and fix bugs on FreeBSD, rather than cleaning up the entire problem. I'll follow up with a more complete fix after the freeze.

ui: rename tmpdir parameter to more specific repopath This was requested by Augie and I agree that repopath is more descriptive.

pager: wrap _runcommand() no matter if stdout is redirected We've made chg utilize the common code path implemented in pager.py (by 815e1cefd082 and 493935e0327a), but the chg server does not always start with a tty. Because of this, uisetup() of the pager extension could be skipped on the chg server. Kudos given to Sean Farley for dogfooding new chg and spotting this problem.

shelve: make unshelve not crash when there are missing files (issue4176) This patch makes it possible to unshelve while having missing files in your repo as long as shelved changes don't touch those missing files. It also makes error message better otherwise.

statprof: require input file statprof has a __main__ handler that allows viewing of previously written data files. As Yuya pointed out during review, f42cd5434cc2 broke this. This patch fixes that.

tests: work around FreeBSD's unzip having slightly different output According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's what comes as /usr/bin/unzip, so we may as well cater to it since it's easy.

contrib: fix check-commit to not reject commits from `hg sign` and `hg tag` I'm tired of having a spurious red build every time we do a release. Fix it once and for all.

Added signature for changeset a1dd2c0c479e

Added tag 4.1-rc for changeset a1dd2c0c479e

tests: fix up some http tests for no-zstd case

freeze: merge default into stable for 4.1 code freeze

patchbomb: add tmpdir parameter to ui.edit call

histedit: add tmpdir parameter to ui.edit call

cmdutil: add tmpdir parament to ui.edit calls

ui: add a parameter to set the temporary directory for edit Until callsites are updated, this will have no effect. Once callsites are updated, specifying experimental.editortmpinhg will create editor temporary files in a subdirectory of .hg, which will make it easier for tool integrations to determine what repository is in play when they're asked to edit an hg-related file.

help: update help for `hg update` which was misleading (issue5427)

templater: add '{envvars}' to access environment variables Since the option for ui.exportableenviron is experimental, so is this template until the underlying API is sorted out.

ui: introduce an experimental dict of exportable environment variables Care needs to be taken to prevent leaking potentially sensitive environment variables through hgweb, if template support for environment variables is to be introduced. There are a few ideas about the API for preventing accidental leaking [1]. Option 3 seems best from the POV of not needing to configure anything in the normal case. I couldn't figure out how to do that, so guard it with an experimental option for now. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-January/092383.html

tests: test experimental.spacemovesdown config for commit -i The feature is still very experimental, but at least its behavior is captured in the test.

zstd: prevent potential free() of uninitialized memory This is a cherry pick of an upstream fix. The free() of uninitialed memory could likely only occur if a malloc() inside zstd fails. The patched functions aren't currently used by Mercurial. But I don't like leaving footguns sitting around.

revlog: give EXTSTORED flag value to narrowhg Narrowhg has been using "1 << 14" as its revlog flag value for a long time. We (Google) have many repos with that value in production already. When the same value was reserved for EXTSTORED, it made those repos invalid. Upgrading them will be a little painful. We should clearly have reserved the value for narrowhg a long time ago. Since the EXTSTORED flag is not yet in any release and Facebook also says they have not started using it in production, so it should be okay to change it. This patch gives the current value (1 << 14) back to narrowhg and gives a new value (1 << 13) to EXTSTORED.

help: don't let tools reflow revlog flags list Before this change, the text about revlog flags was reflowed into a single paragraph, which made it a bit hard to read. I don't even know the rules around this, but adding a blank line before each flag seems to prevent the reflowing.

help: format revlog.txt more closely to result The rendered text has spaces before each item in the list

hgweb: simplify calculation of first revision in filelog command

hgweb: restore ascending iteration on revs in filelog web command Follow-up on 96f811bceb85. Adjust back the "parity" generator's offset to keep rendering the same.

context: extract _changesinrange() out of blockancestors() We'll need it to write a blockdescendants function in next changeset.

shelve: allow multiple shelves with --patch and --stat Before this patch, there was a single way to see multiple shelves using `--patch --list` which show all the shelves. Doing `--patch s1 s2` returns an error. This patch allows to show multiple shelves using `--patch` and `--stat`.

zstd: vendor python-zstandard 0.6.0 Commit 63c68d6f5fc8de4afd9bde81b13b537beb4e47e8 from https://github.com/indygreg/python-zstandard is imported without modifications (other than removing unwanted files). This includes minor performance and feature improvements. It also changes the vendored zstd library from 1.1.1 to 1.1.2. # no-check-commit

util: add length argument to util.buffer() util.buffer() either returns inbuilt buffer function or defines a new one which slices. The inbuilt buffer() also has a length argument which is missing from the ones we defined. This patch adds that length argument.

py3: replace pycompat.getenv with encoding.environ.get pycompat.getenv returns os.getenvb on py3 which is not available on Windows. This patch replaces them with encoding.environ.get and checks to ensure no new instances of os.getenv or os.setenv are introduced.

patch: check length of git index header only if integer is specified Otherwise TypeError would be raised. Follows up d1901c4c8ec0.

localrepo: experimental support for non-zlib revlog compression The final part of integrating the compression manager APIs into revlog storage is the plumbing for repositories to advertise they are using non-zlib storage and for revlogs to instantiate a non-zlib compression engine. The main intent of the compression manager work was to zstd all of the things. Adding zstd to revlogs has proved to be more involved than other places because revlogs are... special. Very small inputs and the use of delta chains (which are themselves a form of compression) are a completely different use case from streaming compression, which bundles and the wire protocol employ. I've conducted numerous experiments with zstd in revlogs and have yet to formalize compression settings and a storage architecture that I'm confident I won't regret later. In other words, I'm not yet ready to commit to a new mechanism for using zstd - or any other compression format - in revlogs. That being said, having some support for zstd (and other compression formats) in revlogs in core is beneficial. It can allow others to conduct experiments. This patch introduces *highly experimental* support for non-zlib compression formats in revlogs. Introduced is a config option to control which compression engine to use. Also introduced is a namespace of "exp-compression-*" requirements to denote support for non-zlib compression in revlogs. I've prefixed the namespace with "exp-" (short for "experimental") because I'm not confident of the requirements "schema" and in no way want to give the illusion of supporting these requirements in the future. I fully intend to drop support for these requirements once we figure out what we're doing with zstd in revlogs. A good portion of the patch is teaching the requirements system about registered compression engines and passing the requested compression engine as an opener option so revlogs can instantiate the proper compression engine for new operations. That's a verbose way of saying "we can now use zstd in revlogs!" On an `hg pull` conversion of the mozilla-unified repo with no extra redelta settings (like aggressivemergedeltas), we can see the impact of zstd vs zlib in revlogs: $ hg perfrevlogchunks -c ! chunk ! wall 2.032052 comb 2.040000 user 1.990000 sys 0.050000 (best of 5) ! wall 1.866360 comb 1.860000 user 1.820000 sys 0.040000 (best of 6) ! chunk batch ! wall 1.877261 comb 1.870000 user 1.860000 sys 0.010000 (best of 6) ! wall 1.705410 comb 1.710000 user 1.690000 sys 0.020000 (best of 6) $ hg perfrevlogchunks -m ! chunk ! wall 2.721427 comb 2.720000 user 2.640000 sys 0.080000 (best of 4) ! wall 2.035076 comb 2.030000 user 1.950000 sys 0.080000 (best of 5) ! chunk batch ! wall 2.614561 comb 2.620000 user 2.580000 sys 0.040000 (best of 4) ! wall 1.910252 comb 1.910000 user 1.880000 sys 0.030000 (best of 6) $ hg perfrevlog -c -d 1 ! wall 4.812885 comb 4.820000 user 4.800000 sys 0.020000 (best of 3) ! wall 4.699621 comb 4.710000 user 4.700000 sys 0.010000 (best of 3) $ hg perfrevlog -m -d 1000 ! wall 34.252800 comb 34.250000 user 33.730000 sys 0.520000 (best of 3) ! wall 24.094999 comb 24.090000 user 23.320000 sys 0.770000 (best of 3) Only modest wins for the changelog. But manifest reading is significantly faster. What's going on? One reason might be data volume. zstd decompresses faster. So given more bytes, it will put more distance between it and zlib. Another reason is size. In the current design, zstd revlogs are *larger*: debugcreatestreamclonebundle (size in bytes) zlib: 1,638,852,492 zstd: 1,680,601,332 I haven't investigated this fully, but I reckon a significant cause of larger revlogs is that the zstd frame/header has more bytes than zlib's. For very small inputs or data that doesn't compress well, we'll tend to store more uncompressed chunks than with zlib (because the compressed size isn't smaller than original). This will make revlog reading faster because it is doing less decompression. Moving on to bundle performance: $ hg bundle -a -t none-v2 (total CPU time) zlib: 102.79s zstd: 97.75s So, marginal CPU decrease for reading all chunks in all revlogs (this is somewhat disappointing). $ hg bundle -a -t <engine>-v2 (total CPU time) zlib: 191.59s zstd: 115.36s This last test effectively measures the difference between zlib->zlib and zstd->zstd for revlogs to bundle. This is a rough approximation of what a server does during `hg clone`. There are some promising results for zstd. But not enough for me to feel comfortable advertising it to users. We'll get there...

revlog: use compression engine APIs for decompression Now that compression engines declare their header in revlog chunks and can decompress revlog chunks, we refactor revlog.decompress() to use them. Making full use of the property that revlog compressor objects are reusable, revlog instances now maintain a dict mapping an engine's revlog header to a compressor object. This is not only a performance optimization for engines where compressor object reuse can result in better performance, but it also serves as a cache of header values so we don't need to perform redundant lookups against the compression engine manager. (Yes, I measured and the overhead of a function call versus a dict lookup was observed.) Replacing the previous inline lookup table with a dict lookup was measured to make chunk reading ~2.5% slower on changelogs and ~4.5% slower on manifests. So, the inline lookup table has been mostly preserved so we don't lose performance. This is unfortunate. But many decompression operations complete in microseconds, so Python attribute lookup, dict lookup, and function calls do matter. The impact of this change on mozilla-unified is as follows: $ hg perfrevlogchunks -c ! chunk ! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6) ! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6) ! chunk batch ! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6) ! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6) $ hg perfrevlogchunks -m ! chunk ! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4) ! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4) ! chunk batch ! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5) ! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4) Changelog chunk reading is slightly faster but manifest reading is slower. What gives? On this repo, 99.85% of changelog entries are zlib compressed (the 'x' header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch swapped the test order of 'x' and '\0' so now 'x' is tested first. This makes changelogs faster since they almost always hit the first branch. This makes a significant percentage of manifest '\0' chunks slower because that code path now performs an extra test. Yes, I too can't believe we're able to measure the impact of an if..elif with simple string compares. I reckon this code would benefit from being written in C...

hgweb: build the "entries" list directly in filelog command There's no apparent reason to have this "entries" generator function that builds a list and then yields its elements in reverse order and which is only called to build the "entries" list. So just build the list directly, in reverse order. Adjust "parity" generator's offset to keep rendering the same.

convert: remove "replacecommitter" action As pointed out by Yuya, this action doesn't add much (any?) value.

ui: check EOF of getpass() response read from command-server channel readline() returns '' only when EOF is encountered, in which case, Python's getpass() raises EOFError. We should do the same to abort the session as "response expected." This bug was reported to https://bitbucket.org/tortoisehg/thg/issues/4659/

convert: config option to control Git committer actions When converting a Git repository to Mercurial at Mozilla, I encountered a scenario where I didn't want `hg convert` to automatically add the "committer: <committer>" line to commit messages. While I can hack around this by rewriting the Git commit before it is fed into `hg convert`, I figured it would be a useful knob to control. This patch introduces a config option that allows lots of control over the committer value. I initially implemented this as a single boolean flag to control whether to save the committer message. But then there was feedback that it would be useful to save the committer in extra data. While this patch doesn't implement support for saving in extra data, it does add a mechanism for extending which actions to take on the committer field. We should be able to easily add actions to save in extra data. Some of the implemented features weren't asked for. But I figured they could be useful. If nothing else they demonstrate the extensibility of this mechanism.

help: make "mergetool" an alias for "merge-tools" I've probably typed `hg help mergetool` dozens of times. I'm tired of it not working.

templatekw: force noprefix=False to insure diffstat consistency (issue4755) The result of diffstatdata should not depend on having noprefix set or not, as was reported in issue 4755. Forcing noprefix to false on call makes sure the parser receives the diff in the correct format and returns the proper result. Another way to fix this would have been to change the regular expressions in path.diffstatdata(), but that would have introduced many unecessary special cases.

check-code: reject module-level @cachefunc Module-level @cachefunc usage is risky because it can easily create a memory "leak". Let's reject it completely for now. If a valid usage comes up in the future, we can always improve the check or reconsider.

similar: remove caching from the module level To prevent Bad Things™ from happening, let's rework the logic to not use util.cachefunc.

patch: add label for coloring the similarity extended header Just like the summary says, this will colorize the: similarity index 88% line in the diff output.

patch: use opt.showsimilarity to calculate and show the similarity Tests have been added.

patch: add similarity config knob in experimental section This config knob will control whether or not to show the similarity calculation in the diff output: diff --git a/README.md b/foo.md similarity index 88% rename from README.md rename to foo.md --- a/README.md +++ b/foo.md

similar: move score function to module level Future patches will use this to report the similarity of a rename / copy in the patch output.

revset: abuse x:y syntax to specify line range of followlines() This slightly complicates the parsing (see the previous patch), but the overall result seems not bad. I keep x:, :y and : for future extension.

revset: do not transform range* operators in parsed tree This allows us to handle x:y range as a general range object. A primary user of it is followlines().

revset: add default value to getinteger() helper This seems handy.

revset: factor out getinteger() helper We have 4 revset functions that take integer arguments, and they handle their arguments in slightly different ways. This patch unifies them: - getstring() in place of getsymbol(), which is more consistent with the handling of integer revisions (both 1 and '1' are valid) - say "expects" instead of "requires" for type errors We don't need to catch TypeError since getstring() must return a string.

revset: rename rev argument of followlines() to startrev The rev argument has the same meaning as startrev of follow(), and I think startrev is more informative. followlines() is new function, we can make BC now.

help: use :hg: role and canonical name to point to revset string patterns Follows up 5dd67f0993ce. Now revisions.txt and revsets.txt has been merged, so use revisions.* as a pointer.

util: compression APIs to support revlog decompression Previously, compression engines had APIs for performing revlog compression but no mechanism to perform revlog decompression. This patch changes that. Revlog decompression is slightly more complicated than compression because in the compression case there is (currently) only a single engine that can be used at a time. However for decompression, a revlog could contain chunks from multiple compression engines. This means decompression needs to map to multiple engines and decompressors. This functionality is outside the scope of this patch. But it drives the decision for engines to declare a byte header sequence that identifies revlog data as belonging to an engine and an API for obtaining an engine from a revlog header.

crecord: add an experimental option for space key to move cursor down I really want to have an option of toggling a selection on a line and also moving cursor down as a single keystroke. It also kinda makes sense for space key to do this, because some other curses UIs in the wild do this (e.g. various file managers, htop). So I got an idea to make a config option that defaults to False for compatibility, but allows making crecord UI a lot more useful for people with big hunks. We add this an experimental option to experiment with this behavior.

perf: support multiple compression engines in perfrevlogchunks Now that the revlog has a reference to a compressor, it is possible to swap in other compression engines. So, teach `hg perfrevlogchunks` to do that. The default behavior of `hg perfrevlogchunks` is now to measure the compression performance of all compression engines implementing the revlog compressor API. This effectively adds the no-op "none" compressor and zstd (when available) into the default set. While we can't yet plug alternate compressors into revlogs, this command gives us a preview of the performance. On the mozilla-unified repository: $ hg perfrevlogchunks -c ! compress w/ none ! wall 0.115159 comb 0.110000 user 0.110000 sys 0.000000 (best of 86) ! compress w/ zlib ! wall 5.681406 comb 5.680000 user 5.680000 sys 0.000000 (best of 3) ! compress w/ zstd ! wall 2.624781 comb 2.620000 user 2.620000 sys 0.000000 (best of 4) $ hg perfrevlogchunks -m ! compress w/ none ! wall 0.124486 comb 0.120000 user 0.120000 sys 0.000000 (best of 79) ! compress w/ zlib ! wall 10.144701 comb 10.150000 user 10.150000 sys 0.000000 (best of 3) ! compress w/ zstd ! wall 4.383118 comb 4.390000 user 4.390000 sys 0.000000 (best of 3) Those numbers for zstd look promising. But they aren't the full story. For that, we'll need to look at decompression times and storage sizes. Stay tuned...

revlog: use compression engine API for compression This commit swaps in the just-added revlog compressor API into the revlog class. Instead of implementing zlib compression inline in compress(), we now store a cached-on-first-use revlog compressor on each revlog instance and invoke its "compress()" method. As part of this, revlog.compress() has been refactored a bit to use a cleaner code flow and modern formatting (e.g. avoiding parenthesis around returned tuples). On a mozilla-unified repo, here are the "compress" times for a few commands: $ hg perfrevlogchunks -c ! wall 5.772450 comb 5.780000 user 5.780000 sys 0.000000 (best of 3) ! wall 5.795158 comb 5.790000 user 5.790000 sys 0.000000 (best of 3) $ hg perfrevlogchunks -m ! wall 9.975789 comb 9.970000 user 9.970000 sys 0.000000 (best of 3) ! wall 10.019505 comb 10.010000 user 10.010000 sys 0.000000 (best of 3) Compression times did seem to slow down just a little. There are 360,210 changelog revisions and 359,342 manifest revisions. For the changelog, mean time to compress a revision increased from ~16.025us to ~16.088us. That's basically a function call or an attribute lookup. I suppose this is the price you pay for abstraction. It's so low that I'm not concerned.

util: compression APIs to support revlog compression As part of "zstd all of the things," we need to teach revlogs to use non-zlib compression formats. Because we're routing all compression via the "compression manager" and "compression engine" APIs, we need to introduction functionality there for performing revlog operations. Ideally, revlog compression and decompression operations would be implemented in terms of simple "compress" and "decompress" primitives. However, there are a few considerations that make us want to have a specialized primitive for handling revlogs: 1) Performance. Revlogs tend to do compression and especially decompression operations in batches. Any overhead for e.g. instantiating a "context" for performing an operation can be noticed. For this reason, our "revlog compressor" primitive is reusable. For zstd, we reuse the same compression "context" for multiple operations. I've measured this to have a performance impact versus constructing new contexts for each operation. 2) Specialization. By having a primitive dedicated to revlog use, we can make revlog-specific choices and leave the door open for more functionality in the future. For example, the zstd revlog compressor may one day make use of dictionary compression. A future patch will introduce a decompress() on the compressor object. The code for the zlib compressor is basically copied from revlog.compress(). Although it doesn't handle the empty input case, the null first byte case, and the 'u' prefix case. These cases will continue to be handled in revlog.py once that code is ported to use this API.

revlog: move decompress() from module to revlog class (API) Upcoming patches will convert revlogs to use the compression engine APIs to perform all things compression. The yet-to-be-introduced APIs support a persistent "compressor" object so the same object can be reused for multiple compression operations, leading to better performance. In addition, compression engines like zstd may wish to tweak compression engine state based on the revlog (e.g. per-revlog compression dictionaries). A global and shared decompress() function will shortly no longer make much sense. So, we move decompress() to be a method of the revlog class. It joins compress() there. On the mozilla-unified repo, we can measure the impact of this change on reading performance: $ hg perfrevlogchunks -c ! chunk ! wall 1.932573 comb 1.930000 user 1.900000 sys 0.030000 (best of 6) ! wall 1.955183 comb 1.960000 user 1.930000 sys 0.030000 (best of 6) ! chunk batch ! wall 1.787879 comb 1.780000 user 1.770000 sys 0.010000 (best of 6 ! wall 1.774444 comb 1.770000 user 1.750000 sys 0.020000 (best of 6) "chunk" appeared to become slower but "chunk batch" got faster. Upon further examination by running both sets multiple times, the numbers appear to converge across all runs. This tells me that there is no perceived performance impact to this refactor.

revlog: make compressed size comparisons consistent revlog.compress() compares the compressed size to the input size and throws away the compressed data if it is larger than the input. This is the correct thing to do, as storing compressed data that is larger than the input takes up more storage space and makes reading slower. However, the comparison was implemented inconsistently. For the streaming compression mode, we threw away the result if it was greater than or equal to the input size. But for the one-shot compression, we threw away the compression only if it was greater than the input size! This patch changes the comparison for the simple case so it is consistent with the streaming case. As a few tests demonstrate, this adds 1 byte to some revlog entries. This is because of an added 'u' header on the chunk. It seems somewhat wrong to increase the revlog size here. However, IMO the cost of 1 byte in storage is insignificant compared to the performance gains of avoiding decompression. This patch should invite questions around the heuristic for throwing away compressed data. For example, I'd argue we should be more liberal about rejecting compressed data, additionally doing so where the number of bytes saved fails to reach a threshold. But we can have this discussion another time.

similar: rename local variable to not collide with previous Future patches will move the score function to the module level, so let's not shadow that.

patch: add label for coloring the index extended header Just like the summary says, this will colorize the: index 3d3ba4b65e11..57274a0f46b2 100644 line in the diff output.

patch: add index line for diff output This helps highlighting in third-party diff coloring (which assumes git output) and maintains pedantic correctness with diff --git. Tests will be added at the end of the series.

patch: add config knob for displaying the index header This config knob can take an integer between 0 and 40 or a keyword ('none', 'short', 'full') to control the length of hash to output. It will display diffs with the git index header as such, diff --git a/mercurial/mdiff.py b/mercurial/mdiff.py index 112edf7..d6b52c5 100644 We'll put this in the experimental section for now.

bisect: refer directly to bisect() revset predicate in help We have specific syntax for displaying the help text for a particular revset predicate, so let's refer directly to the bisect() revset in the verbose bisect help. It seems likely that the user doesn't care about other revsets at that point, so they will probably not miss the text about the other revset predicates.

tests: use "hg help revisions.<predicate>" instead of grepping We have specific syntax for displaying the help text for a particular revset predicate, so use that instead of grepping through the full output.

help: remove now-redundant pointer to revsets help "hg help revisions" and "hg help revsets" now point to the same text, so drop the revsets reference.

help: eliminate duplicate text for revset string patterns There's no reason to duplicate this so many times, and it's likely an instance will be missed if support for a new pattern is added and documented. The stringmatcher is mostly used by revsets, though it is also used for the 'tag' related templates, and namespace filtering in the journal extension. So maybe there's a better place to document it. `hg help patterns` seems inappropriate, because that is all file pattern matching. While here, indicate how to perform case insensitive regex searches.

revset: add regular expression support to 'desc' This is a case insensitive predicate like 'author', so it conforms to the existing behavior of performing a case insensitive regex.

revset: stop lowercasing the regex pattern for 'author' It was probably unintentional for regex, as the meaning of some sequences like \S and \s is actually inverted by changing the case. For backward compatibility however, the matching is forced to case insensitive.

repair: clean up stale lock file from store backup Since we did a directory rename on the stores, the source repository's lock path now references the dest repository's lock path and the dest repository's lock path now references a non-existent filename. So releasing the lock on the source will unlock the dest and releasing the lock on the dest will no-op because it fails due to file not found. So we clean up the dest's lock manually.

repair: copy non-revlog store files during upgrade The store contains more than just revlogs. This patch teaches the upgrade code to copy regular files as well. As the test changes demonstrate, the phaseroots file is now copied.

repair: migrate revlogs during upgrade Our next step for in-place upgrade is to migrate store data. Revlogs are the biggest source of data within the store and a store is useless without them, so we implement their migration first. Our strategy for migrating revlogs is to walk the store and call `revlog.clone()` on each revlog. There are some minor complications. Because revlogs have different storage options (e.g. changelog has generaldelta and delta chains disabled), we need to obtain the correct class of revlog so inserted data is encoded properly for its type. Various attempts at implementing progress indicators that didn't lead to frustration from false "it's almost done" indicators were made. I initially used a single progress bar based on number of revlogs. However, this quickly churned through all filelogs, got to 99% then effectively froze at 99.99% when it got to the manifest. So I converted the progress bar to total revision count. This was a little bit better. But the manifest was still significantly slower than filelogs and it took forever to process the last few percent. I then tried both revision/chunk bytes and raw bytes as the denominator. This had the opposite effect: because so much data is in manifests, it would churn through filelogs without showing much progress. When it got to manifests, it would fill in 90+% of the progress bar. I finally gave up having a unified progress bar and instead implemented 3 progress bars: 1 for filelog revisions, 1 for manifest revisions, and 1 for changelog revisions. I added extra messages indicating the total number of revisions of each so users know there are more progress bars coming. I also added extra messages before and after each stage to give extra details about what is happening. Strictly speaking, this isn't necessary. But the numbers are impressive. For example, when converting a non-generaldelta mozilla-central repository, the messages you see are: migrating 2475593 total revisions (1833043 in filelogs, 321156 in manifests, 321394 in changelog) migrating 1.67 GB in store; 2508 GB tracked data migrating 267868 filelogs containing 1833043 revisions (1.09 GB in store; 57.3 GB tracked data) finished migrating 1833043 filelog revisions across 267868 filelogs; change in size: -415776 bytes migrating 1 manifests containing 321156 revisions (518 MB in store; 2451 GB tracked data) That "2508 GB" figure really blew me away. I had no clue that the raw tracked data in mozilla-central was that large. Granted, 2451 GB is in the manifest and "only" 57.3 GB is in filelogs. But still. It's worth noting that gratuitous loading of source revlogs in order to display numbers and progress bars does serve a purpose: it ensures we can open all source revlogs. We don't want to spend several minutes copying revlogs only to encounter a permissions error or similar later. As part of this commit, we also add swapping of the store directory to the upgrade function. After revlogs are converted, we move the old store into the backup directory then move the temporary repo's store into the old store's location. On well-behaved systems, this should be 2 atomic operations and the window of inconsistency show be very narrow. There are still a few improvements to be made to store copying and upgrading. But this commit gets the bulk of the work out of the way.

revlog: add clone method Upcoming patches will introduce functionality for in-place repository/store "upgrades." Copying the contents of a revlog feels sufficiently low-level to warrant being in the revlog class. So this commit implements that functionality. Because full delta recomputation can be *very* expensive (we're talking several hours on the Firefox repository), we support multiple modes of execution with regards to delta (re)use. This will allow repository upgrades to choose the "level" of processing/optimization they wish to perform when converting revlogs. It's not obvious from this commit, but "addrevisioncb" will be used for progress reporting.

repair: begin implementation of in-place upgrading Now that all the upgrade planning work is in place, we can start doing the real work: actually upgrading a repository. The main goal of this commit is to get the "framework" for running in-place upgrade actions in place. Rather than get too clever and low-level with regards to in-place upgrades, our strategy is to create a new, temporary repository, copy data to it, then replace the old data with the new. This allows us to reuse a lot of code in localrepo.py around store interaction, which will eventually consume the bulk of the upgrade code. But we have to start small. This patch implements adding new repository requirements. But it still sets up a temporary repository and locks it and the source repo before performing the requirements file swap. This means all the plumbing is in place to implement store copying in subsequent commits.

repair: determine what upgrade will do This commit introduces code for determining what actions/improvements an upgrade should perform. The "upgradefindimprovements" function introduces a mechanism to return a list of improvements that can be made to a repository. Each improvement is effectively an action that an upgrade will perform. Associated with each of these improvements is metadata that will be used to inform users what's wrong and what an upgrade will do. Each "improvement" is categorized as a "deficiency" or an "optimization." TBH, I'm not thrilled about the terminology and am receptive to constructive bikeshedding. The main difference between a "deficiency" and an "optimization" is a deficiency is always corrected (if it deviates from the current config) and an "optimization" is an optional action that goes above and beyond to improve the state of the repository (usually by requiring more CPU during upgrade). Our initial set of improvements identifies missing repository requirements, a single, easily correctable problem with changelog storage, and a set of "optimizations" related to delta recalculation. The main "upgraderepo" function has been expanded to handle improvements. It queries for the list of improvements and determines which of them will run based on the current repository state and user I went through numerous iterations of the output format before settling on a ReST-inspired definition list format. (I used bulleted lists in the first submission of this commit and could not get it to format just right.) Even with the various iterations, I'm still not super thrilled with the format. But, this is a debug* command, so that should mean we can refine the output without BC concerns.

repair: implement requirements checking for upgrades This commit introduces functionality for upgrading a repository in place. The first part that's implemented is testing for upgrade "compatibility." This is done by examining repository requirements. There are 5 functions returning sets of requirements that control upgrading. Why so many functions? Mainly to support extensions. Functions are easier to monkeypatch than module variables. Astute readers will see that we don't support "manifestv2" and "treemanifest" requirements in the upgrade mechanism. I don't have a great answer for why other than this is a complex set of patches and I don't want to deal with the complexity of these experimental features just yet. We can teach the upgrade mechanism about them later, once the basic upgrade mechanism is in place. This commit also introduces the "upgraderepo" function. This will be our main routine for performing an in-place upgrade. Currently, it just implements requirements checking. The structure of some code in this function may look a bit weird (e.g. the inline function that is only called once). But this will make sense after future commits.

debugcommands: stub for debugupgraderepo command Currently, if Mercurial introduces a new repository/store feature or changes behavior of an existing feature, users must perform an `hg clone` to create a new repository with hopefully the correct/optimal settings. Unfortunately, even `hg clone` may not give the correct results. For example, if you do a local `hg clone`, you may get hardlinks to revlog files that inherit the old state. If you `hg clone` from a remote or `hg clone --pull`, changegroup application may bypass some optimization, such as converting to generaldelta. Optimizing a repository is harder than it seems and requires more than a simple `hg` command invocation. This commit starts the process of changing that. We introduce `hg debugupgraderepo`, a command that performs an in-place upgrade of a repository to use new, optimal features. The command is just a stub right now. Features will be added in subsequent commits. This commit does foreshadow some of the behavior of the new command, notably that it doesn't do anything by default and that it takes arguments that influence what actions it performs. These will be explained more in subsequent commits.

util: teach stringmatcher to handle forced case insensitive matches The 'author' and 'desc' revsets are documented to be case insensitive. Unfortunately, this was implemented in 'author' by forcing the input to lowercase, including for regex like '\B'. (This actually inverts the meaning of the sequence.) For backward compatibility, we will keep that a case insensitive regex, but by using matcher options instead of brute force. This doesn't preclude future hypothetical 'icase-literal:' style prefixes that can be provided by the user. Such user specified cases can probably be handled up front by stripping 'icase-', setting the variable, and letting it drop through the existing code.

revset: point to 'grep' in the 'keyword' help for regex searches The help for 'grep' already points to 'keyword'.