debian: update mailing list address

Added signature for changeset e1526da1e6d8

Added tag 4.1 for changeset e1526da1e6d8

merge with i18n

i18n-pt_BR: synchronized with dfc6663f97ca

merge: more safe detection of criss cross merge conflict between dm and r 41f6af50c0d8 introduced handling of a crash in this case. A review comment suggested that it was not entirely obvious that a 'dm' always would have a 'r' for the source file. To mitigate that risk, make the code more conservative and make less assumptions.

tests: correct (I think) command in test-largefiles-update When this test was introduced, it used the short-form of all the flags on this update invocation. I suspect, based on the "start with clean dirstates" comment and the fact that the no-exec branch of the #if guard leaves dirstate clean, that this should have been 'update -qCr' instead of 'update -qcr', but that a bug in largefiles --check handling left this problem unnoticed. I'll leave a breadcrumb further up about the current failure mode in the hopes that we can fix this some day. This was previously discussed in [0] but the trail in that thread goes cold after a few replies. Given that this is still a flaky test, that appears to only be passing by bad fortune, I think it's worth correcting the code of the test to make a correct assertion, and to keep track of the suspected bug with some other mechanism than an invalid test (if we had support for "expected failure" blocks this might be a worthwhile use of them?). 0: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089501.html

tests: expand flags to long form in test-largefiles-update.t I spent some time confused by this test. I'm pretty sure that this line intends to be cleaning the dirstate, not checking that it's clean before updating: the preceding #if block leaves the dirstate clean in the noexec case, and dirty in the exec case, so we can't expect consistent behavior across that platform variation. A subsequent patch will modify this command to use --clean instead of --check. I'll elaborate in that patch about the hypothetical bug here.

merge: fix crash on criss cross merge with dir move and delete (issue5020) Work around that 'dm' in the data model only can have one operation for the target file, but still can have multiple and conflicting operations on the source file where the other operation is a 'rm'. The move would thus fail with 'abort: No such file or directory'. In this case it is "obvious" that the file should be removed, either before or after moving it. We thus keep the 'rm' of the source file but drop the 'dm'. This is not a pretty fix but quite "obviously" safe (famous last words...) as it only touches a rare code path that used to crash. It is possible that it would be better to swap the files for 'dm' as suggested on https://bz.mercurial-scm.org/show_bug.cgi?id=5020#c13 but it is not entirely obvious that it not just would create conflicts on the other file. That can be revisited later.

tests: use 'f' in test-merge-criss-cross.t to prepare for recursive dumping Prepare for adding a test case with files in a directory.

util: make sortdict.keys() return a copy dict.keys() is documented to return a copy, so it's surprising that sortdict.keys() did not. I noticed this because we have an extension that calls readlocaltags(). That method tries to remove any tags that point to non-existent revisions (most likely stripped). However, since it's unintentionally working on the instance it's modifying, it sometimes fails to remove tags when there are multiple bad tags in a row. This was not caught because localrepo.tags() does an additional layer of filtering. sortdict is also used in other places, but I have not checked whether its keys() and/or __delitem__() methods are used there.

test-highlight: add normalization rule for Pygments 2.2 The test failed on Debian sid because of new class="vm".

tests: account for different newline behavior between Solaris and GNU grep GNU grep, when emitting a matching line that doesn't have a terminating newline, will add an extra newline. Solaris grep passes the original line through without the newline. This causes differences in test output when looking at the last line of the output of get-with-headers.py, which doesn't usually emit (and certainly doesn't guarantee) a terminating newline. Both grep implementations succeed in matching the requested pattern, though, so rely on specifying the full pattern on grep's commandline instead of expecting it in the output, and send the output to /dev/null.

tests: also allow "Protocol not supported" in test-http-proxy error I've seen this in a (misconfigured) FreeBSD jail which has ::1 as an entry for localhost, but IPv6 support is disabled in the jail. It took me months to figure out what was going on (and I only figured it out when tinyproxy.py got confused by similar IPv4-level misconfiguration of the localhost domain in /etc/hosts.) I don't feel strongly about this patch: on the one hand, it's papering over a host-level misconfiguration, but on the other it avoids some weird and hard to diagnose problems that can occur in weirdly restricted environments.

revset: prevent using outgoing() and remote() in hgweb session (BC) outgoing() and remote() may stall for long due to network I/O, which seems unsafe per definition, "whether a predicate is safe for DoS attack." But I'm not 100% sure about this. If our concern isn't elapsed time but CPU resource, these predicates are considered safe. Perhaps that would be up to the web/application server configuration? Anyway, outgoing() and remote() wouldn't be useful in hgweb, so I think it's okay to ban them.

tests: use an absolute path to get around '..' being invalid on a dead CWD Only FreeBSD seems to be this picky. Note that this explicit absolute-path `cd` exposes a defect in the test, in that we end up still inside the cwd-vanish repository, but that's not a regression in this change. Since we're in a code freeze, I'm doing the smallest thing possible to try and fix bugs on FreeBSD, rather than cleaning up the entire problem. I'll follow up with a more complete fix after the freeze.

ui: rename tmpdir parameter to more specific repopath This was requested by Augie and I agree that repopath is more descriptive.

pager: wrap _runcommand() no matter if stdout is redirected We've made chg utilize the common code path implemented in pager.py (by 815e1cefd082 and 493935e0327a), but the chg server does not always start with a tty. Because of this, uisetup() of the pager extension could be skipped on the chg server. Kudos given to Sean Farley for dogfooding new chg and spotting this problem.

shelve: make unshelve not crash when there are missing files (issue4176) This patch makes it possible to unshelve while having missing files in your repo as long as shelved changes don't touch those missing files. It also makes error message better otherwise.

statprof: require input file statprof has a __main__ handler that allows viewing of previously written data files. As Yuya pointed out during review, f42cd5434cc2 broke this. This patch fixes that.

tests: work around FreeBSD's unzip having slightly different output According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's what comes as /usr/bin/unzip, so we may as well cater to it since it's easy.

contrib: fix check-commit to not reject commits from `hg sign` and `hg tag` I'm tired of having a spurious red build every time we do a release. Fix it once and for all.

Added signature for changeset a1dd2c0c479e

Added tag 4.1-rc for changeset a1dd2c0c479e

tests: fix up some http tests for no-zstd case

freeze: merge default into stable for 4.1 code freeze

patchbomb: add tmpdir parameter to ui.edit call

histedit: add tmpdir parameter to ui.edit call

cmdutil: add tmpdir parament to ui.edit calls

ui: add a parameter to set the temporary directory for edit Until callsites are updated, this will have no effect. Once callsites are updated, specifying experimental.editortmpinhg will create editor temporary files in a subdirectory of .hg, which will make it easier for tool integrations to determine what repository is in play when they're asked to edit an hg-related file.

help: update help for `hg update` which was misleading (issue5427)

templater: add '{envvars}' to access environment variables Since the option for ui.exportableenviron is experimental, so is this template until the underlying API is sorted out.

ui: introduce an experimental dict of exportable environment variables Care needs to be taken to prevent leaking potentially sensitive environment variables through hgweb, if template support for environment variables is to be introduced. There are a few ideas about the API for preventing accidental leaking [1]. Option 3 seems best from the POV of not needing to configure anything in the normal case. I couldn't figure out how to do that, so guard it with an experimental option for now. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-January/092383.html

tests: test experimental.spacemovesdown config for commit -i The feature is still very experimental, but at least its behavior is captured in the test.

zstd: prevent potential free() of uninitialized memory This is a cherry pick of an upstream fix. The free() of uninitialed memory could likely only occur if a malloc() inside zstd fails. The patched functions aren't currently used by Mercurial. But I don't like leaving footguns sitting around.

revlog: give EXTSTORED flag value to narrowhg Narrowhg has been using "1 << 14" as its revlog flag value for a long time. We (Google) have many repos with that value in production already. When the same value was reserved for EXTSTORED, it made those repos invalid. Upgrading them will be a little painful. We should clearly have reserved the value for narrowhg a long time ago. Since the EXTSTORED flag is not yet in any release and Facebook also says they have not started using it in production, so it should be okay to change it. This patch gives the current value (1 << 14) back to narrowhg and gives a new value (1 << 13) to EXTSTORED.

help: don't let tools reflow revlog flags list Before this change, the text about revlog flags was reflowed into a single paragraph, which made it a bit hard to read. I don't even know the rules around this, but adding a blank line before each flag seems to prevent the reflowing.

help: format revlog.txt more closely to result The rendered text has spaces before each item in the list

hgweb: simplify calculation of first revision in filelog command

hgweb: restore ascending iteration on revs in filelog web command Follow-up on 96f811bceb85. Adjust back the "parity" generator's offset to keep rendering the same.

context: extract _changesinrange() out of blockancestors() We'll need it to write a blockdescendants function in next changeset.

shelve: allow multiple shelves with --patch and --stat Before this patch, there was a single way to see multiple shelves using `--patch --list` which show all the shelves. Doing `--patch s1 s2` returns an error. This patch allows to show multiple shelves using `--patch` and `--stat`.

zstd: vendor python-zstandard 0.6.0 Commit 63c68d6f5fc8de4afd9bde81b13b537beb4e47e8 from https://github.com/indygreg/python-zstandard is imported without modifications (other than removing unwanted files). This includes minor performance and feature improvements. It also changes the vendored zstd library from 1.1.1 to 1.1.2. # no-check-commit

util: add length argument to util.buffer() util.buffer() either returns inbuilt buffer function or defines a new one which slices. The inbuilt buffer() also has a length argument which is missing from the ones we defined. This patch adds that length argument.

py3: replace pycompat.getenv with encoding.environ.get pycompat.getenv returns os.getenvb on py3 which is not available on Windows. This patch replaces them with encoding.environ.get and checks to ensure no new instances of os.getenv or os.setenv are introduced.

patch: check length of git index header only if integer is specified Otherwise TypeError would be raised. Follows up d1901c4c8ec0.

localrepo: experimental support for non-zlib revlog compression The final part of integrating the compression manager APIs into revlog storage is the plumbing for repositories to advertise they are using non-zlib storage and for revlogs to instantiate a non-zlib compression engine. The main intent of the compression manager work was to zstd all of the things. Adding zstd to revlogs has proved to be more involved than other places because revlogs are... special. Very small inputs and the use of delta chains (which are themselves a form of compression) are a completely different use case from streaming compression, which bundles and the wire protocol employ. I've conducted numerous experiments with zstd in revlogs and have yet to formalize compression settings and a storage architecture that I'm confident I won't regret later. In other words, I'm not yet ready to commit to a new mechanism for using zstd - or any other compression format - in revlogs. That being said, having some support for zstd (and other compression formats) in revlogs in core is beneficial. It can allow others to conduct experiments. This patch introduces *highly experimental* support for non-zlib compression formats in revlogs. Introduced is a config option to control which compression engine to use. Also introduced is a namespace of "exp-compression-*" requirements to denote support for non-zlib compression in revlogs. I've prefixed the namespace with "exp-" (short for "experimental") because I'm not confident of the requirements "schema" and in no way want to give the illusion of supporting these requirements in the future. I fully intend to drop support for these requirements once we figure out what we're doing with zstd in revlogs. A good portion of the patch is teaching the requirements system about registered compression engines and passing the requested compression engine as an opener option so revlogs can instantiate the proper compression engine for new operations. That's a verbose way of saying "we can now use zstd in revlogs!" On an `hg pull` conversion of the mozilla-unified repo with no extra redelta settings (like aggressivemergedeltas), we can see the impact of zstd vs zlib in revlogs: $ hg perfrevlogchunks -c ! chunk ! wall 2.032052 comb 2.040000 user 1.990000 sys 0.050000 (best of 5) ! wall 1.866360 comb 1.860000 user 1.820000 sys 0.040000 (best of 6) ! chunk batch ! wall 1.877261 comb 1.870000 user 1.860000 sys 0.010000 (best of 6) ! wall 1.705410 comb 1.710000 user 1.690000 sys 0.020000 (best of 6) $ hg perfrevlogchunks -m ! chunk ! wall 2.721427 comb 2.720000 user 2.640000 sys 0.080000 (best of 4) ! wall 2.035076 comb 2.030000 user 1.950000 sys 0.080000 (best of 5) ! chunk batch ! wall 2.614561 comb 2.620000 user 2.580000 sys 0.040000 (best of 4) ! wall 1.910252 comb 1.910000 user 1.880000 sys 0.030000 (best of 6) $ hg perfrevlog -c -d 1 ! wall 4.812885 comb 4.820000 user 4.800000 sys 0.020000 (best of 3) ! wall 4.699621 comb 4.710000 user 4.700000 sys 0.010000 (best of 3) $ hg perfrevlog -m -d 1000 ! wall 34.252800 comb 34.250000 user 33.730000 sys 0.520000 (best of 3) ! wall 24.094999 comb 24.090000 user 23.320000 sys 0.770000 (best of 3) Only modest wins for the changelog. But manifest reading is significantly faster. What's going on? One reason might be data volume. zstd decompresses faster. So given more bytes, it will put more distance between it and zlib. Another reason is size. In the current design, zstd revlogs are *larger*: debugcreatestreamclonebundle (size in bytes) zlib: 1,638,852,492 zstd: 1,680,601,332 I haven't investigated this fully, but I reckon a significant cause of larger revlogs is that the zstd frame/header has more bytes than zlib's. For very small inputs or data that doesn't compress well, we'll tend to store more uncompressed chunks than with zlib (because the compressed size isn't smaller than original). This will make revlog reading faster because it is doing less decompression. Moving on to bundle performance: $ hg bundle -a -t none-v2 (total CPU time) zlib: 102.79s zstd: 97.75s So, marginal CPU decrease for reading all chunks in all revlogs (this is somewhat disappointing). $ hg bundle -a -t <engine>-v2 (total CPU time) zlib: 191.59s zstd: 115.36s This last test effectively measures the difference between zlib->zlib and zstd->zstd for revlogs to bundle. This is a rough approximation of what a server does during `hg clone`. There are some promising results for zstd. But not enough for me to feel comfortable advertising it to users. We'll get there...

revlog: use compression engine APIs for decompression Now that compression engines declare their header in revlog chunks and can decompress revlog chunks, we refactor revlog.decompress() to use them. Making full use of the property that revlog compressor objects are reusable, revlog instances now maintain a dict mapping an engine's revlog header to a compressor object. This is not only a performance optimization for engines where compressor object reuse can result in better performance, but it also serves as a cache of header values so we don't need to perform redundant lookups against the compression engine manager. (Yes, I measured and the overhead of a function call versus a dict lookup was observed.) Replacing the previous inline lookup table with a dict lookup was measured to make chunk reading ~2.5% slower on changelogs and ~4.5% slower on manifests. So, the inline lookup table has been mostly preserved so we don't lose performance. This is unfortunate. But many decompression operations complete in microseconds, so Python attribute lookup, dict lookup, and function calls do matter. The impact of this change on mozilla-unified is as follows: $ hg perfrevlogchunks -c ! chunk ! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6) ! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6) ! chunk batch ! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6) ! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6) $ hg perfrevlogchunks -m ! chunk ! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4) ! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4) ! chunk batch ! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5) ! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4) Changelog chunk reading is slightly faster but manifest reading is slower. What gives? On this repo, 99.85% of changelog entries are zlib compressed (the 'x' header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch swapped the test order of 'x' and '\0' so now 'x' is tested first. This makes changelogs faster since they almost always hit the first branch. This makes a significant percentage of manifest '\0' chunks slower because that code path now performs an extra test. Yes, I too can't believe we're able to measure the impact of an if..elif with simple string compares. I reckon this code would benefit from being written in C...