fuzz: exercise more of the lazymanifest code Differential Revision: https://phab.mercurial-scm.org/D5643

fuzz: restrict manifest input size Again, let's keep the fuzzer from getting excited about huge inputs. Differential Revision: https://phab.mercurial-scm.org/D5642

fuzz: exercise more of the revlog API I noticed in the coverage report that we didn't have much coverage in revlog.py. Let's try and get some of the more interesting bits tested by the fuzzer. I ran this locally for a few minutes to verify that I appear to be calling the various functions in reasonable ways. Differential Revision: https://phab.mercurial-scm.org/D5641

fuzz: don't allow enormous revlog inputs either I'm about to make the fuzzer do more, and without this it was getting enthusiastic about large (and therefore slow) inputs that I don't think buy us much. Differential Revision: https://phab.mercurial-scm.org/D5640

merge with stable

Added signature for changeset 4ea21df312ec

Added tag 4.9.1 for changeset 4ea21df312ec

record: prevent commits that don't pick up dirty subrepo changes (issue6102) This path covers interactive mode for commit, amend, and shelve, as well as the deprecated record extension. Since shelf creation uses commit without -S in the non-interactive case, aborting here should be OK. (I didn't check what happens to non interactive shelve creation if `ui.commitsubrepos=True` is set.) subrepoutil.precommit() will abort on a dirty subrepo if the config option isn't set, but the hint recommends using --subrepos to commit. Since only the commit command currently supports that option, the error has to be raised here to omit the hint. Doing the check before asking about all of the hunks in the MQ test seems like an improvement on its own. There's probably an additional check on this path that can be removed.

test-https: add some more known failure messages of client certs (issue6030) I don't think the exact error message is important here. On Debian sid, ECONNRESET is raised, and "[SSL] tlsv13 alert certificate required" on NetBSD.

test-https: turn off system OpenSSL configuration This mostly fixes the test failure on Debian sid where TLS 1.0 and 1.1 are disabled by default. https://sources.debian.org/patches/openssl/1.1.1a-1/Set-systemwide-default-settings-for-libssl-users.patch/ $OPENSSL_CONF could be set by run-tests.py, but the other tests should work without a "legacy" TLS, so I decided to not.

rebase: add missing dashes in help text

test: stabilize bundle generation for test-sparse-revlog.t To reduce the instability in the bundle binary content, we force it to contains delta against p1 in all cases. In the previous changeset, we already stabilized the processing of the bundle. So we don't see any output change in the test itself.

test: don't trust delta bases from the bundle in test-sparse-revlog.t The point of the test is to check the strategy sparse-revlog uses to pick delta base. If we trust the bases used in the bundle, we no longer fully test this logic. In order to force this computation we have to use the side effect of a legacy format configuration `format.generaldelta`. The lack of a more official way to do so will be fixed in a later changeset.

test: update test-sparse-revlog.t output This test is skipped unless a large artefact is pre-build. It seems like nobody ran it in a while. Changeset 3764330f76a6 changed the expected output but nobody noticed. This changeset focus on the first and simpler step: putting the expected output back to what one would get by running this test. However this test changes highlight a couple of deeper issues: 1) Even if the revision content did not changed, a change in the delta contained in the bundle affected the delta stored in the final revlog, changing the test result. Since we are testing the delta computation strategy with sparse, we should not blindly reuse the delta-base from the bundled delta. 2) A change in the format of the repository used to generate the bundle changed the delta stored in the bundle. We should get a more stable output to avoid future instabilities of this test. 3) The test is it not run by CI or developer. We'll try to address all this issues in the coming changesets.

tests: add a missing abort exit code on Windows in test-audit-subrepo.t

subrepo: add test for Windows relative-ish path with drive letter Matt Harbison pointed out that Windows had some weird path syntax. Fortunately it's rejected appropriately by pathauditor, so we're safe. Let's test the behavior as we have a special handling for Windows drive letters. This patch includes a basic example. Maybe we'll need to extend the test case further, but writing such tests on Linux isn't easy.

tests: fix regression tests failing on CentOS 7 Differential Revision: https://phab.mercurial-scm.org/D5877

packaging: modify rc detection to work with X.Yrc instead of X.Y-rc rc detection on CentOS failed without this change, resulting in upgrades from 4.9rc to 4.9 not working (4.9rc was considered more recent than 4.9). Differential Revision: https://phab.mercurial-scm.org/D5876

subrepo: avoid false unsafe path detection on Windows Subrepo paths are not normalized for the OS, so what was happening in the subsequent root path check was: root -> $TESTTMP\issue1852a\sub/repo util.expandpath(...) -> $TESTTMP\issue1852a\sub/repo os.path.realpath(...) -> $TESTTMP\issue1852a\sub\repo

Added signature for changeset 83377b4b4ae0

Added tag 4.9 for changeset 83377b4b4ae0

subrepo: reject potentially unsafe subrepo paths (BC) (SEC) In addition to the previous patch, this prohibits '~', '$nonexistent', etc. for any subrepo types. I think this is safer, and real-world subrepos wouldn't use such (local) paths.

subrepo: prohibit variable expansion on creation of hg subrepo (SEC) It's probably wrong to expand path at localrepo.*repository() layer, but fixing the layering issue would require careful inspection of call paths. So, this patch adds add a validation to the subrepo constructor. os.path.realpath(util.expandpath(root)) is what vfsmod.vfs() would do.

subrepo: extend path auditing test to include more weird patterns (SEC) While reviewing patches for the issue 5739, "$foo in repository path expanded", I realized that subrepo paths can also be cheated. This patch includes various subrepo paths which are potentially unsafe. Since an expanded subrepo path isn't audited, this bug allows symlink check bypass. As a result, a malicious subrepository could be checked out to a sub tree of e.g. $HOME directory. The good news is that the destination directory must be empty or nonexistent, so the existing ~/.bashrc wouldn't be overwritten. See the last part of the tests for details.

copyright: update to 2019 Differential Revision: https://phab.mercurial-scm.org/D5779

revlog: make sure we never use sparserevlog without general delta (issue6056) We are getting user report where the delta code tries to use `sparse-revlog` logic on repository where `generaldelta` is disabled. This can't work so we ensure the two booleans have a consistent value. Creating this kind of repository is not expected to be possible the current bug report point at a clonebundle related bug that is still to be properly isolated (Yuya Nishihara seems to a have done it). Corrupting a repository to reproduce the issue is possible. A test using this method is included in this fix.

sparserevlog: document the config option This was overlooked when this graduated from experimental.

rust-cpython: raising error.WdirUnsupported The Graph implementation of hg-cpython returns the appropriate error upon encounter with the working directory special revision number, and this gives us in particular a code path to test from test-rust-ancestors.py In the current implementation, the exception is actually raised from the iterator instantiation; we are nonetheless consuming the iterator in the test with `list()` in order not to depend on implementation details.

rust: error for WdirUnsupported with cpython conversion as exception This introduces WorkingDirectoryUnsupported as an enum variant of GraphError in the core and converts it to the expected `mercurial.error.WdirUnsupported`.

rust: working directory revision number constant This introduces the constant, but does not use it anywhere yet.

ui: remove unreachable branches and function calls from write() (issue6059) This is at least faster than ui.write() of 4.8.2. $ HGRCPATH=/dev/null hg files -R mozilla-central --time >/dev/null 4.8.2: time: real 2.340 secs (user 2.310+0.000 sys 0.020+0.000) 4.9rc0: time: real 2.580 secs (user 2.550+0.000 sys 0.020+0.000) this: time: real 2.230 secs (user 2.210+0.000 sys 0.020+0.000) Maybe the formatter should own a resolved write() function because it will just call dest.write(msg) most of the time, but that would be too much for stable.

ui: inline _writenobuf() into write() due to performance issue I'll remove redundant conditions later in this series.

ui: inline _write() into write() due to performance issue I'll remove redundant conditions later in this series.

ui: optimize buffered write with no label This was spotted while making fastannotate faster again after ditching its own formatter. Since I'm going to inline _write() into ui.write(), I decided to include this patch in this series. Here, the cost of '(self.label(a, label) for a in args)' was significant in hot loops.

partialdiscovery: avoid `undecided` related computation sooner than necessary Changeset 1d30be90c move the update of the `undecided` set within the `partialdiscovery` object in order to clarify the API. The update to the `undecided` set was unconditional in 1d30be90c and the first access to the `self.undecided` property triggered the initial computation of the set of undecided revisions. As a result, the set was computed much earlier, at a time where less information is available, immediately followed by an update of this set to remove common revisions. To fix this regression, we ignore the `undecided` related logic in `addcommons` when that `undecided` set has not been computed yet. Code that actually needs to know the `undecided` set will trigger its computation later. The change has no effects on semantic because the initial computation `undecided` set takes all knowns `common` into account. Example performance running `hg debugdiscovery` from a pypy repo missing 10 changesets: 870a89c6909d: 52.3ms (regression parent) 1d30be90c9dc: 72.0ms (regression) 5a5f504a7175: 64.8ms (this fix parent) this fix: 52.6ms

revlog: fix resolution of revlog version 0 This partially backs out cecf3f8bccd3, "revlog: always process opener options." My understanding is that if there's no "revlog1" nor "revlog2" in .hg/requires, the repository should stick to the v0 format. The reasoning is briefly described in 31a5973fcf96, "revlog: get rid of defversion." Maybe we can drop support for missing opener options, but I didn't do that in this patch.

rust: add comment about lack of wdirrev handling If hg is compiled with rust support, 'only(wdir())' crashed as "rustext.GraphError: ('ParentOutOfRange', 2147483647)", which should instead say "abort: working directory revision cannot be specified."

templatekw: fix crash on multiple latesttags resolution at wdir (issue6055) It appears not easy to fix only() to support wdir(), so this patch works around the issue by getlatesttags(). The "+1" after len(changes) doesn't matter since ctx never changes while sorting. It's just for clarity.

test-template-keywords: add test for {latesttag} of wdir() revision It's probably broken since fb672eac2702, "templatekw: choose {latesttag} by len(changes), not date (issue5659)". only() doesn't support wdir.

narrow: fix crash when restoring backup in legacy repo Using --addremove when committing in an old repo (before we started keeping .hg/narrowspec.dirstate) results in a crash. The test case modified in this patch would crash like this: abort: $ENOENT$ The issue is that when the dirstateguard is aborted, it tries to restore the backup of .hg/narrowspec.dirstate. However, since we were in an old repo, that file did not get created when the dirstateguard was created. Note that the dirstateguard is not used unless --addremove is passed. This patch fixes the bug by making restorewcbackup() not fail if the backup doesn't exist. I also made clearwcbackup() safe, just in case. Differential Revision: https://phab.mercurial-scm.org/D5634

revset: introduce an internal `_rev` predicate for '%d' usage In 24a1f67bb75a, we aligned "%d" behavior on "%ld" one, invalid revisions got silently ignored. However, soon after in 8aca89a694d4 and 26b0a7514f01, a side effect changed the behavior of "%ld" to no longer silently filter invalid revisions. After discussion on the mailing list, it was decided to align on the new %ld behavior: https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-January/127291.html This changeset introduce a '_rev()' predicated that keep the benefit from 24a1f67bb75a while enforcing a more strict checking on the inputs.

mmap: backed out changeset 875d2af8cb4e There have been concrete and theoretical issues raised, this will need more work during the next cycle.

mmap: backed out changeset 74a9f428227e There have been concrete and theoretical issues raised, this will need more work during the next cycle.

help: document the minimumhgversion variable for extensions

help: modernize the example for command registration

Added signature for changeset 593718ff5844

Added tag 4.9rc0 for changeset 593718ff5844

merge default into stable for 4.9 release

update: fix edge-case with update.atomic-file and read-only files We used to create the tempfile with the original file mode. That means creating a read-only tempfile when the original file is read-only, which crash if we need to write on the tempfile. The file in the working directory ends up being writable with and without the atomic update config, so the behavior is the same.

scmutil: drop unreachable except clause socket.error is a subclass of IOError, which we catch higher up. It seems to have been this way since 020a896a5292 (dispatch: sort exception handlers, 2009-01-12), so let's celebrate the 10 year anniversary (a few days late) of it being wrong by deleting it. Differential Revision: https://phab.mercurial-scm.org/D5626

tests: suppress "Checked out 1 paths of <hash>" from modern git test-convert-git.t is failiing since git commit 0f086e6dca (checkout: print something when checking out paths, 2018-11-13). Suppress the new output by disambiguating the arguments with a "--" separator. Differential Revision: https://phab.mercurial-scm.org/D5625

revlog: document that mmap resources are released implicitly by GC It's okay-ish, but currently the open fd and the mapping itself are leaked until the indexdata is deallocated. If revlog had close(), the underlying resources should be closed there as well, but AFAIK there's no such hook point.

ui: proxy protect/restorestdio() calls to update internal flag It should be better to manage the redirection flag solely by the ui class.

ui: move protectedstdio() context manager from procutil This is a follow-up series for 23a00bc90a3c, "chgserver: do not send system() back to client if stdio redirected." The function is renamed using ui terms.

cext: clang-format new code coming from stable branch

match: support rooted globs in hgignore In a .hgignore, "glob:foo" always means "**/foo". This cannot be avoided because there is no syntax like "^" in regexes to say you don't want the implied "**/" (of course one can use regexes, but glob syntax is nice). When you have a long list of fairly specific globs like path/to/some/thing, this has two consequences: 1. unintended files may be ignored (not too common though) 2. matching performance can suffer significantly Here is vanilla hg status timing on a private repository: Using syntax:glob everywhere real 0m2.199s user 0m1.545s sys 0m0.619s When rooting the appropriate globs real 0m1.434s user 0m0.847s sys 0m0.565s (tangentially, none of this shows up in --profile's output. It seems that C code doesn't play well with profiling) The code already supports this but there is no syntax to make use of it, so it seems reasonable to create such syntax. I create a new hgignore syntax "rootglob". Differential Revision: https://phab.mercurial-scm.org/D5493

resolve: fix mark-check when a file was deleted on one side (issue6020) wvfs.open raises an error if one of the files does not exist. Ignoring the error if it's ENOENT is done in several other places in this code, so I'm in good company :) Differential Revision: https://phab.mercurial-scm.org/D5243

discovery: compute newly discovered missing in a more efficient way Calling "descendants" is expensive, instead, we bound the walk inside the know set of undecided revision. This help with discovery performance: # without the revset '%ld' improvement $ hg perfdiscovery -R pypy-left pypy-right before: wall 0.675631 comb 0.680000 user 0.670000 sys 0.010000 (median of 15) after: wall 0.520145 comb 0.530000 user 0.510000 sys 0.020000 (median of 19) There is another series in flight that greatly improves performances of "%ld" substitution in `repo.revs` call. If this changeset is applied above it, we see a similar performance boost. # with the revset '%ld' improvement $ hg perfdiscovery -R pypy-left pypy-right before: wall 0.477848 comb 0.480000 user 0.480000 sys 0.000000 (median of 22) after: wall 0.404163 comb 0.400000 user 0.400000 sys 0.000000 (median of 24)

exthelper: drop the addattr() decorator Yuya pointed out that this goes against the typical advice to not add attributes to classes[1]. The evolve extension still uses this a handful of times, so maybe it should be brought back in the future if a general use is found. But it isn't nice to have a new helper API that can lead to easy problems. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-December/126330.html

revsetbenchmark: add more example for roots usages We test the `roots` revset in setting similar to our test for `heads`. Note that the algorithm used for roots can give result without consuming the full input set. This provides a significant speedup when testing or accessing a single value. We can't just replace it with simple, full algorithm like we did for `heads`. See performance number below: 0) roots((tip~100::) - (tip~100::tip)) 1) roots((0::) - (0::tip)) 2) roots(tip~100:) 3) roots(:42) 4) roots(not public()) 5) roots((0:tip)::) 6) roots(0::tip) 7) 42:68 and roots(42:tip) 8) roots(0:tip) 9) roots((:42) + (tip~42:)) 10) roots(all()) 11) roots(-10000:-1) 12) (-5000:-1000) and roots(-10000:-1) 13) roots(matching(tip, "author")) 14) roots(matching(tip, "author")) and -10000:-1 15) (-10000:-1) and roots(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 00) 0.000789 0.000801 0.000801 0.000819 0.000784 0.000774 0.000793 0.000816 0.000815 0.000831 0.000799 01) 0.097610 0.002717 0.096706 0.002615 0.059189 0.089033 0.059862 0.002644 0.098058 0.002640 0.058992 02) 0.000709 0.000117 0.000382 0.000136 0.000384 0.000724 0.000412 0.000133 0.000733 0.000159 0.000416 03) 0.000075 0.000064 0.000093 0.000080 0.000097 0.000089 0.000123 0.000079 0.000105 0.000102 0.000126 04) 0.000055 0.000071 0.000070 0.000087 0.000075 0.000066 0.000100 0.000085 0.000082 0.000110 0.000102 05) 0.088043 0.001084 0.087816 0.001097 0.048049 0.072454 0.047673 0.001089 0.088491 0.001163 0.047824 06) 0.058761 0.001727 0.059324 0.001850 0.058562 0.059198 0.058998 0.001743 0.058556 0.001874 0.059420 07) 0.000131 0.000121 0.000145 0.000138 0.000150 0.000142 0.000178 0.000135 0.000160 0.000163 0.000179 08) 0.058003 0.000077 0.032327 0.000093 0.031966 0.056812 0.031753 0.000092 0.057113 0.000116 0.031933 09) 0.000503 0.000145 0.000469 0.000161 0.000476 0.000564 0.000502 0.000160 0.000537 0.000187 0.000500 10) 0.056654 0.000058 0.033104 0.000073 0.032157 0.056598 0.031877 0.000071 0.056433 0.000094 0.031819 11) 0.005842 0.000081 0.001907 0.000101 0.001883 0.005868 0.001915 0.000099 0.005836 0.000122 0.001896 12) 0.003237 0.000634 0.001784 0.000655 0.001803 0.003245 0.001837 0.000649 0.003231 0.000680 0.001858