tests: use NO_CHECK_EOF as heredoc limit mark to omit checking code fragments This patch uses NO_CHECK_EOF as heredoc limit mark instead of EOF, in order to avoid checking all python code fragments in test-contrib-check-code.t, because almost all of them has un-recommended implementations intentionally.

contrib: add an utility module to parse test scripts This patch centralizes the logic to pick up code fragments embedded in *.t script, in order to: - apply checking with patterns in check-code.py on such embedded code fragments Now, check-code.py completely ignores embedded code fragments. I'll post another patch series to check them. - replace similar code path in contrib/import-checker.py Current import-checker.py has problems below. Fixing each of them is a little difficult, because parsing logic and pattern strings are tightly coupled. - overlook (or mis-detect) the end of inline script in doctest style 8a8dd6e4a97a fixed a part of this issue, but not enough. - it overlooks inline script in doctest style at the end of file (and ignores invalid un-closed heredoc at the end of file, too) - it overlooks code fragment in styles below - "python <<EOF" (heredoc should be "cat > file <<EOF" style) - "cat > foobar.py << ANYLIMIT" (limit mark should be "EOF") - "cat << EOF > foobar.py" (filename should be placed before limit mark) - "cat >> foobar.py << EOF" (appending is ignored) - it is not extensible for other than python code fragments (e.g. shell script, hgrc file, and so on) This new module can detect python code fragments in styles below: - inline script in doctest style (starting by " >>> " line) - python invocation with heredoc script ("python <<EOF") - python script in heredoc style (redirected into ".py" file) As an example of extensibility of new module, this patch also contains implementation to pick up code fragment below. This will be useful to add additional restriction for them, for example. - shell script in heredoc style (redirected into ".sh" file) - hgrc configuration in heredoc style (redirected into hgrc or $HGRCPATH)

tests: use environment variable indirectly Using environment variable directly in heredoc python code will cause syntax error at checking module importation by import-checker.py strictly, because "$varname" is invalid in Python syntax. "$varname" becomes valid after environment variable substitution by shell at writing text into file. Current import-checker.py overlooks code fragment changed in this patch, because of a restriction below for a line starting code fragment. - filename must be specified before limit mark NG: cat <<EOF > FILE.py OK: cat > FILE.py <<EOF import-checker.py itself is fixed in subsequent patch.

tests: import multiple modules separately Current import-checker.py overlooks code fragment changed in this patch, because of restrictions below for a line starting code fragment. - filename must be specified before limit mark NG: cat <<EOF > FILE.py OK: cat > FILE.py <<EOF - limit mark must not be quoted NG: cat > FILE.py <<'EOF' OK: cat > FILE.py <<EOF import-checker.py itself is fixed in subsequent patch.

fuzz: allow manifest fuzzer to detect leaks Huzzah! Differential Revision: https://phab.mercurial-scm.org/D4907

fuzzers: init Python in LLVMFuzzerInitialize and intentionally leak it This sidesteps leaks (or "leaks", I'm not sure) in CPython, and lets our fuzzer work. Differential Revision: https://phab.mercurial-scm.org/D4906

revlog: if the module is initialized more than once, don't leak nullentry Caught (annoyingly) by the manifest fuzzer. Differential Revision: https://phab.mercurial-scm.org/D4905

narrow: move remaining narrow-limited dirstate walks to core In most places we now filter at a higher level (the context object), but there are few places that relied on the dirstate walk to be filtered by the narrowspec. The important cases are those used by `hg add` and `hg addremove`. This patch updates them to pass in a matcher instead of relying on the dirstate to do the filtering. The dirstate filtering is also dropped in narrowdirstate.py. Not always filtering in the dirstate should be useful for a future `hg status --include-outside-narrow` option. These places now end up doing an unrestricted dirstate walk after this patch: * debugfileset * perfwalk * sparse (but restricted to sparse config) * largefiles I'll let anyone who cares about these cases adapt them to work with narrow if necessary. Differential Revision: https://phab.mercurial-scm.org/D4901

narrow: allow repo.narrowmatch(match) to include exact matches from "match" Differential Revision: https://phab.mercurial-scm.org/D4900

narrow: filter files by narrowspec in ctx.matches() This has no effect yet because 1) for committed changes, ctx.matches() just calls ctx.walk(), which we updated in the previous patch, and 2) for the working copy, the filtering is also done in the overridden dirstate.walk() in narrowdirstate. Differential Revision: https://phab.mercurial-scm.org/D4899

narrow: only walk files within narrowspec also for committed revisions Narrow has been walking only paths matching the narrowspec when walking the working copy. We have not done the same filtering when walking committed revisions (e.g. "hg files -r "), which seems a little odd. Let's make it consistent. Differential Revision: https://phab.mercurial-scm.org/D4898

status: intersect matcher with narrow matcher instead of filtering afterwards I seem to have done a very naive move of the code from the narrow extension into core in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). It seems obvious that a better way is to intersect the matchers. Note that this means that when requesting status for the working directory in a narrow repo, we now pass the narrow matcher (possibly intersected with a user-provided matcher) into _buildstatus() and then into dirstate.status() and dirstate.walk(), which will the intersect it again with the narrow matcher. That's functionally fine, but wasteful. I hope to later remove the dirstate wrapping that adds the second layer of matcher intersection. Differential Revision: https://phab.mercurial-scm.org/D4897

localrepo: allow narrowmatch() to accept matcher to intersect with It's pretty common that we need to intersect a matcher we already have (usually from the user) with the narrow matcher. Let's make repo.narrowmatch() take an optional matcher to intersect with. Differential Revision: https://phab.mercurial-scm.org/D4896

obsolete: fix ValueError when stored note contains ':' char (issue5783) The newer version of `amend -n 'Some some'` accepts containing ':' char. The information contained in this note 'Testing::Obstore' gives ValueError, because we are trying to store more than 2 values in key and value. Differential Revision: https://phab.mercurial-scm.org/D4883 Differential Revision: https://phab.mercurial-scm.org/D4882

narrow: update TODO.rst now that we share format with sparse The narrowspec format was unified with the sparse format in f64ebe7d2259 (narrowspec: use sparse.parseconfig() to parse narrowspec file (BC), 2018-08-03). Differential Revision: https://phab.mercurial-scm.org/D4904

narrow: update TODO.rst now that we filter status in ctx The comment referred to was addressed in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). I also think 84092edd5c88 (narrow: drop unnecessary overrides of patch, 2018-09-28) suggests that it was the right thing to do. Differential Revision: https://phab.mercurial-scm.org/D4903

narrow: update TODO.rst now that the narrowspec is in .hg/store We no longer have the unfortunate wrappostshare() and unsharenarrowspec() since 576eef1ab43d (narrow: move .hg/narrowspec to .hg/store/narrowspec (BC), 2018-08-02). Differential Revision: https://phab.mercurial-scm.org/D4902

py3: add 8 new passing tests to whitelist found by buildbot We are getting close! Differential Revision: https://phab.mercurial-scm.org/D4893

py3: use '%f' for floats instead of '%s' I remember Yuya saying we need to use bytestr() or '%r' because '%s' was clever. Not sure it applies to this or not. Differential Revision: https://phab.mercurial-scm.org/D4894

narrow: move adding of narrow server capabilities to core We use the experimental.narrow config option introduced in one of the previous patch and move the functionality of adding narrow server capabilities to core. Differential Revision: https://phab.mercurial-scm.org/D4891

wireprotoserver: move narrow capabilities to wireprototypes.py This is done because wireprotoserver import wireprotov1server, so you cannot import wireprotoserver in wireprotov1server to use the capabilities constants. Differential Revision: https://phab.mercurial-scm.org/D4890

narrow: introduce a config option to check if narrow is enabled or not This patch introduces a new config option experimental.narrow which is set to False by default and set to True by the narrow extension. While moving narrow related logic into core, we need to know at places whether narrow extension is enabled or not. Checking the list of extension enabled is one solution but once narrow is inbuilt, we will definitely want a config option to check whether narrow is turned on or not. So this patch introduces a config option, which will evolve to the main point to turn narrow capability on and off once all the narrow is in core. Differential Revision: https://phab.mercurial-scm.org/D4889

narrow: move the code to generate a widening bundle2 to core This is a part of moving more narrow related bits to core. Differential Revision: https://phab.mercurial-scm.org/D4888

narrow: start returning bundle2 from widen_bundle() Differential Revision: https://phab.mercurial-scm.org/D4838