test-subrepo: glob away an unstable hash This is the instability mentioned at the beginning of the series. I don't like hiding it, but I don't want to sit on a fix for a user reported problem while trying to figure this out. The instability seems related to the cset with a .hgsub with a remote URL. (There's very little existing remote URL subrepo testing.)

subrepo: activate clone pooling to enable sharing with remote URLs This is the easiest way to ensure that repositories with remote subrepo references can share the subrepos, consistent with how local subrepos can be shared.

subrepo: don't attempt to share remote sources (issue5793) Untangling _abssource() to resolve the new subrepo relative to the shared parent's share path, and then either sharing from there (if it exists), or cloning to that location and then sharing, is probably more than should be attempted on stable. Absolute subrepo references are discouraged, so for now, this resumes the behavior prior to 68e0bcb90357 of cloning the absolute subrepo locally.

test-subrepo: demonstrate problems with subrepo sharing and absolute paths This affects remote paths in .hgsub, as well as clone pooling from a remote source. For reasons unknown, there are stability issues with the relative-path.t tests. If run as a single test, it is stable. If run with --loop, or with -jX for X>1, the hash of the parent repo changes. I'm seeing this on both Windows and Fedora 26. I added an `hg log --debug`, and the manifest hash changes, but I have no idea why.

revsetlang: add a hint for more useful parse errors This logic is largely based on the similar logic added to template error messages in D2608 and D2609, but with a few tweaks based on how revsets actually work. Differential Revision: https://phab.mercurial-scm.org/D2619

setup: ignore extension load failures when finding working hg Previously, `make local` would fail if any extension was not properly loading. Differential Revision: https://phab.mercurial-scm.org/D2589

profile: colorize output on Windows Previously, the ANSI codes were printed to the screen, throwing off the alignment. We could probably do this unconditionally, but it's kind of a hack, so I figured I'd limit the scope.

dispatch: don't clamp the range of the exit code twice We already limit the range to (0, 255) in the call to sys.exit(). The duplicated operation can't possibly be hurting us, but let's clean it up to avoid confusion.

bdiff: avoid pointer arithmetic on void* MSVC 2008 complains: mercurial/cext/bdiff.c(106) : error C2036: 'void *' : unknown size mercurial/cext/bdiff.c(107) : error C2036: 'void *' : unknown size Maybe it's a gcc extension? https://stackoverflow.com/questions/37460579/error-c2036-void-unknown-size

fuzz: add a quick README to try and document how to test new fuzzers Differential Revision: https://phab.mercurial-scm.org/D2633

fuzz: add a fuzzer for xdiff Based entirely on the fuzzer for bdiff. Differential Revision: https://phab.mercurial-scm.org/D2632

tests: add tests about diff quality These show the differences between bdiff and xdiff. Differential Revision: https://phab.mercurial-scm.org/D2604

run-tests: allow #require inside #if Used by the next patch. Differential Revision: https://phab.mercurial-scm.org/D2605

mdiff: add a config option to use xdiff algorithm The `experimental.xdiff` will affect the default diffopts and make mdiff use the xdiff algorithm for better diff quality. Differential Revision: https://phab.mercurial-scm.org/D2603

bdiff: add a xdiffblocks method This is similar to `bdiff.blocks`, but uses xdiff as the backend. The indent heuristic is turned on by default since it has little overhead and improves diff quality significantly. Differential Revision: https://phab.mercurial-scm.org/D2602

xdiff: reduce indent heuristic overhead Adds some threshold to avoid expensive cases, like: ``` #!python open('a', 'w').write(" \n" * 1000000) open('b', 'w').write(" \n" * 1000001) ``` The indent heuristic is O(N * 20) (N = 1000000) for the above case, and makes diff much slower. Before this patch (system git 2.14.2): ``` git diff --no-indent-heuristic a b 0.21s user 0.03s system 100% cpu 0.239 total git diff --indent-heuristic a b 0.77s user 0.02s system 99% cpu 0.785 total ``` After this patch (git 2fc74f41, with xdiffi.c patched): ``` # with the changed xdiffi.c git diff --indent-heuristic a b 0.16s user 0.01s system 90% cpu 0.188 total git diff --no-indent-heuristic a b 0.18s user 0.01s system 99% cpu 0.192 total ``` Now turning on indent-heuristic has no visible impact on performance. Differential Revision: https://phab.mercurial-scm.org/D2601

xdiff: add a bdiff hunk mode xdiff generated hunks for the differences (ex. questionmarks in the `@@ -?,? +?,? @@` part from `diff --git` output). However, bdiff generates matched hunks instead. This patch adds a `XDL_EMIT_BDIFFHUNK` flag used by the output function `xdl_call_hunk_func`. Once set, xdiff will generate bdiff-like hunks instead. That makes it easier to use xdiff as a drop-in replacement of bdiff. Note that since `bdiff('', '')` returns `[(0, 0, 0, 0)]`, the shortcut path `if (xscr)` is removed. I have checked functions called with `xscr` argument (`xdl_mark_ignorable`, `xdl_call_hunk_func`, `xdl_emit_diff`, `xdl_free_script`) work just fine with `xscr = NULL`. Test Plan: Will be tested in a later patch. Differential Revision: https://phab.mercurial-scm.org/D2575

xdiff: remove patience and histogram diff algorithms Patience diff is the normal diff algorithm, plus some greediness that unconditionally matches common common unique lines. That means it is easy to construct cases to let it generate suboptimal result, like: ``` open('a', 'w').write('\n'.join(list('a' + 'x' * 300 + 'u' + 'x' * 700 + 'a\n'))) open('b', 'w').write('\n'.join(list('b' + 'x' * 700 + 'u' + 'x' * 300 + 'b\n'))) ``` Patience diff has been advertised as being able to generate better results for some C code changes. However, the more scientific way to do that is the indention heuristic [1]. Since patience diff could generate suboptimal result more easily and its "better" diff feature could be replaced by the new indention heuristic, let's just remove it and its variant histogram diff to simplify the code. [1]: https://github.com/git/git/commit/433860f3d0beb0c6f205290bd16cda413148f098 Test Plan: `gcc -fPIC *.c --shared -o xdiff.so` still builds. Differential Revision: https://phab.mercurial-scm.org/D2573

xdiff: vendor xdiff library from git Vendor git's xdiff library from git commit d7c6c2369d7c6c2369ac21141b7c6cceaebc6414ec3da14ad using GPL2+ license. There is another recent user report that hg diff generates suboptimal result. It seems the fix to issue4074 isn't good enough. I crafted some other interesting cases, and hg diff barely has any advantage compared with gnu diffutils or git diff. | testcase | gnu diffutils | hg diff | git diff | | | lines time | lines time | lines time | | patience | 6 0.00 | 602 0.08 | 6 0.00 | | random | 91772 0.90 | 109462 0.70 | 91772 0.24 | | json | 2 0.03 | 1264814 1.81 | 2 0.29 | "lines" means the size of the output, i.e. the count of "+/-" lines. "time" means seconds needed to do the calculation. Both are the smaller the better. "hg diff" counts Python startup overhead. Git and GNU diffutils generate optimal results. For the "json" case, git can have an optimization that does a scan for common prefix and suffix first, and match them if the length is greater than half of the text. See https://neil.fraser.name/news/2006/03/12/. That would make git the fastest for all above cases. About testcases: patience: Aiming for the weakness of the greedy "patience diff" algorithm. Using git's patience diff option would also get suboptimal result. Generated using the Python script: ``` open('a', 'w').write('\n'.join(list('a' + 'x' * 300 + 'u' + 'x' * 700 + 'a\n'))) open('b', 'w').write('\n'.join(list('b' + 'x' * 700 + 'u' + 'x' * 300 + 'b\n'))) ``` random: Generated using the script in `test-issue4074.t`. It practically makes the algorithm suffer. Impressively, git wins in both performance and diff quality. json: The recent user reported case. It's a single line movement near the end of a very large (800K lines) JSON file. Test Plan: Code taken as-is. Differential Revision: https://phab.mercurial-scm.org/D2572 # no-check-commit for vendored code

templater: provide hint for multi-line templates with parse errors Previously, we punted. Now we "rewrite" the template's newlines to r'\n' and offset the hint appropriately. Differential Revision: https://phab.mercurial-scm.org/D2609

templater: add hint to template parse errors to help locate issues Previously, we would print the error name and location, but this isn't as helpful as we can be. Let's add a hint that shows the location where we encountered the parse error. Differential Revision: https://phab.mercurial-scm.org/D2608

py3: use b"%d" to covert integer to bytes instead of str Differential Revision: https://phab.mercurial-scm.org/D2618

py3: use bytes() instead of str() Differential Revision: https://phab.mercurial-scm.org/D2617

py3: replace __str__ to __bytes__ in hgext/journal.py Differential Revision: https://phab.mercurial-scm.org/D2616