fuzz: add a quick README to try and document how to test new fuzzers Differential Revision: https://phab.mercurial-scm.org/D2633

fuzz: add a fuzzer for xdiff Based entirely on the fuzzer for bdiff. Differential Revision: https://phab.mercurial-scm.org/D2632

tests: add tests about diff quality These show the differences between bdiff and xdiff. Differential Revision: https://phab.mercurial-scm.org/D2604

run-tests: allow #require inside #if Used by the next patch. Differential Revision: https://phab.mercurial-scm.org/D2605

mdiff: add a config option to use xdiff algorithm The `experimental.xdiff` will affect the default diffopts and make mdiff use the xdiff algorithm for better diff quality. Differential Revision: https://phab.mercurial-scm.org/D2603

bdiff: add a xdiffblocks method This is similar to `bdiff.blocks`, but uses xdiff as the backend. The indent heuristic is turned on by default since it has little overhead and improves diff quality significantly. Differential Revision: https://phab.mercurial-scm.org/D2602

xdiff: reduce indent heuristic overhead Adds some threshold to avoid expensive cases, like: ``` #!python open('a', 'w').write(" \n" * 1000000) open('b', 'w').write(" \n" * 1000001) ``` The indent heuristic is O(N * 20) (N = 1000000) for the above case, and makes diff much slower. Before this patch (system git 2.14.2): ``` git diff --no-indent-heuristic a b 0.21s user 0.03s system 100% cpu 0.239 total git diff --indent-heuristic a b 0.77s user 0.02s system 99% cpu 0.785 total ``` After this patch (git 2fc74f41, with xdiffi.c patched): ``` # with the changed xdiffi.c git diff --indent-heuristic a b 0.16s user 0.01s system 90% cpu 0.188 total git diff --no-indent-heuristic a b 0.18s user 0.01s system 99% cpu 0.192 total ``` Now turning on indent-heuristic has no visible impact on performance. Differential Revision: https://phab.mercurial-scm.org/D2601

xdiff: add a bdiff hunk mode xdiff generated hunks for the differences (ex. questionmarks in the `@@ -?,? +?,? @@` part from `diff --git` output). However, bdiff generates matched hunks instead. This patch adds a `XDL_EMIT_BDIFFHUNK` flag used by the output function `xdl_call_hunk_func`. Once set, xdiff will generate bdiff-like hunks instead. That makes it easier to use xdiff as a drop-in replacement of bdiff. Note that since `bdiff('', '')` returns `[(0, 0, 0, 0)]`, the shortcut path `if (xscr)` is removed. I have checked functions called with `xscr` argument (`xdl_mark_ignorable`, `xdl_call_hunk_func`, `xdl_emit_diff`, `xdl_free_script`) work just fine with `xscr = NULL`. Test Plan: Will be tested in a later patch. Differential Revision: https://phab.mercurial-scm.org/D2575

xdiff: remove patience and histogram diff algorithms Patience diff is the normal diff algorithm, plus some greediness that unconditionally matches common common unique lines. That means it is easy to construct cases to let it generate suboptimal result, like: ``` open('a', 'w').write('\n'.join(list('a' + 'x' * 300 + 'u' + 'x' * 700 + 'a\n'))) open('b', 'w').write('\n'.join(list('b' + 'x' * 700 + 'u' + 'x' * 300 + 'b\n'))) ``` Patience diff has been advertised as being able to generate better results for some C code changes. However, the more scientific way to do that is the indention heuristic [1]. Since patience diff could generate suboptimal result more easily and its "better" diff feature could be replaced by the new indention heuristic, let's just remove it and its variant histogram diff to simplify the code. [1]: https://github.com/git/git/commit/433860f3d0beb0c6f205290bd16cda413148f098 Test Plan: `gcc -fPIC *.c --shared -o xdiff.so` still builds. Differential Revision: https://phab.mercurial-scm.org/D2573

xdiff: vendor xdiff library from git Vendor git's xdiff library from git commit d7c6c2369d7c6c2369ac21141b7c6cceaebc6414ec3da14ad using GPL2+ license. There is another recent user report that hg diff generates suboptimal result. It seems the fix to issue4074 isn't good enough. I crafted some other interesting cases, and hg diff barely has any advantage compared with gnu diffutils or git diff. | testcase | gnu diffutils | hg diff | git diff | | | lines time | lines time | lines time | | patience | 6 0.00 | 602 0.08 | 6 0.00 | | random | 91772 0.90 | 109462 0.70 | 91772 0.24 | | json | 2 0.03 | 1264814 1.81 | 2 0.29 | "lines" means the size of the output, i.e. the count of "+/-" lines. "time" means seconds needed to do the calculation. Both are the smaller the better. "hg diff" counts Python startup overhead. Git and GNU diffutils generate optimal results. For the "json" case, git can have an optimization that does a scan for common prefix and suffix first, and match them if the length is greater than half of the text. See https://neil.fraser.name/news/2006/03/12/. That would make git the fastest for all above cases. About testcases: patience: Aiming for the weakness of the greedy "patience diff" algorithm. Using git's patience diff option would also get suboptimal result. Generated using the Python script: ``` open('a', 'w').write('\n'.join(list('a' + 'x' * 300 + 'u' + 'x' * 700 + 'a\n'))) open('b', 'w').write('\n'.join(list('b' + 'x' * 700 + 'u' + 'x' * 300 + 'b\n'))) ``` random: Generated using the script in `test-issue4074.t`. It practically makes the algorithm suffer. Impressively, git wins in both performance and diff quality. json: The recent user reported case. It's a single line movement near the end of a very large (800K lines) JSON file. Test Plan: Code taken as-is. Differential Revision: https://phab.mercurial-scm.org/D2572 # no-check-commit for vendored code