mercurial/grep.py

copies-rust: tokenize all paths into integer

Copy information for each changesets tend to affect a small new number of path.
However, each of these path might be handled a large number of time. Handling
HgPathBuf (aka `Vec<u8>`) is expensive. Handling integer is cheap.

With this patch we:
- turn any input path into an integer "token" early,
- do all the internal logic using such "token",
- turn "token" back into path right before returning a result.

This gives use a quite significant performance boost in our slower cases.

Repo            Case                                  Source-Rev   Dest-Rev      # of revisions  old time      new time      Difference    Factor    time per rev
---------------------------------------------------------------------------------------------------------------------------------------------------------------
pypy            x000_revs_x000_added_x_copies         8a3b5bfd266e 2c68e87c3efe :   6780 revs,   0.092828 s,   0.081225 s,  -0.011603 s, × 0.8750,    11 µs/rev
pypy            x0000_revs_x_added_0_copies           d1defd0dc478 c9cb1334cc78 :  43645 revs,   0.711975 s,   0.586011 s,  -0.125964 s, × 0.8231,    13 µs/rev
pypy            x0000_revs_xx000_added_x000_copies    08ea3258278e d9fa043f30c0 :  11316 revs,   0.124505 s,   0.114173 s,  -0.010332 s, × 0.9170,    10 µs/rev
netbeans        x000_revs_x000_added_x000_copies      ff453e9fee32 411350406ec2 :   5750 revs,   0.072072 s,   0.061004 s,  -0.011068 s, × 0.8464,    10 µs/rev
netbeans        x0000_revs_xx000_added_x000_copies    588c2d1ced70 1aad62e59ddd :  66949 revs,   0.682732 s,   0.535874 s,  -0.146858 s, × 0.7849,     8 µs/rev
mozilla-central x00000_revs_x0000_added_x0000_copies  6832ae71433c 4c222a1d9a00 : 153721 revs,   1.935918 s,   1.781383 s,  -0.154535 s, × 0.9202,    11 µs/rev
mozilla-central x00000_revs_x00000_added_x000_copies  76caed42cf7c 1daa622bbe42 : 204976 revs,   2.827320 s,   2.603867 s,  -0.223453 s, × 0.9210,    12 µs/rev
mozilla-try     x0000_revs_xx000_added_x000_copies    89294cd501d9 7ccb2fc7ccb5 :  97052 revs,   3.243010 s,   1.529120 s,  -1.713890 s, × 0.4715,    15 µs/rev
mozilla-try     x00000_revs_x_added_0_copies          6a320851d377 1ebb79acd503 : 363753 revs,   5.693818 s,   4.842699 s,  -0.851119 s, × 0.8505,    13 µs/rev
mozilla-try     x00000_revs_x_added_x_copies          5173c4b6f97c 95d83ee7242d : 362229 revs,   5.677655 s,   4.761732 s,  -0.915923 s, × 0.8387,    13 µs/rev
mozilla-try     x00000_revs_x000_added_x_copies       9126823d0e9c ca82787bb23c : 359344 revs,   5.563370 s,   4.733912 s,  -0.829458 s, × 0.8509,    13 µs/rev
mozilla-try     x00000_revs_x0000_added_x0000_copies  8d3fafa80d4b eb884023b810 : 192665 revs,   2.864099 s,   2.593410 s,  -0.270689 s, × 0.9055,    13 µs/rev
mozilla-try     x00000_revs_x00000_added_x0000_copies 1b661134e2ca 1ae03d022d6d : 228985 revs, 113.297287 s,  41.041198 s, -72.256089 s, × 0.3622,   179 µs/rev
mozilla-try     x00000_revs_x00000_added_x000_copies  9b2a99adc05e 8e29777b48e6 : 382065 revs,  59.498652 s,  27.915689 s, -31.582963 s, × 0.4692,    73 µs/rev





Full timing comparison between this revision and the previous one:


Repo            Case                                  Source-Rev   Dest-Rev      # of revisions  old time      new time      Difference    Factor    time per rev
---------------------------------------------------------------------------------------------------------------------------------------------------------------
mercurial       x_revs_x_added_0_copies               ad6b123de1c7 39cfcef4f463 :      1 revs,   0.000042 s,   0.000042 s,  +0.000000 s, × 1.0000,    42 µs/rev
mercurial       x_revs_x_added_x_copies               2b1c78674230 0c1d10351869 :      6 revs,   0.000104 s,   0.000110 s,  +0.000006 s, × 1.0577,    18 µs/rev
mercurial       x000_revs_x000_added_x_copies         81f8ff2a9bf2 dd3267698d84 :   1032 revs,   0.004913 s,   0.004918 s,  +0.000005 s, × 1.0010,     4 µs/rev
pypy            x_revs_x_added_0_copies               aed021ee8ae8 099ed31b181b :      9 revs,   0.000191 s,   0.000195 s,  +0.000004 s, × 1.0209,    21 µs/rev
pypy            x_revs_x000_added_0_copies            4aa4e1f8e19a 359343b9ac0e :      1 revs,   0.000050 s,   0.000049 s,  -0.000001 s, × 0.9800,    49 µs/rev
pypy            x_revs_x_added_x_copies               ac52eb7bbbb0 72e022663155 :      7 revs,   0.000112 s,   0.000112 s,  +0.000000 s, × 1.0000,    16 µs/rev
pypy            x_revs_x00_added_x_copies             c3b14617fbd7 ace7255d9a26 :      1 revs,   0.000288 s,   0.000324 s,  +0.000036 s, × 1.1250,   324 µs/rev
pypy            x_revs_x000_added_x000_copies         df6f7a526b60 a83dc6a2d56f :      6 revs,   0.010411 s,   0.010611 s,  +0.000200 s, × 1.0192,  1768 µs/rev
pypy            x000_revs_xx00_added_0_copies         89a76aede314 2f22446ff07e :   4785 revs,   0.052852 s,   0.050835 s,  -0.002017 s, × 0.9618,    10 µs/rev
pypy            x000_revs_x000_added_x_copies         8a3b5bfd266e 2c68e87c3efe :   6780 revs,   0.092828 s,   0.081225 s,  -0.011603 s, × 0.8750,    11 µs/rev
pypy            x000_revs_x000_added_x000_copies      89a76aede314 7b3dda341c84 :   5441 revs,   0.063269 s,   0.061291 s,  -0.001978 s, × 0.9687,    11 µs/rev
pypy            x0000_revs_x_added_0_copies           d1defd0dc478 c9cb1334cc78 :  43645 revs,   0.711975 s,   0.586011 s,  -0.125964 s, × 0.8231,    13 µs/rev
pypy            x0000_revs_xx000_added_0_copies       bf2c629d0071 4ffed77c095c :      2 revs,   0.012771 s,   0.012824 s,  +0.000053 s, × 1.0042,  6412 µs/rev
pypy            x0000_revs_xx000_added_x000_copies    08ea3258278e d9fa043f30c0 :  11316 revs,   0.124505 s,   0.114173 s,  -0.010332 s, × 0.9170,    10 µs/rev
netbeans        x_revs_x_added_0_copies               fb0955ffcbcd a01e9239f9e7 :      2 revs,   0.000082 s,   0.000085 s,  +0.000003 s, × 1.0366,    42 µs/rev
netbeans        x_revs_x000_added_0_copies            6f360122949f 20eb231cc7d0 :      2 revs,   0.000111 s,   0.000108 s,  -0.000003 s, × 0.9730,    54 µs/rev
netbeans        x_revs_x_added_x_copies               1ada3faf6fb6 5a39d12eecf4 :      3 revs,   0.000171 s,   0.000175 s,  +0.000004 s, × 1.0234,    58 µs/rev
netbeans        x_revs_x00_added_x_copies             35be93ba1e2c 9eec5e90c05f :      9 revs,   0.000708 s,   0.000719 s,  +0.000011 s, × 1.0155,    79 µs/rev
netbeans        x000_revs_xx00_added_0_copies         eac3045b4fdd 51d4ae7f1290 :   1421 revs,   0.010608 s,   0.010175 s,  -0.000433 s, × 0.9592,     7 µs/rev
netbeans        x000_revs_x000_added_x_copies         e2063d266acd 6081d72689dc :   1533 revs,   0.015635 s,   0.015569 s,  -0.000066 s, × 0.9958,    10 µs/rev
netbeans        x000_revs_x000_added_x000_copies      ff453e9fee32 411350406ec2 :   5750 revs,   0.072072 s,   0.061004 s,  -0.011068 s, × 0.8464,    10 µs/rev
netbeans        x0000_revs_xx000_added_x000_copies    588c2d1ced70 1aad62e59ddd :  66949 revs,   0.682732 s,   0.535874 s,  -0.146858 s, × 0.7849,     8 µs/rev
mozilla-central x_revs_x_added_0_copies               3697f962bb7b 7015fcdd43a2 :      2 revs,   0.000090 s,   0.000090 s,  +0.000000 s, × 1.0000,    45 µs/rev
mozilla-central x_revs_x000_added_0_copies            dd390860c6c9 40d0c5bed75d :      8 revs,   0.000210 s,   0.000281 s,  +0.000071 s, × 1.3381,    35 µs/rev
mozilla-central x_revs_x_added_x_copies               8d198483ae3b 14207ffc2b2f :      9 revs,   0.000182 s,   0.000187 s,  +0.000005 s, × 1.0275,    20 µs/rev
mozilla-central x_revs_x00_added_x_copies             98cbc58cc6bc 446a150332c3 :      7 revs,   0.000594 s,   0.000660 s,  +0.000066 s, × 1.1111,    94 µs/rev
mozilla-central x_revs_x000_added_x000_copies         3c684b4b8f68 0a5e72d1b479 :      3 revs,   0.003102 s,   0.003385 s,  +0.000283 s, × 1.0912,  1128 µs/rev
mozilla-central x_revs_x0000_added_x0000_copies       effb563bb7e5 c07a39dc4e80 :      6 revs,   0.060234 s,   0.069812 s,  +0.009578 s, × 1.1590, 11635 µs/rev
mozilla-central x000_revs_xx00_added_0_copies         6100d773079a 04a55431795e :   1593 revs,   0.006300 s,   0.006503 s,  +0.000203 s, × 1.0322,     4 µs/rev
mozilla-central x000_revs_x000_added_x_copies         9f17a6fc04f9 2d37b966abed :     41 revs,   0.004817 s,   0.004988 s,  +0.000171 s, × 1.0355,   121 µs/rev
mozilla-central x000_revs_x000_added_x000_copies      7c97034feb78 4407bd0c6330 :   7839 revs,   0.065451 s,   0.063963 s,  -0.001488 s, × 0.9773,     8 µs/rev
mozilla-central x0000_revs_xx000_added_0_copies       9eec5917337d 67118cc6dcad :    615 revs,   0.026282 s,   0.026225 s,  -0.000057 s, × 0.9978,    42 µs/rev
mozilla-central x0000_revs_xx000_added_x000_copies    f78c615a656c 96a38b690156 :  30263 revs,   0.206873 s,   0.201377 s,  -0.005496 s, × 0.9734,     6 µs/rev
mozilla-central x00000_revs_x0000_added_x0000_copies  6832ae71433c 4c222a1d9a00 : 153721 revs,   1.935918 s,   1.781383 s,  -0.154535 s, × 0.9202,    11 µs/rev
mozilla-central x00000_revs_x00000_added_x000_copies  76caed42cf7c 1daa622bbe42 : 204976 revs,   2.827320 s,   2.603867 s,  -0.223453 s, × 0.9210,    12 µs/rev
mozilla-try     x_revs_x_added_0_copies               aaf6dde0deb8 9790f499805a :      2 revs,   0.000842 s,   0.000845 s,  +0.000003 s, × 1.0036,   422 µs/rev
mozilla-try     x_revs_x000_added_0_copies            d8d0222927b4 5bb8ce8c7450 :      2 revs,   0.000870 s,   0.000862 s,  -0.000008 s, × 0.9908,   431 µs/rev
mozilla-try     x_revs_x_added_x_copies               092fcca11bdb 936255a0384a :      4 revs,   0.000165 s,   0.000161 s,  -0.000004 s, × 0.9758,    40 µs/rev
mozilla-try     x_revs_x00_added_x_copies             b53d2fadbdb5 017afae788ec :      2 revs,   0.001145 s,   0.001163 s,  +0.000018 s, × 1.0157,   581 µs/rev
mozilla-try     x_revs_x000_added_x000_copies         20408ad61ce5 6f0ee96e21ad :      1 revs,   0.026500 s,   0.032414 s,  +0.005914 s, × 1.2232, 32414 µs/rev
mozilla-try     x_revs_x0000_added_x0000_copies       effb563bb7e5 c07a39dc4e80 :      6 revs,   0.059407 s,   0.070149 s,  +0.010742 s, × 1.1808, 11691 µs/rev
mozilla-try     x000_revs_xx00_added_0_copies         6100d773079a 04a55431795e :   1593 revs,   0.006325 s,   0.006526 s,  +0.000201 s, × 1.0318,     4 µs/rev
mozilla-try     x000_revs_x000_added_x_copies         9f17a6fc04f9 2d37b966abed :     41 revs,   0.005171 s,   0.005187 s,  +0.000016 s, × 1.0031,   126 µs/rev
mozilla-try     x000_revs_x000_added_x000_copies      1346fd0130e4 4c65cbdabc1f :   6657 revs,   0.066837 s,   0.065047 s,  -0.001790 s, × 0.9732,     9 µs/rev
mozilla-try     x0000_revs_x_added_0_copies           63519bfd42ee a36a2a865d92 :  40314 revs,   0.314252 s,   0.301129 s,  -0.013123 s, × 0.9582,     7 µs/rev
mozilla-try     x0000_revs_x_added_x_copies           9fe69ff0762d bcabf2a78927 :  38690 revs,   0.304160 s,   0.280683 s,  -0.023477 s, × 0.9228,     7 µs/rev
mozilla-try     x0000_revs_xx000_added_x_copies       156f6e2674f2 4d0f2c178e66 :   8598 revs,   0.089223 s,   0.084897 s,  -0.004326 s, × 0.9515,     9 µs/rev
mozilla-try     x0000_revs_xx000_added_0_copies       9eec5917337d 67118cc6dcad :    615 revs,   0.026711 s,   0.026620 s,  -0.000091 s, × 0.9966,    43 µs/rev
mozilla-try     x0000_revs_xx000_added_x000_copies    89294cd501d9 7ccb2fc7ccb5 :  97052 revs,   3.243010 s,   1.529120 s,  -1.713890 s, × 0.4715,    15 µs/rev
mozilla-try     x0000_revs_x0000_added_x0000_copies   e928c65095ed e951f4ad123a :  52031 revs,   0.756500 s,   0.738709 s,  -0.017791 s, × 0.9765,    14 µs/rev
mozilla-try     x00000_revs_x_added_0_copies          6a320851d377 1ebb79acd503 : 363753 revs,   5.693818 s,   4.842699 s,  -0.851119 s, × 0.8505,    13 µs/rev
mozilla-try     x00000_revs_x00000_added_0_copies     dc8a3ca7010e d16fde900c9c :  34414 revs,   0.590904 s,   0.596946 s,  +0.006042 s, × 1.0102,    17 µs/rev
mozilla-try     x00000_revs_x_added_x_copies          5173c4b6f97c 95d83ee7242d : 362229 revs,   5.677655 s,   4.761732 s,  -0.915923 s, × 0.8387,    13 µs/rev
mozilla-try     x00000_revs_x000_added_x_copies       9126823d0e9c ca82787bb23c : 359344 revs,   5.563370 s,   4.733912 s,  -0.829458 s, × 0.8509,    13 µs/rev
mozilla-try     x00000_revs_x0000_added_x0000_copies  8d3fafa80d4b eb884023b810 : 192665 revs,   2.864099 s,   2.593410 s,  -0.270689 s, × 0.9055,    13 µs/rev
mozilla-try     x00000_revs_x00000_added_x0000_copies 1b661134e2ca 1ae03d022d6d : 228985 revs, 113.297287 s,  41.041198 s, -72.256089 s, × 0.3622,   179 µs/rev
mozilla-try     x00000_revs_x00000_added_x000_copies  9b2a99adc05e 8e29777b48e6 : 382065 revs,  59.498652 s,  27.915689 s, -31.582963 s, × 0.4692,    73 µs/rev


Full timing comparison between this revision and the filelog copy tracing.

Repo            Case                                  Source-Rev   Dest-Rev      # of revisions   filelog      sidedata      Difference    Factor    time per rev
---------------------------------------------------------------------------------------------------------------------------------------------------------------
mercurial       x_revs_x_added_0_copies               ad6b123de1c7 39cfcef4f463 :       1 revs,   0.000903 s,  0.000042 s,  -0.000861 s, × 0.0465,    41 µs/rev
mercurial       x_revs_x_added_x_copies               2b1c78674230 0c1d10351869 :       6 revs,   0.001861 s,  0.000110 s,  -0.001751 s, × 0.0591,    18 µs/rev
mercurial       x000_revs_x000_added_x_copies         81f8ff2a9bf2 dd3267698d84 :    1032 revs,   0.018577 s,  0.004918 s,  -0.013659 s, × 0.2647,     4 µs/rev
pypy            x_revs_x_added_0_copies               aed021ee8ae8 099ed31b181b :       9 revs,   0.001519 s,  0.000195 s,  -0.001324 s, × 0.1283,    21 µs/rev
pypy            x_revs_x000_added_0_copies            4aa4e1f8e19a 359343b9ac0e :       1 revs,   0.213855 s,  0.000049 s,  -0.350d73 s, × 0.0002,    48 µs/rev
pypy            x_revs_x_added_x_copies               ac52eb7bbbb0 72e022663155 :       7 revs,   0.017022 s,  0.000112 s,  -0.016910 s, × 0.0065,    15 µs/rev
pypy            x_revs_x00_added_x_copies             c3b14617fbd7 ace7255d9a26 :       1 revs,   0.019398 s,  0.000324 s,  -0.019074 s, × 0.0167,   323 µs/rev
pypy            x_revs_x000_added_x000_copies         df6f7a526b60 a83dc6a2d56f :       6 revs,   0.769467 s,  0.010611 s,  -0.758856 s, × 0.0137,  1768 µs/rev
pypy            x000_revs_xx00_added_0_copies         89a76aede314 2f22446ff07e :    4785 revs,   1.221952 s,  0.050835 s,  -1.171117 s, × 0.0416,    10 µs/rev
pypy            x000_revs_x000_added_x_copies         8a3b5bfd266e 2c68e87c3efe :    6780 revs,   1.304007 s,  0.081225 s,  -1.222782 s, × 0.0622,    11 µs/rev
pypy            x000_revs_x000_added_x000_copies      89a76aede314 7b3dda341c84 :    5441 revs,   1.686610 s,  0.061291 s,  -1.625319 s, × 0.0363,    11 µs/rev
pypy            x0000_revs_x_added_0_copies           d1defd0dc478 c9cb1334cc78 :   43645 revs,   0.001107 s,  0.586011 s,  +0.584904 s, × 529.36,    13 µs/rev
pypy            x0000_revs_xx000_added_0_copies       bf2c629d0071 4ffed77c095c :       2 revs,   1.100760 s,  0.012824 s,  -1.087936 s, × 0.0116,  6408 µs/rev
pypy            x0000_revs_xx000_added_x000_copies    08ea3258278e d9fa043f30c0 :   11316 revs,   1.350547 s,  0.114173 s,  -1.236374 s, × 0.0845,    10 µs/rev
netbeans        x_revs_x_added_0_copies               fb0955ffcbcd a01e9239f9e7 :       2 revs,   0.027864 s,  0.000085 s,  -0.027779 s, × 0.0030,    42 µs/rev
netbeans        x_revs_x000_added_0_copies            6f360122949f 20eb231cc7d0 :       2 revs,   0.132479 s,  0.000108 s,  -0.132371 s, × 0.0008,    53 µs/rev
netbeans        x_revs_x_added_x_copies               1ada3faf6fb6 5a39d12eecf4 :       3 revs,   0.025405 s,  0.000175 s,  -0.025230 s, × 0.0068,    58 µs/rev
netbeans        x_revs_x00_added_x_copies             35be93ba1e2c 9eec5e90c05f :       9 revs,   0.053244 s,  0.000719 s,  -0.052525 s, × 0.0135,    79 µs/rev
netbeans        x000_revs_xx00_added_0_copies         eac3045b4fdd 51d4ae7f1290 :    1421 revs,   0.038017 s,  0.010175 s,  -0.027842 s, × 0.2676,     7 µs/rev
netbeans        x000_revs_x000_added_x_copies         e2063d266acd 6081d72689dc :    1533 revs,   0.198308 s,  0.015569 s,  -0.182739 s, × 0.0785,    10 µs/rev
netbeans        x000_revs_x000_added_x000_copies      ff453e9fee32 411350406ec2 :    5750 revs,   0.949749 s,  0.061004 s,  -0.888745 s, × 0.0642,    10 µs/rev
netbeans        x0000_revs_xx000_added_x000_copies    588c2d1ced70 1aad62e59ddd :   66949 revs,   3.932262 s,  0.535874 s,  -3.396388 s, × 0.1362,     8 µs/rev
mozilla-central x_revs_x_added_0_copies               3697f962bb7b 7015fcdd43a2 :       2 revs,   0.024490 s,  0.000090 s,  -0.024400 s, × 0.0036,    44 µs/rev
mozilla-central x_revs_x000_added_0_copies            dd390860c6c9 40d0c5bed75d :       8 revs,   0.143885 s,  0.000281 s,  -0.143604 s, × 0.0019,    35 µs/rev
mozilla-central x_revs_x_added_x_copies               8d198483ae3b 14207ffc2b2f :       9 revs,   0.025471 s,  0.000187 s,  -0.025284 s, × 0.0073,    20 µs/rev
mozilla-central x_revs_x00_added_x_copies             98cbc58cc6bc 446a150332c3 :       7 revs,   0.086013 s,  0.000660 s,  -0.085353 s, × 0.0076,    94 µs/rev
mozilla-central x_revs_x000_added_x000_copies         3c684b4b8f68 0a5e72d1b479 :       3 revs,   0.200726 s,  0.003385 s,  -0.197341 s, × 0.0168,  1127 µs/rev
mozilla-central x_revs_x0000_added_x0000_copies       effb563bb7e5 c07a39dc4e80 :       6 revs,   2.224171 s,  0.069812 s,  -2.154359 s, × 0.0313, 11633 µs/rev
mozilla-central x000_revs_xx00_added_0_copies         6100d773079a 04a55431795e :    1593 revs,   0.090780 s,  0.006503 s,  -0.084277 s, × 0.0716,     4 µs/rev
mozilla-central x000_revs_x000_added_x_copies         9f17a6fc04f9 2d37b966abed :      41 revs,   0.764805 s,  0.004988 s,  -0.759817 s, × 0.0065,   121 µs/rev
mozilla-central x000_revs_x000_added_x000_copies      7c97034feb78 4407bd0c6330 :    7839 revs,   1.161405 s,  0.063963 s,  -1.097442 s, × 0.0550,     8 µs/rev
mozilla-central x0000_revs_xx000_added_0_copies       9eec5917337d 67118cc6dcad :     615 revs,   6.816186 s,  0.026225 s,  -6.789961 s, × 0.0038,    42 µs/rev
mozilla-central x0000_revs_xx000_added_x000_copies    f78c615a656c 96a38b690156 :   30263 revs,   3.374819 s,  0.201377 s,  -3.173442 s, × 0.0596,     6 µs/rev
mozilla-central x00000_revs_x0000_added_x0000_copies  6832ae71433c 4c222a1d9a00 :  153721 revs,  16.285469 s,  1.781383 s, -14.504086 s, × 0.1093,    11 µs/rev
mozilla-central x00000_revs_x00000_added_x000_copies  76caed42cf7c 1daa622bbe42 :  204976 revs,  21.207733 s,  2.603867 s, -18.603866 s, × 0.1227,    12 µs/rev
mozilla-try     x_revs_x_added_0_copies               aaf6dde0deb8 9790f499805a :       2 revs,   0.080843 s,  0.000845 s,  -0.079998 s, × 0.0104,   422 µs/rev
mozilla-try     x_revs_x000_added_0_copies            d8d0222927b4 5bb8ce8c7450 :       2 revs,   0.511068 s,  0.000862 s,  -0.510206 s, × 0.0016,   430 µs/rev
mozilla-try     x_revs_x_added_x_copies               092fcca11bdb 936255a0384a :       4 revs,   0.021573 s,  0.000161 s,  -0.021412 s, × 0.0074,    40 µs/rev
mozilla-try     x_revs_x00_added_x_copies             b53d2fadbdb5 017afae788ec :       2 revs,   0.227726 s,  0.001163 s,  -0.226563 s, × 0.0051,   581 µs/rev
mozilla-try     x_revs_x000_added_x000_copies         20408ad61ce5 6f0ee96e21ad :       1 revs,   1.120448 s,  0.032414 s,  -1.088034 s, × 0.0289, 32381 µs/rev
mozilla-try     x_revs_x0000_added_x0000_copies       effb563bb7e5 c07a39dc4e80 :       6 revs,   2.241713 s,  0.070149 s,  -2.171564 s, × 0.0312, 11689 µs/rev
mozilla-try     x000_revs_xx00_added_0_copies         6100d773079a 04a55431795e :    1593 revs,   0.090633 s,  0.006526 s,  -0.084107 s, × 0.0720,     4 µs/rev
mozilla-try     x000_revs_x000_added_x_copies         9f17a6fc04f9 2d37b966abed :      41 revs,   0.770403 s,  0.005187 s,  -0.765216 s, × 0.0067,   126 µs/rev
mozilla-try     x000_revs_x000_added_x000_copies      1346fd0130e4 4c65cbdabc1f :    6657 revs,   1.184557 s,  0.065047 s,  -1.119510 s, × 0.0549,     9 µs/rev
mozilla-try     x0000_revs_x_added_0_copies           63519bfd42ee a36a2a865d92 :   40314 revs,   0.085790 s,  0.301129 s,  +0.215339 s, × 3.5100,     7 µs/rev
mozilla-try     x0000_revs_x_added_x_copies           9fe69ff0762d bcabf2a78927 :   38690 revs,   0.080616 s,  0.280683 s,  +0.200067 s, × 3.4817,     7 µs/rev
mozilla-try     x0000_revs_xx000_added_x_copies       156f6e2674f2 4d0f2c178e66 :    8598 revs,   7.712554 s,  0.084897 s,  -7.627657 s, × 0.0110,     9 µs/rev
mozilla-try     x0000_revs_xx000_added_0_copies       9eec5917337d 67118cc6dcad :     615 revs,   6.937294 s,  0.026620 s,  -6.910674 s, × 0.0038,    43 µs/rev
mozilla-try     x0000_revs_xx000_added_x000_copies    89294cd501d9 7ccb2fc7ccb5 :   97052 revs,   7.712313 s,  1.529120 s,  -6.183193 s, × 0.1982,    15 µs/rev
mozilla-try     x0000_revs_x0000_added_x0000_copies   e928c65095ed e951f4ad123a :   52031 revs,   9.966910 s,  0.738709 s,  -9.228201 s, × 0.0741,    14 µs/rev
mozilla-try     x00000_revs_x_added_0_copies          6a320851d377 1ebb79acd503 :  363753 revs,   0.090397 s,  4.842699 s,  +4.752302 s, × 53.571,    13 µs/rev
mozilla-try     x00000_revs_x00000_added_0_copies     dc8a3ca7010e d16fde900c9c :   34414 revs,  27.817167 s,  0.596946 s, -27.220221 s, × 0.0214,    17 µs/rev
mozilla-try     x00000_revs_x_added_x_copies          5173c4b6f97c 95d83ee7242d :  362229 revs,   0.091305 s,  4.761732 s,  +4.670427 s, × 52.151,    13 µs/rev
mozilla-try     x00000_revs_x000_added_x_copies       9126823d0e9c ca82787bb23c :  359344 revs,   0.231183 s,  4.733912 s,  +4.502729 s, × 20.476,    13 µs/rev
mozilla-try     x00000_revs_x0000_added_x0000_copies  8d3fafa80d4b eb884023b810 :  192665 revs,  19.830617 s,  2.593410 s, -17.237207 s, × 0.1307,    13 µs/rev
mozilla-try     x00000_revs_x00000_added_x0000_copies 1b661134e2ca 1ae03d022d6d :  228985 revs,  21.743873 s, 41.041198 s, +19.297325 s, × 1.8874,   179 µs/rev
mozilla-try     x00000_revs_x00000_added_x000_copies  9b2a99adc05e 8e29777b48e6 :  382065 revs,  25.935037 s, 27.915689 s,  +1.980652 s, × 1.0763,    73 µs/rev

Differential Revision: https://phab.mercurial-scm.org/D9493

# grep.py - logic for history walk and grep
#
# Copyright 2005-2007 Matt Mackall <[e-mail]>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

from __future__ import absolute_import

import difflib
import errno

from .i18n import _

from . import (
    error,
    match as matchmod,
    pycompat,
    scmutil,
    util,
)


def matchlines(body, regexp):
    begin = 0
    linenum = 0
    while begin < len(body):
        match = regexp.search(body, begin)
        if not match:
            break
        mstart, mend = match.span()
        linenum += body.count(b'\n', begin, mstart) + 1
        lstart = body.rfind(b'\n', begin, mstart) + 1 or begin
        begin = body.find(b'\n', mend) + 1 or len(body) + 1
        lend = begin - 1
        yield linenum, mstart - lstart, mend - lstart, body[lstart:lend]


class linestate(object):
    def __init__(self, line, linenum, colstart, colend):
        self.line = line
        self.linenum = linenum
        self.colstart = colstart
        self.colend = colend

    def __hash__(self):
        return hash(self.line)

    def __eq__(self, other):
        return self.line == other.line

    def findpos(self, regexp):
        """Iterate all (start, end) indices of matches"""
        yield self.colstart, self.colend
        p = self.colend
        while p < len(self.line):
            m = regexp.search(self.line, p)
            if not m:
                break
            if m.end() == p:
                p += 1
            else:
                yield m.span()
                p = m.end()


def difflinestates(a, b):
    sm = difflib.SequenceMatcher(None, a, b)
    for tag, alo, ahi, blo, bhi in sm.get_opcodes():
        if tag == 'insert':
            for i in pycompat.xrange(blo, bhi):
                yield (b'+', b[i])
        elif tag == 'delete':
            for i in pycompat.xrange(alo, ahi):
                yield (b'-', a[i])
        elif tag == 'replace':
            for i in pycompat.xrange(alo, ahi):
                yield (b'-', a[i])
            for i in pycompat.xrange(blo, bhi):
                yield (b'+', b[i])


class grepsearcher(object):
    """Search files and revisions for lines matching the given pattern

    Options:
    - all_files to search unchanged files at that revision.
    - diff to search files in the parent revision so diffs can be generated.
    - follow to skip files across copies and renames.
    """

    def __init__(
        self, ui, repo, regexp, all_files=False, diff=False, follow=False
    ):
        self._ui = ui
        self._repo = repo
        self._regexp = regexp
        self._all_files = all_files
        self._diff = diff
        self._follow = follow

        self._getfile = util.lrucachefunc(repo.file)
        self._getrenamed = scmutil.getrenamedfn(repo)

        self._matches = {}
        self._copies = {}
        self._skip = set()
        self._revfiles = {}

    def skipfile(self, fn, rev):
        """Exclude the given file (and the copy at the specified revision)
        from future search"""
        copy = self._copies.get(rev, {}).get(fn)
        self._skip.add(fn)
        if copy:
            self._skip.add(copy)

    def searchfiles(self, revs, makefilematcher):
        """Walk files and revisions to yield (fn, ctx, pstates, states)
        matches

        states is a list of linestate objects. pstates may be empty unless
        diff is True.
        """
        for ctx in scmutil.walkchangerevs(
            self._repo, revs, makefilematcher, self._prep
        ):
            rev = ctx.rev()
            parent = ctx.p1().rev()
            for fn in sorted(self._revfiles.get(rev, [])):
                states = self._matches[rev][fn]
                copy = self._copies.get(rev, {}).get(fn)
                if fn in self._skip:
                    if copy:
                        self._skip.add(copy)
                    continue
                pstates = self._matches.get(parent, {}).get(copy or fn, [])
                if pstates or states:
                    yield fn, ctx, pstates, states
            del self._revfiles[rev]
            # We will keep the matches dict for the duration of the window
            # clear the matches dict once the window is over
            if not self._revfiles:
                self._matches.clear()

    def _grepbody(self, fn, rev, body):
        self._matches[rev].setdefault(fn, [])
        m = self._matches[rev][fn]
        if body is None:
            return

        for lnum, cstart, cend, line in matchlines(body, self._regexp):
            s = linestate(line, lnum, cstart, cend)
            m.append(s)

    def _readfile(self, ctx, fn):
        rev = ctx.rev()
        if rev is None:
            fctx = ctx[fn]
            try:
                return fctx.data()
            except IOError as e:
                if e.errno != errno.ENOENT:
                    raise
        else:
            flog = self._getfile(fn)
            fnode = ctx.filenode(fn)
            try:
                return flog.read(fnode)
            except error.CensoredNodeError:
                self._ui.warn(
                    _(
                        b'cannot search in censored file: '
                        b'%(filename)s:%(revnum)s\n'
                    )
                    % {b'filename': fn, b'revnum': pycompat.bytestr(rev)}
                )

    def _prep(self, ctx, fmatch):
        rev = ctx.rev()
        pctx = ctx.p1()
        self._matches.setdefault(rev, {})
        if self._diff:
            parent = pctx.rev()
            self._matches.setdefault(parent, {})
        files = self._revfiles.setdefault(rev, [])
        if rev is None:
            # in `hg grep pattern`, 2/3 of the time is spent is spent in
            # pathauditor checks without this in mozilla-central
            contextmanager = self._repo.wvfs.audit.cached
        else:
            contextmanager = util.nullcontextmanager
        with contextmanager():
            # TODO: maybe better to warn missing files?
            if self._all_files:
                fmatch = matchmod.badmatch(fmatch, lambda f, msg: None)
                filenames = ctx.matches(fmatch)
            else:
                filenames = (f for f in ctx.files() if fmatch(f))
            for fn in filenames:
                # fn might not exist in the revision (could be a file removed by
                # the revision). We could check `fn not in ctx` even when rev is
                # None, but it's less racy to protect againt that in readfile.
                if rev is not None and fn not in ctx:
                    continue

                copy = None
                if self._follow:
                    copy = self._getrenamed(fn, rev)
                    if copy:
                        self._copies.setdefault(rev, {})[fn] = copy
                        if fn in self._skip:
                            self._skip.add(copy)
                if fn in self._skip:
                    continue
                files.append(fn)

                if fn not in self._matches[rev]:
                    self._grepbody(fn, rev, self._readfile(ctx, fn))

                if self._diff:
                    pfn = copy or fn
                    if pfn not in self._matches[parent] and pfn in pctx:
                        self._grepbody(pfn, parent, self._readfile(pctx, pfn))