largefiles: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6015

commit: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6014

tests: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6013

templatekw: migrate to new method for getting copy info Differential Revision: https://phab.mercurial-scm.org/D6012

convert: migrate to new method for getting copy source Differential Revision: https://phab.mercurial-scm.org/D6011

context: move equivalent renamed() implementations to superclass Now that renamed() in workingfilectx and in overlayfilectx are written in terms of copysource(), they are functionally identical and can be reused. Differential Revision: https://phab.mercurial-scm.org/D6010

context: rewrite renamed() in terms of new copysource() where appropriate It's only the filectx for committed files that have the copy source's nodeid easily available; workingfilectx's and overlayfilectx's renamed() simply find the nodeid by looking in the changeset's p1's manifest. Differential Revision: https://phab.mercurial-scm.org/D6009

context: add specialized way of getting copy source file only I'm working on support for storing copy metadata in the changeset instead of the filelog. I don't intend to include the file nodeid there, but most callers don't need that anyway. This patch introduces a method similar to ctx.renamed(), but the new method returns only the source filename, not the nodeid. Subsequent patches will move callers over to this new method. Differential Revision: https://phab.mercurial-scm.org/D6008

changegroup: move non-pruning of non-ellipsis manifests to _prunemanifests() Google has an extension that overrides _prunemanifests() and removes nodes that we fetch using another mechanism. That broke when _prunemanifests() no longer got called. It works again if we move the check for "not self._ellipses" inside _prunemanifests(). Differential Revision: https://phab.mercurial-scm.org/D6004

context: use includematcher when checking dir/file conflicts This is for performance; patternmatcher can't easily interpret its results to make visitchildrenset be the "optimal" set of paths to inspect, but includematcher can. Since there aren't any special patterns being used here, I believe that the two matchers are equivalent. Differential Revision: https://phab.mercurial-scm.org/D5999

tests: change the paths slightly in test-rebase-inmemory.t c/c was a little difficult to understand (and verify that it was the *correct* 'c/' that was being talked about), and it's useful to have multiple directories to prove that we are able to detect this even if there's no files (just a subdirectory) in the immediate directory that's conflicting. Differential Revision: https://phab.mercurial-scm.org/D5998

rust-cpython: using rustext.dagop.headrevs in revlog As with the previous oxidation series, revlog plays the role of the factory, either using its parents function, or passing the index. We include below results of revsetbenchmarks.py taken on the PyPy repository on those of contrib/all-revsets.tx that involve `heads()`. In most of the cases, this seems to be either neutral or an improvement. In the cases where it's actually a bit slower, we suspect that differences in `heads()` performance is actually burried in variance on the incoming revset (probably several orders of magnitude slower). The precheck for filtered revisions of parent changeset has a significative performance benefit, too. Result by revset ================ Revision: 0) 0c7b353ce100; rust-cpython: binding for headrevs() 1) Parent of this changeset; changelog: prefilter in headrevs() 2) This changeset revset #0: heads(commonancestors(last(head(), 2))) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.001379 0.001361 0.001381 0.001410 0.001393 0.001372 0.001414 0.001387 0.001411 0.001429 0.001415 1) 0.001351 0.001373 0.001383 0.001392 0.001401 0.001385 0.001405 0.001406 0.001385 0.001424 0.001399 2) 0.001365 0.001362 0.001375 0.001393 0.001370 0.001365 0.001413 0.001386 0.001377 0.001415 0.001411 revset #1: heads(commonancestors(head())) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.047578 0.048578 0.047764 0.048065 0.047289 0.047305 0.047729 0.047370 0.047611 0.048005 0.047755 1) 0.048072 0.047471 0.048351 0.048193 0.048380 0.047968 0.047683 0.047355 0.048587 0.047044 0.048299 2) 0.047124 0.046699 0.046896 0.047250 0.046920 0.047379 0.046855 0.047753 0.047289 0.047219 0.046991 revset #2: heads(all()) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.037654 0.037814 0.037149 0.037457 0.037609 0.037053 0.036825 0.037054 0.037739 0.036816 0.037604 1) 0.021845 58% 0.022172 58% 0.022148 59% 0.022059 58% 0.022261 59% 0.022246 60% 0.021691 58% 0.021967 59% 0.022156 58% 0.021820 59% 0.023141 61% 2) 0.014459 66% 0.014470 65% 0.014420 65% 0.014413 65% 0.014421 64% 0.014492 65% 0.014512 66% 0.014579 66% 0.014500 65% 0.014501 66% 0.014537 62% revset #3: heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.003696 0.003681 0.003719 0.003746 0.003725 0.003750 0.003692 0.003747 0.003712 0.003754 0.003763 1) 0.002131 57% 0.002142 58% 0.002147 57% 0.002203 58% 0.002143 57% 0.002208 58% 0.002158 58% 0.002182 58% 0.002169 58% 0.002209 58% 0.002201 58% 2) 0.001490 69% 0.001524 71% 0.001515 70% 0.001528 69% 0.001531 71% 0.001520 68% 0.001549 71% 0.001542 70% 0.001560 71% 0.001559 70% 0.001544 70% revset #4: (-5000:-1000) and heads(-10000:-1) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 0.003832 0.003816 0.003747 0.003814 0.003749 0.003894 0.003784 0.003796 0.003915 0.003829 0.003795 1) 0.002282 59% 0.002208 57% 0.002220 59% 0.002240 58% 0.002210 58% 0.002276 58% 0.002250 59% 0.002250 59% 0.002311 59% 0.002230 58% 0.002241 59% 2) 0.001658 72% 0.001662 75% 0.001568 70% 0.001599 71% 0.001588 71% 0.001696 74% 0.001615 71% 0.001593 70% 0.001710 73% 0.001622 72% 0.001616 72% revset #5: heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.826449 7.563260 7.581034 7.688493 7.634001 7.777860 7.768228 8.026097 7.767422 7.565254 7.938643 1) 7.750766 7.562555 7.660426 7.574089 7.492220 7.438582 7.562015 7.530635 93% 7.636343 7.636712 7.645113 2) 7.617941 7.519601 7.584922 7.507653 7.547440 7.524436 7.575291 7.883991 7.792142 7.709622 7.868595 revset #6: heads(matching(tip, "author")) and -10000:-1 plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.744489 7.728684 7.734065 7.928513 7.875949 7.883727 7.815492 7.791335 7.784793 7.761218 7.815731 1) 7.808956 7.480446 7.618759 7.920270 7.676343 7.803613 7.770210 7.713100 7.584420 7.767335 7.825140 2) 7.519987 7.938748 106% 7.805328 7.694162 7.750129 7.714229 7.603825 7.580734 7.555291 7.524207 7.504580 revset #7: (-10000:-1) and heads(matching(tip, "author")) plain min max first last reverse rev..rst rev..ast sort sor..rst sor..ast 0) 7.909321 7.694357 7.666021 7.538686 7.771821 7.876217 7.852103 7.812727 7.545919 7.788860 7.764585 1) 7.749232 7.683715 7.968393 7.895257 7.764160 8.314884 105% 7.921697 7.882613 7.867209 7.684707 7.544501 2) 7.824903 7.784605 7.727846 7.566613 7.581994 7.539205 90% 7.555316 7.535572 7.581786 7.901795 7.662832

changelog: prefilter in headrevs() In case where headrevs() is called on some revisions, we perform the check that aren't filtered in advance, and switch revlog to use its unchecked form. This allows to work with alternative implementations that don't have knowledge of the filtering system, such as the Rust one.

tests: fixed test too dependent on actual exception wording On one of the machines I use to run the tests prior to submission, the default Python is 2.7.5, with the following wording: must be encoded string without NULL bytes, not str This third form (and possible future ones) are motivation to use a wider catching regexp.

contrib: enforce wrapping too-long lines with () instead of \ This is the style I prefer, and an anecdotal exploration of styles recommended in style guides etc. Further, to quote pep8: > The preferred way of wrapping long lines is by using Python's implied > line continuation inside parentheses, brackets and braces. Long lines > can be broken over multiple lines by wrapping expressions in > parentheses. These should be used in preference to using a backslash > for line continuation. So I think this is a virtuous change. Differential Revision: https://phab.mercurial-scm.org/D5995

cleanup: prefer nested context managers to \-continuations I'd prefer Python accept a tuple of context managers, but alas it isn't meant to be. This will have to suffice. Differential Revision: https://phab.mercurial-scm.org/D5994

cleanup: use () to wrap long lines instead of \ This is a little less brittle, and often helps indentation. In a surprising number of cases the entire cleanup was deleting the \, as the expression was *already* parenthesized in a workable way. Differential Revision: https://phab.mercurial-scm.org/D5993

tests: use () instead of \ to wrap lines This should auto-format more consistently, and is slightly more typical Python. Differential Revision: https://phab.mercurial-scm.org/D5992

merge with stable

copies: do copy tracing based on ctx.p[12]copies() if configured This adds an option to do copy tracing in a changeset-optimized way. If the metadata is stored in filelogs, this is obviously going to be suboptimal. The point is that it provides a way of transitioning to changeset-stored metadata. Some of the tests behave a little differently, but they all seem resonable to me. The config option may very well be renamed later when it's clearer what options we want and how they will behave. When the test suite is run with --extra-config-opt to use the new copy tracing, all tests pass, besides test-copies.t (which fails in the same way as you can see in this patch). `hg debugpathcopies 4.0 4.8` reports 82 copies. With this option enabled, the only difference is this: -mercurial/pure/bdiff.py -> mercurial/cffi/bdiff.py +setup_bdiff_cffi.py -> mercurial/cffi/bdiff.py I believe that happened because it was renamed in different ways on different sides of a merge and the new algorithm arbitrarily prefers copies that happened on p1. The runtime is about 0.85 seconds with the old copy tracing and 5.7 seconds with the new copy tracing. That's kind of slow, but actually better than I had expected. Differential Revision: https://phab.mercurial-scm.org/D5991

context: introduce p[12]copies() methods and debugp[12]copies commands As mentioned earlier, I'm working on support for storing copy metadata in the changeset instead of in the filelog. In order to transition a repo from storing metadata in filelogs to storing it in the changeset, I'm going to provide a config option for reading the metadata from the changeset, but falling back to getting it from the filelog if it's not in the changeset. In this compatiblity mode, the changeset-optmized algorithms will be used. We will then need to convert the filelog copy metadata to look like that provided by changeset copy metadata. This patch introduces methods that do just that. By having these methods here, we can start writing changeset-optimized algorithms that should work already before we add any support for storing the metadata in the changesets. This commit also includes new debugp[12]copies commands and exercises them in test-copies.t. Differential Revision: https://phab.mercurial-scm.org/D5990

copies: filter out copies when target is not in destination manifest When chaining a series of commits that copied a file with a series that removed the destination file, we would still include the copy in the result. Similar to the previous patch, I have checked that `hg status --copies` is not affected by this bug, but I wouldn't be surprised if some commands are. Differential Revision: https://phab.mercurial-scm.org/D5989

copies: make _backwardrenames() filter out copies by destination As shown by the test case, _backwardrenames() doesn't filter by the matcher. It doesn't show up in `hg status --copies` because that only prints files changed between the two commits. I wouldn't be surprised if some other command that replies on pathcopies() was broken before this patch, but I haven't bothered to check other commands. Differential Revision: https://phab.mercurial-scm.org/D5988

copies: respect narrowmatcher in "parent -> working dir" case I don't know when this case happens and we don't seem to have tests for it, but let's fix it anyway since I happened to notice it. Differential Revision: https://phab.mercurial-scm.org/D5987

tests: add tests of pathcopies() I'm working on support for storing copy metadata in the changeset instead of in the filelog. When storing it in the changeset, it will obviously be efficient to get the copy metadata for all files in a single changeset, but it will be more expensive to get the copy metadata all revisions of a single file. Some algorithms will then need to be optimized differently. The first method I'm going to rewrite is pathcopies(). This commit adds many tests for pathcopies(), so we can run the tests with both old and new versions of the code, as well as with metadata stored in filelog or in changeset (later). They use the debugpathcopies command I recently added (with no tests when it was added). They show a few bugs and few cases of slightly weird behavior. I'll fix the bugs in the next few commits. Differential Revision: https://phab.mercurial-scm.org/D5986

uncommit: add config option to keep commit by default We have a Google-internal extension that keeps track of "review units" (like Phabricator reviews, or Gerrit's Change-Id). This information is stored outside of the commit. It is updated with rewrites. Every now and then we get reports from users who are confused because `hg uncommit` lost track of their review. Keeping the empty commit by default would reduce this confusion. It may also cause confusion about the empty commit. This patch adds a config option that lets us easily test both behaviors on our users. Differential Revision: https://phab.mercurial-scm.org/D5970

debugpathcopies: sort output so it's deterministic Differential Revision: https://phab.mercurial-scm.org/D5983

subrepo: use root-repo-relative path from `hg files` with ui.relative-paths=no The fix is to pass in a "subuipathfn" as we do everywhere else. Differential Revision: https://phab.mercurial-scm.org/D5978

subrepo: demonstrate broken `hg files` with ui.relative-paths=no Differential Revision: https://phab.mercurial-scm.org/D5977

py3: whitelist 5 new passing tests caught by buildbot Thanks to indygreg and durin42 recent patches. Differential Revision: https://phab.mercurial-scm.org/D5979

uncommit: inform user if the commit is empty after uncommit Differential Revision: https://phab.mercurial-scm.org/D5969

test: stabilize test-run-tests.t output We have reached a point where the duration in JSON reports of `test-run-tests.t` were greater or equal than 10 seconds, which doesn't match anymore the regex. For example here: https://ci.octobus.net/blue/organizations/jenkins/MercurialPy2/detail/MercurialPy2/276/pipeline ``` "diff": "", ? (re) - "end": "\s*[\d\.]{4,5}", ? (re) + "end": "10.040", "result": "skip", ? (re) ``` Instead of accepting more characters, I changed the regex to accept any number of digits before the `.` then 3 or 4 digits after. This way the regex is more precise (only one `.` is authorized and we can ensure that the precision doesn't change). Differential Revision: https://phab.mercurial-scm.org/D5966

test: stabilize test-remotefilelog-bgprefetch.t flaky output When running the test suite with multiple processes, we often get flaky outputs, like here: https://ci.octobus.net/job/MercurialPy2/274/console ``` - $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histidx - $TESTTMP/hgcache/master/packs/8f1443d44e57fec96f72fb2412e01d2818767ef2.histpack - $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.dataidx - $TESTTMP/hgcache/master/packs/f4d50848e0b465e9bfd2875f213044c06cfd7407.datapack + $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/1406e74118627694268417491f018a4a883152f0 + $TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/ef95c5376f34698742fe34f315fd82136f8f68c0 + $TESTTMP/hgcache/master/39/5df8f7c51f007019cb30201c49e884b46b92fa/69a1b67522704ec122181c0890bd16e9d3e7516a + $TESTTMP/hgcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/076f5e2225b3ff0400b98c92aa6cdf403ee24cca + $TESTTMP/hgcache/master/af/f024fe4ab0fece4091de044c58c9ae4233383a/bb6ccd5dceaa5e9dc220e0dad65e051b94f69a2c ``` Add some sleeps after `debugwaitonrepack` calls as it seems it needs some extra time to cleanup. Differential Revision: https://phab.mercurial-scm.org/D5960

merge with stable

color: change color of grep.rev label (BC) GNU grep uses green for line numbers as we do, but I sometimes get a bit confused when I do `hg grep --diff -n` and get both line numbers and revisions and don't know which one is which. A different colour can help.

color: give colours to the grep.inserted and grep.deleted labels I find the "bold" makes it stand out a bit more when the green is next to the line number. Note that these labels are applied before the grep.change label, which is now disabled. To get the old colour, users can restore [color] grep.change = green dim to their hgrc.

grep: give different labels to + and - symbols I find it more useful to give different colours to plus and minus, but it's difficult to do so if the default output uses the same label for both. The following augments the names of columns with some extra labels, akin to the diff.inserted and diff.deleted labels for the diff command. This is done by adding an extra label field to the columns tuples.

tests: bulk changes to avoid whitespace errors of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. This revision avoids "whitespace" errors of check-code.py below: - missing whitespace after "," - missing whitespace in expression - no whitespace around "=" for named parameters - wrong whitespace around "="

tests: replace "naked except clause" to avoid check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: bulk changes to avoid "omit superfluous pass" check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: replace imported module to avoid check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: bulk change to avoid "unneeded trailing ',' ..." error of check-code.py This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. "trailing ','" at the end of command option list in the code fragment of test-bundle2-format.t is still left as it is, in order to follow the style of command definitions in Mercurial core.

tests: define class in modern style to avoid check-code.py error This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts.

tests: use NO_CHECK_EOF as heredoc limit mark to omit checking code fragments This is a part of preparation to apply checking with check-code.py on code fragments embedded in *.t test scripts. "primes.py" embedded in test-highlight.t causes an error of check-commit.py below: don't use .next(), use next(...) But changing embedded primes.py is painful, because it is committed in test script, and affects hash IDs. On the other hand, primes.py itself is never executed in test script. Therefore, this patch uses NO_CHECK_EOF as heredoc limit mark in order to omit any checking on this code fragments.

tests: conditionalize msys path mangling in test-bundle.t This broke in 252cc56c9ff6 when the variables were printed one per line. The only reason I can think of is that MSYS knew the former string couldn't be a list of paths, but thinks this form is.

tests: glob over quoting differences in test-blackbox.t for Windows

chistedit: ensure a locale is set My paternal surname was showing incorrectly without this fix.

templatekw: make negrev return empty for wdir() and nullrev I considered just returning the same output that {rev} returns here, but {rev} also returns essentially gibberish: either an INT_MAX-kind of variable for wdir() or -1 for null. Since these are numbers that are intended to be used for calculations, and since the numbers for wdir() and -1 are not really very helpful for calculation (and worse, when used as a revision number -1 is equal to unhidden tip), I figured the most reasonable thing to do here is to just return nothing for negrev. This could potentially break scripts that are expecting to parse a nonempty integer out of a {negrev}, but that seems like a very remote concern at this juncture.

test-sqlitestore: run sqlite3 with no init file These tests were failing for me because I have a ~/.sqliterc that alters the default output. Tests should ignore any init file.

copies: return True instead of filename as it is expected to return boolean As the function documentation says this function should True if we hit a match and False otherwise. And I see that we are not using that returned filename anywhere. Differential Revision: https://phab.mercurial-scm.org/D5964

narrow: fix command name in error messsage Differential Revision: https://phab.mercurial-scm.org/D5982

templatekw: add a {negrev} keyword Revision numbers are getting much maligned for two reasons: they are too long in large repos and users get confused by their local-only nature. It just occurred to me that negative revision numbers avoid both of those problems. Since negative revision numbers change whenever the repo changes, it's much more obvious that they are a local-only convenience. Additionally, for the recent commits that we usually care about the most, negative revision numbers are always near zero. This commit adds a negrev templatekw to more easily expose negative revision numbers. It's not easy to reliably produce this output with existing keywords due to hidden commits while at the same time ensuring good performance.

tests: correct the remaining fallout from recent path style changes on Windows Per @martinvonz, `ui.slash` set by the test runner is now capable of playing a more active role.[1] I verified that both of these work by setting `ui.slash` to False, but these changes seem cleaner. The problem with check-perf-code.py was that the proper imports were not being whitelisted due to '\' vs '/'. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-February/128701.html

templates: adding a config() function for template customization This allows templates to be written such that users can customize them easily, or that they can be customized based on other configuration of the system. For enterprise deployments, we often have complex template aliases, and right now the only way individual users can customize those is by replacing the whole template alias (which means they won't get company-wide updates to it anymore, plus most users don't want to have to get a complex template right). With this change, they can just set a config option which feeds into our templates for common changes (e.g. whether to limit commit descriptions to the width of their terminal or not). To work around the issue of having to register the config options, I declared a dedicated section [templateconfig] for these options to be dynamically declared. They can still reference any other config option that's registered elsewhere. I only did string, bool and int at this time - list and date would add other complications with parsing the default so I'll leave that as an exercise to the reader :) Differential Revision: https://phab.mercurial-scm.org/D5959

changegroup: don't try to prune manifest nodes if not ellipses In non-ellipses case, the number of manifest nodes can be very big, and finding whether one of them can be pruned or not is very costly. For each node, we try to find the rev and the linkrev, which is expensive. Sending bit more manifest nodes seems better and much faster here. On our internal repository, this saves around 5 seconds on `hg tracked --addinclude <some_path>` on a narrow repo with ellipses disabled. Differential Revision: https://phab.mercurial-scm.org/D5782

branchmap: improve doc about BranchMapCache class It was confusing that we are having two different branchcache and BranchMapCache classes. The doc in BranchMapCache class was not much helpful to understand the difference. This patch improves the doc there. Differential Revision: https://phab.mercurial-scm.org/D5933

rust: itering less on MissingAncestors.bases for max() Instead of iterating on the whole `self.bases` each time to find its max, we keep the latter in a separate member attribute and keep it up to date in `add_bases()` On a perfdiscovery done on PyPy, with repos prepared with `contrib/discovery-helper.sh 50 100`, this gives a slight improvement (around 0.5% on wall time, but 10% on CPU) before: ! wall 0.172801 comb 0.180000 user 0.180000 sys 0.000000 (median of 541) after: ! wall 0.171798 comb 0.160000 user 0.160000 sys 0.000000 (median of 551) (perf command run time upped because of bigger variability during this test). Differential Revision: https://phab.mercurial-scm.org/D5945

rust: stop putting NULL_REVISION in MissingAncestors.bases As noted in initial review of MissingAncestors, adding NULL_REVISION in constructor in case the given `bases` is empty wasn't really useful, yet it's been kept for identity with the Python implementation Differential Revision: https://phab.mercurial-scm.org/D5944

rust: less set lookups in MissingAncestors using the return values of HashSet::remove(), we can factor pairs of `contains()/remove()` into a single `remove()`. On a perfdiscovery run done on the PyPy repository, prepared with contrib/discovery-helper.sh 50 100, I do get a modest improvement with this (mean of medians of three runs is better by 2%) Sample readings, before this change: ! wall 0.175609 comb 0.180000 user 0.180000 sys 0.000000 (median of 58) With this change: ! wall 0.171662 comb 0.180000 user 0.170000 sys 0.010000 (median of 60) Differential Revision: https://phab.mercurial-scm.org/D5943

rust: less set lookups in AncestorsIterator This uses the boolean return of `HashSet::insert()` to factor pairs of contains()/insert() into a single insert() On the mozilla-central repository (450k changesets), I get about a bit more than 10% better medians in perfancestors (taking the mean of three runs) Best run for parent changeset: ! wall 0.106474 comb 0.110000 user 0.110000 sys 0.000000 (median of 93) Best run for this changeset: ! wall 0.093191 comb 0.090000 user 0.090000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D5942