context: schedule file prefetch before comparing for cleanliness When using a system like remotefilelog, we can occasionally run into scenarios where the local content cache does not have the data we need to perform these comparisons. These will be fetched on-demand, but individually; if the remotefilelog server isn't extremely low latency, the runtime of the command becomes dominated by the multiple getfile requests for individual files. By performing the prefetch, we can download these files in bulk, and save server resources and many network roundtrips. Differential Revision: https://phab.mercurial-scm.org/D5532

manifest: convert a recursive function to iterative one using stacks I am debugging a memory issue from yesterday where `hg update` goes upto taking 22GB of memory on our internal treemanifest repository. This is an interesting function and I saw memory consumption increasing while this function was running. It's sometimes hard to understand a recursive function and also the profile won't show you actual operations which took time, rather it will show you the function again and again in profile. I am yet to notice any memory consumption decrease with this patch, but I believe this will help like in making this a generator. Differential Revision: https://phab.mercurial-scm.org/D5413

obsutil: fix the issue5686 While traversing the obsolescence graph to find the successors sets of csets: In its 4th case (read comments of obsutil.successorssets to see all 4 cases) where we know successors sets of all direct successors of CURRENT, we were just missing a condition to filter out the case when a cset is pruned. And without this condition (that this patch added) it was making a whole successor set to [] just because of one pruned marker. For e.g:if following is the successors set of a cset A: A -> [a, b, c] if we prune c, we expect A's successors set to be [a, b] but you would get: A -> [] So this patch make sure that we calculate the right successorsset of csets considering the pruned cset (in split case). Differential Revision: https://phab.mercurial-scm.org/D5474

histedit: add user input to warning message on editing tagged commits This is a follow-up patch to D5489. Now, the user will be able to input yes/no(default) on the warning message. Initially, it was the sleep of 1s and histedit window opens. Changes were made as suggested by @mharbison72 and @yuja. Differential Revision: https://phab.mercurial-scm.org/D5494

sparse: fix debugrebuilddirsate when narrow extension is enabled dirstate wrapping in narrow extension converts the manifest object to a list. So let's assume we get a list of files in sparse extension. Differential Revision: https://phab.mercurial-scm.org/D5481

tests: show that debugrebuilddirstate is broken with narrow+sparse This adds a test to show that debugrebuildirstate is broken when both narrow and sparse extensions are enabled. Differential Revision: https://phab.mercurial-scm.org/D5480

sparse: don't enable on clone if it was a narrow clone When both sparse and narrow extensions are enabled and we do a narrow clone, sparse enables itself because it reads --include flag and thinks that user is trying to do a sparse clone. This patch changes that behavior, and now if both extensions are enabled and user passes `--narrow`, then the includes and excludes won't be considered as part of sparse profile. Differential Revision: https://phab.mercurial-scm.org/D5479

tests: add tests for narrow clone when both sparse and narrow are enabled This patch adds tests for doing a narrow clone when client has both sparse and narrow enabled. As shown by this patch, we can see that the narrow clone has sparse enabled which we does not want here. Upcoming patch will fix this. Also while writing tests, I realized we don't have a flag to list the current sparse profile. Differential Revision: https://phab.mercurial-scm.org/D5478

progress: check what type of progress bar to use only once per topic This seems to have sped up `hg perfprogress` from 1.78 s to 1.41 s. Differential Revision: https://phab.mercurial-scm.org/D5530

progress: split up _print() method in bar-updating and debug-printing I just thought this was clearer, but it turned out to also simplify the next patch. This seems to have sped up `hg perfprogress` from 1.85 s to 1.78 s. Differential Revision: https://phab.mercurial-scm.org/D5529

progress: move cached debug flag from progress.progbar to scmutil.progress It's simpler this way. One possible drawback (and a possisble advantage) is that we now check the debug flag once per topic, so processes that generate new topics all the time will still check the flag frequently. Differential Revision: https://phab.mercurial-scm.org/D5528

progress: write ui.progress() in terms of ui.makeprogress() I think ui.makeprogress() should be the preferred interface and we should deprecate ui.progress(). All in-core callers already use ui.makeprogress(). Moving the logic to the scmutil.progress() will let us make further improvements. This seems to have sped up `hg perfprogress` from 1.92 s to 1.85 s, perhaps because we now skip the indirection of updating the progress bar via ui.progress(). Differential Revision: https://phab.mercurial-scm.org/D5527

testrunner: avoid capturing a regex group we don't care about Differential Revision: https://phab.mercurial-scm.org/D5536

remotefilelog: add newlines to ui.log() invocations The comment at the top of ui.log() says that the message should be a newline-terminated string. When using the blackbox logger, if the string does not end in a newline, the appearance in the file gets quite messy. This sometimes leaves a string with just the newline as the message, these are logged by the blackbox logger just fine. I don't know what other loggers do when logging structured data and a message that is just the newline. Differential Revision: https://phab.mercurial-scm.org/D5533

xdiff: don't attempt to use fuzzer inputs larger than 100k This is the recommended approach from [0], and limiting the input was suggested in https://github.com/google/oss-fuzz/issues/2076 when discussing our broken coverage build. 0: https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md#custom-libfuzzer-options-for-clusterfuzz Differential Revision: https://phab.mercurial-scm.org/D5525

tests: support passing testcase after .t paths that have path separators This probably could have been implemented by changing the regex above this bit of code, but I wasn't sure if it would end up handling various OSes correctly, so I decided to go with this version instead. Previously: $ tests/run-tests.py tests/test-ssh.t -l running 2 tests using 2 parallel processes .. # Ran 2 tests, 0 skipped, 0 failed. $ tests/run-tests.py tests/test-ssh.t#sshv1 -l running 0 tests using 0 parallel processes # Ran 0 tests, 0 skipped, 0 failed. Now: $ tests/run-tests.py tests/test-ssh.t -l running 2 tests using 2 parallel processes .. # Ran 2 tests, 0 skipped, 0 failed. $ tests/run-tests.py tests/test-ssh.t#sshv1 -l running 1 tests using 1 parallel processes . # Ran 1 tests, 0 skipped, 0 failed. Differential Revision: https://phab.mercurial-scm.org/D5535

progress: document progress.debug config option I think it was not spotted by test-check-config.t that we had not documented it because no caller refers to the ui object simply as "ui" (it was either "self.ui" or just "self"). Differential Revision: https://phab.mercurial-scm.org/D5526

tests: migrate test-wireproto-serverreactor.py to our internal CBOR This leaves the only client of thirdparty.cbor as test-cbor.py, which appears to be testing the behavior of cborutil against cbor. We should figure out some appropriate test strings and drop thirdparty.cbor, but that's a mission for another day. Differential Revision: https://phab.mercurial-scm.org/D5522

state: update comment about use of CBOR We use our internal cbor library, not the vendored one in thirdparty. Differential Revision: https://phab.mercurial-scm.org/D5518

rust-ancestors: adjust branches and inline comments per previous change Now the top-level "if" can be read as both_visit|revs_visit|bases_visit.

rust-ancestors: remove unreachable conditions from missing_ancestors()

rust-ancestors: duplicate loop that visits parents of revs/bases As the inline comment says, it can't be cleanly implemented in Rust. It's better to duplicate the code instead of inserting "if"s. The loop will be cleaned up by future commits.

rust-ancestors: adjust indent level to make next change easier to follow

rust-cpython: remove invalid __package__ attribute Since mercurial.rustext is a package, its __package__ should be, if set, "mercurial.rustext". AFAIK, we don't have to set this attribute manually as the rustext module will be imported by the system importer. https://stackoverflow.com/a/21233334/10435339

merge with stable

tests: use more portable flags in test-fix.t `head --lines N` is known to not exist on OSX. Differential Revision: https://phab.mercurial-scm.org/D5517

amend: add -D/--currentdate option It bumps the date field even if there's no other change. The help text is copied from commands.graft(). Unlike graft, --date with --currentdate is disallowed, which I think is saner behavior.

amend: add boolean to ignore date-only change I want to add -D/--currentdate option, which should behave slightly different from the update-timestamp option.

amend: refactor commit date handling There's no need to parse a date tuple. Just do it for user option.

test-amend: remove uninteresting fields from log output to deduplicate tests We aren't testing the behavior of obsolescence-based amend.

pull: use opts.get('bookmark') instead of opts['bookmark'] This is done because at places in hgsubversion, we call the function directly. I expect there might be more instances in extensions out there which calls commands.push() directly. So let's not require explicitly passing of bookmark value. The use of opts['bookmark'] was introduced in bad05a6afdc89cc58a2af320698ab29bd8de62d4. Differential Revision: https://phab.mercurial-scm.org/D5484

amend: add config option to update time to current in hg amend (issue5828) The given config option i.e. `rewrite.update-timestamp` updates date to current when //True//. However when only date is to be updated to current with the working directory clean and no other attributes changing then it does not amend as stated in issue 5828. Further when `--date` flag is specified along with the new config option then `--date` is given priority over the config option. Differential Revision: https://phab.mercurial-scm.org/D5491

tests: correct version check in clientreactor test Experimentally, `(3, 6, 3)` is less than `sys.version_info` on an actual Python 3.6.3 installation. Let's just check `< (3, 6, 4)` instead of `<= (3, 6, 3)` so we stop seeing this bogus failure. Differential Revision: https://phab.mercurial-scm.org/D5502

py3: buildbot spotted more passing tests Differential Revision: https://phab.mercurial-scm.org/D5501

tests: add lots of b prefix goo to test-fastannotate-revmap.py All the paths are now bytes, so now things work correctly. # skip-blame just bytes/str issues in this test Differential Revision: https://phab.mercurial-scm.org/D5500

fastannotate: add a missing b prefix Spotted while debugging a test failure, but this wasn't the problem. # skip-blame b prefix Differential Revision: https://phab.mercurial-scm.org/D5499

rust-cpython: using the new bindings from Python The Python callers detect if we have cpython or direct-ffi bindings and fallback to the Python implementation if none is present. This intermediate state allows to compare the three possibilities. Differential Revision: https://phab.mercurial-scm.org/D5442

rust-cpython: binding for LazyAncestors The `mercurial.rustext.ancestor` module will not in the foreseeable future be a drop-in replacement for the pure `mercurial.ancestor`, because the Rust variants take the index at instantiation whereas the Python ones take a parents function. From the Python side, using the index from `ancestor` would leak internal details out of `mercurial.revlog`, and that's unwanted. Therefore, given that classes defined in `rust-cpython` have the same names in both language, we keep the Rust naming convention (CamelCase). Eventually, though, the ancestor module can be placed under control of `mercurial.policy`, but it will still be up to `revlog` to be aware of that and play the role of a factory for instantiation. Differential Revision: https://phab.mercurial-scm.org/D5441

discovery: move common heads computation inside partialdiscovery object This remove one of the private attribute access. In additions, head tracking and computation is a typical operation we can speed up using Rust.

discovery: introduce a partialdiscovery object This object will ultimately gather the data about common, undecided and missing revs in a single place and deal with most graph related computations. The goal is both to clarify the algorithm and to help provides a simple and clear API that can be reimplemented in Rust. For now, we only moved the `common` set in the object. In this commit, some direct access to the "private" `disco._common` attribute persist. They have not been removed yet because we won't need to expose a full API identical to `incrementalmissingancestors` and it seems simpler to access the attribute directly until the replacement is in place.

discovery: move handling of sampling special case inside sampling function The handling of cases where the number of revisions to sample is smaller than the sample size can be moved with the sample function themselves. This simplifies main logic, preparing a coming refactoring.

discovery: minor fix to some conditionals Since `size` is the upper limit of the sample, we should include it in the check. Otherwize the `more` variable will be zero and the sampling will be useless

delta: reuse _findsnapshot call from previous stage Two different stage of the sparse-revlog logic needs the _findsnapshot data. To avoid recomputing it twice, make it possible to reuse the first computation in the second step. example affected manifest write before: 0.067141s after: 0.064252s (-5%) (total gain since start of series: 95%)

delta: have a native implementation of _findsnapshot The function might traverse a lot of revision, a native implementation get significantly faster. example affected manifest write before: 0.114989 after: 0.067141 (-42%)

transaction: do not overwrite atomic-temp files on error Even though the original files can be restored from the backup, it should be better to not write back a temporary file if we know it can be corrupted.

help: fix typo

rust: use 'impl Trait' in method argument of AncestorsIterator I just didn't know it's stabilized. Let's switch to new convenient syntax. Differential Revision: https://phab.mercurial-scm.org/D5451

rust: use .rev() for reverse range Differential Revision: https://phab.mercurial-scm.org/D5450