progress: specify updatebar() function by constructor argument This makes it easy for ui extensions to intercept progress messages. It also seems slightly nicer in that scmutil.progress doesn't touch ui internals.

progress: change _updatebar() to take parameters as arguments I want to move updatebar() back to ui. See the next patch for why.

archival: construct progress helper through ui.makeprogress() No idea why we didn't.

help: document rewrite.backup-bundle option

repair: move ui.history-editing-backup to [rewrite] section Since we have the "rewrite" section for general history-editing options, the backup option should be there.

revlog: use separate variables to track version flags It wasn't obvious to me that "versionflags" is used both to define the default version+features value for new revlogs and to track the value read from a revlog. We rename the former use and add explicit assignment of "versionflags" later to differentiate between the two. Differential Revision: https://phab.mercurial-scm.org/D5564

revlog: inline opener options logic into _loadindex() We always call _loadindex() during __init__. But we also call _loadindex() as part of censorrevision(). Before, when reloading the index during censorrevision(), we would lose the configured mmapindexthreshold setting from the opener. By inlining the logic in _loadindex(), we ensure that opener options are always respected when loading the index. Differential Revision: https://phab.mercurial-scm.org/D5563

revlog: store mmaplargeindex as an instance attribute This makes it more consistent with everything else. It also makes it possible to move the opener/features processing into _loadindex(). Differential Revision: https://phab.mercurial-scm.org/D5562

revlog: always enable generaldelta on version 2 revlogs This commit starts the process of diverging version 2 revlogs from version 1 revlogs. generaldelta is a useful feature and has been enabled by default for ages. I can't think of a good reason why the feature should be disabled. Yes, it is true changelogs today don't have generaldelta enabled. But that's because they don't have delta chains enabled, so generaldelta makes no sense there. This commit makes generaldelta always enabled on version 2 revlogs. As part of this, one-off code in changelog.py mucking with revlog.version had to be made conditional on the revlog version, as we don't want to change revlog feature flags on version 2 revlogs. The fact this code exists is horrible and stems from revlog options being shared by the opener. We probably want a better API here. But that can wait for another patch. Differential Revision: https://phab.mercurial-scm.org/D5561

revlog: rename v to versionflags Single letter variables are harder to read. Differential Revision: https://phab.mercurial-scm.org/D5560

revlog: always process opener options I'm not sure when ``opener.options`` would ever be explicitly set to None. It is definitely not possible to construct a repo this way because ``localrepo.resolvestorevfsoptions()`` always returns a dict and ``localrepo.makelocalrepository()`` always sets ``opener.options`` to this value. Because we always execute this code now, if options are empty we defaulted to creating version 0 revlogs. So we had to change the code slightly to fall back to the default revlog version and flags. As astute reader will note that it is not possible to create version 0 revlogs now. However, I don't think it was possible before, as this required ``opener.options`` being unset, which I don't think was possible. I suspect this means our test coverage for version 0 revlog repositories is possibly non-existent! Since I don't see a config option to disable revlog v1, I'm not even sure if we had a way to create new repos with version 0 revlogs! Who knows. Differential Revision: https://phab.mercurial-scm.org/D5559

internals: minor rewriting of revlogs documentation In preparation for formalizing revlog version 2. Differential Revision: https://phab.mercurial-scm.org/D5558

tests: make HGCATAPULTSERVERPIPE imply HGTESTCATAPULTSERVERPIPE I had attempted to do this before, but missed this case. This makes it so that one can do the following to get catapult traces that include both the .t test name (and non-hg commands run by that .t test) *and* the hg-internal tracing, in one trace: HGCATAPULTSERVERPIPE=/tmp/catapult.pipe run-tests.py <args> Without this change, we need to specify both `HG{,TEST}CATAPULTSERVERPIPE`; if we specify just the TEST one, we only get the .t tests (no hg-internals), which is working as intended. If we specify the non-TEST one, we only get the hg-internals (not the rest of the .t test), which was not intended. If you want to restore the previous behavior (just hg internals, not the stuff from the .t tests), run like: HGTESTCATAPULTSERVERPIPE=/dev/null \ HGCATAPULTSERVERPIPE=/tmp/catapult.pipe \ run-tests.py <args> Differential Revision: https://phab.mercurial-scm.org/D5569

tests: add test for warning on histedit with tagged commits This is a follow-up patch to 7b7e081f8954 (D5494). Differential Revision: https://phab.mercurial-scm.org/D5552

extdiff: move external tool command line building into separate function

test: explicit a rebase source in test-rebase-collapse.t At the time of the rebase, the current revision is `null` so this rebase should fail. However, a current bug in formatspec makes it pass, we fix rebase call of the "affected" test to express its actual intent before we fix the actual bug in a later changeset.

tags: join string that's unnecessarily split across lines Differential Revision: https://phab.mercurial-scm.org/D5574

dispatch: add newline after ui.log "ui blocked ms" message Differential Revision: https://phab.mercurial-scm.org/D5573

remotefilelog: fix {file_copies} template keyword remotefilelog overrides templatekw.getrenamedfn(). I changed the return type of that function in ec37df9042f9 (templatekw: make getrenamed() return only filename, not nodeid, 2018-03-26). So we need to change remotefilelog too. The function is used by the {file_copies} template keyword and by `hg log --copies`. Differential Revision: https://phab.mercurial-scm.org/D5576

tests: demonstrate broken {file_copies} template with remotefilelog Differential Revision: https://phab.mercurial-scm.org/D5575

contrib: update window title when possible in perf-revlog-write-plot.py This is useful when comparing multiple graphs.

histedit: add newline after ui.log "# acttions to histedit" message Differential Revision: https://phab.mercurial-scm.org/D5572

rust-cpython: bindings for MissingAncestors The exposition is rather straightforward, except for the remove_ancestors_from() method, which forces us to an inefficient conversion between Python sets and Rust HashSets. Two alternatives are proposed in comments: - changing the inner API to "emit" the revision numbers to discard this would be a substantial change, and it would be better only in the cases where there are more to retain than to discard - mutating the Python set directly: this would force us to define an abstract `RevisionSet` trait, and implement it both for plain `HashSet` and for a struct enclosing a Python set with the GIL marker `Python<'p>`, also a non trivial effort. The main (and seemingly only) caller of this method being `mercurial.setdiscovery`, which is currently undergoing serious refactoring, it's not clear whether these improvements would be worth the effort right now, so we're leaving it as-is. Also, in `get_bases()` (will also be used by `setdiscovery`), we'd prefer to build a Python set directly, but we resort to returning a tuple, waiting to hear back from our PR onto rust-cpython about that Differential Revision: https://phab.mercurial-scm.org/D5550

rust-cpython: generalised conversion function Because `hg::ancestors::MissingAncestors` has methods taking some `HashSet<Revision>` besides `impl IntoIterator<Item = Revision>` as parameters we'll need the more generic `rev_pyiter_collect()` function to also build these Differential Revision: https://phab.mercurial-scm.org/D5549

rust-cpython: style consistency leftovers In particular, during review of `LazyAncestors` bindings, most `match` statements for error conversion have been replaced by higher level methods of `Result` and most personal insecurity comments have been removed. This makes it more systematic. Differential Revision: https://phab.mercurial-scm.org/D5548

rust-cpython: consistency in use of hg-core constructs While not everybody likes the `CoreLazy` and `CoreIterator` aliases, it's better not to mix them with direct references. Note: it's quite possible in the future that these would stop being exposed at the top of the `hg` crate Differential Revision: https://phab.mercurial-scm.org/D5547

rust-cpython: rustdoc improvements By default, `cargo doc` builds the documentation for public constructs only, so we make public those that can. Since `cindex` is not safe, we keep it private. Unfortunately, the macro syntax of rust-cpython doesn't allow us to document the classes directly, so we resort to do that at the module level. Differential Revision: https://phab.mercurial-scm.org/D5546

fuzz: stop building Python in the Makefile $OUT/sanpy is now built by the build.sh in oss-fuzz, which simplifies the setup here considerably. Differential Revision: https://phab.mercurial-scm.org/D5553

py3: add a test to whitelist caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5555

tests: de-flake test-narrow-share.t by making dirstate predictable test-narrow-share.t was sometimes (~0.5% on my machine) failing like this: @@ -61,7 +61,7 @@ A d3/g $ hg -R main debugdirstate --no-dates n 644 2 set d1/f - n 644 2 unset d3/f + n 644 2 set d3/f a 0 -1 unset d3/g n 644 2 set d5/f n 644 2 set d7/f The timestamp for d3/f would get set if it was determined at some point that it was clean. That check is usually done when the user runs `hg st`. We don't do that before the failure in the test case, but it happens at the end of the `hg clone` call. So if the file system's time happens to roll over after the clone's working copy has been written, but before its (final) dirstate has been written, we can end up with a set timestamp there. This patch makes it consistent by sleeping for 2 seconds so the timestamp gets reliably set. Differential Revision: https://phab.mercurial-scm.org/D5568

convert: don't drop commits that are empty in the source when using --filemap I ran into this when using `hg lfconvert --to-normal` (which uses the filemap class internally), and saw that commits with nothing but a branch change were dropped. We could put in an option that only lfconvert uses internally. But silently dropping anything other than a commit where all changes were excluded seems unintended. For example, there's a message in mercurial_sink.putcommit() if it drops an empty commit. (And the reason that isn't kicking in here is because lfconvert isn't passing --filemap, so the self.filemapmode conditional there is always False.) The naive change of `return not files` broke test-convert-filemap.t, so this is a little more elaborate than needed for converting from largefiles.

convert: add missing numcommits() override to hg sources Otherwise, the progressbar in converter.walktree() gets None as its total during the initial scan. (Though that seems harmless, and there are other foreign vcs sources without this.)

testrunner: make `-j100 --runs-per-test=100 test-foo.t` use 100 jobs It would use only 1 job since 1039404c5e1d (run-tests: print number of tests and parallel process count, 2018-10-13), but I'm pretty sure that was unintentional. Differential Revision: https://phab.mercurial-scm.org/D5557

narrow: use merge.ACTION_GET instead of duplicating 'g' constant As suggested by Yuya. Differential Revision: https://phab.mercurial-scm.org/D5512

narrow: drop explicit dirstate write The dirstate is written when the wlock is released, so we don't need to write it explicitly in updateworkingcopy(). I don't know why I put it there in the first place (tests pass without it even in the commit that introduced it). Differential Revision: https://phab.mercurial-scm.org/D5515

histedit: fix --continue and --abort when curses is enabled I overlooked this when I did the initial import. Differential Revision: https://phab.mercurial-scm.org/D5540

graft: abort if --date/user specified with --currentdate/currentuser (BC) They are conflicting options. Let's explicitly ban such combination. Note that this means you need "--no-currentdate --date" to override --currentdate specified in aliases/defaults.

amend: turn currentdate into tri-state bool so config option can be negated Otherwise, rewrite.update-timestap couldn't be overridden by --no-currentdate. FWIW, which do we like? --no-currentdate vs --no-current-date. If the latter is preferred, maybe we can add "graft --current-date" and deprecate "--currentdate".

discovery: improve partial discovery documentation The older documentation could be improved.

discovery: add a simple `addinfo` method The method can directly process a sample result. This makes the main code simpler to follow.

discovery: move missing tracking inside the partialdiscovery object This is the final set that we need to track to have a fully up to date information within the object.

discovery: add a `iscomplete` method to the `partialdiscovery` object The method is used by higher level logic to know if the initial discovery question has been answered.

discovery: update undecided from common directly within partialdiscovery The object knows when there are changes in the "common" set, so it is a good place to update the undecided set.

discovery: move undecided set on the partialdiscovery To initialize it, we need to know the discovery target. This commit only move the set on the `partialdiscovery` object, later changeset will take advantage of it.

histedit: use context manager for locks Differential Revision: https://phab.mercurial-scm.org/D5542

histedit: drop unused constructor arguments (API) All in-tree callers pass a repo and nothing else. Differential Revision: https://phab.mercurial-scm.org/D5541

phabricator: teach {phabreview} to work without --amend There's some possibility of this being wrong (e.g. if the url config was changed). But commit messages can be amended too, so that isn't bulletproof either. Having something seems better than just returning None.

phabricator: warn if unable to amend, instead of aborting after posting There was a divergence in behavior here between obsolete and strip based amending. I first noticed the abort when testing outside of the test harness, but then had trouble recreating it here after reverting the code changes. It turns out, strip based amend was successfully amending the public commit after it was posted! It looks like the protection is in the `commit --amend` command, not in the underlying code that it calls. I considered doing a preflight check and aborting. But the locks are only acquired at the end, if amending, and this is too large a section of code to be wrapped in a maybe-it's-held-or-not context manager for my tastes. Additionally, some people do post-push reviews, and amending is the default behavior, so they shouldn't see a misleading error message. The lack of a 'Differential Revision' entry in the commit message breaks a {phabreview} test, so it had to be partially conditionalized.

discovery: re-adjust a conditional wrongly changed In 71b0db4fa027 we updated this conditional to `<=`. As Yuya Nishihara pointed out, this was wrong.

cleanup: stop including thirdparty.cbor in builds Differential Revision: https://phab.mercurial-scm.org/D5524

tests: get access to thirdparty.cbor without requiring it to be installed This makes the test a noop when run from a tarball, but in return we can stop shipping a library we don't use. Differential Revision: https://phab.mercurial-scm.org/D5523

tests: add simplestorerepo to test-check-interfaces.py I'm not thrilled with this, but it'll help avoid future bitrot. Differential Revision: https://phab.mercurial-scm.org/D5521

simplestorerepo: migrate to in-hg CBOR code This is the only use of thirdparty.cbor outside of a test-* file, so it felt worthwhile to clean it up. Differential Revision: https://phab.mercurial-scm.org/D5520

simplestorerepo: minimal changes required to get this mostly working again I was going to change this code's use of CBOR to use our in-house CBOR code, but discovered it's been broken for a while. This messy change gets it back to a point where it mostly works, I think roughly as well as it ever did. Should we keep this and fix it up the rest of the way, or dump it in favor of the sqlite store? Would this be a good jumping-off point for some sort of union store that could facilitate a cleanup in remotefilelog? Differential Revision: https://phab.mercurial-scm.org/D5519

perfrevlogwrite: use progress helper on modern hg Differential Revision: https://phab.mercurial-scm.org/D5372

merge: make local file storage in the .hg/merge directory extensible This is similar to remotefilelog's 'getlocalkey' method, which must be overridden by systems which rely on full path names for access control purposes. Differential Revision: https://phab.mercurial-scm.org/D5534

context: schedule file prefetch before comparing for cleanliness When using a system like remotefilelog, we can occasionally run into scenarios where the local content cache does not have the data we need to perform these comparisons. These will be fetched on-demand, but individually; if the remotefilelog server isn't extremely low latency, the runtime of the command becomes dominated by the multiple getfile requests for individual files. By performing the prefetch, we can download these files in bulk, and save server resources and many network roundtrips. Differential Revision: https://phab.mercurial-scm.org/D5532

manifest: convert a recursive function to iterative one using stacks I am debugging a memory issue from yesterday where `hg update` goes upto taking 22GB of memory on our internal treemanifest repository. This is an interesting function and I saw memory consumption increasing while this function was running. It's sometimes hard to understand a recursive function and also the profile won't show you actual operations which took time, rather it will show you the function again and again in profile. I am yet to notice any memory consumption decrease with this patch, but I believe this will help like in making this a generator. Differential Revision: https://phab.mercurial-scm.org/D5413

obsutil: fix the issue5686 While traversing the obsolescence graph to find the successors sets of csets: In its 4th case (read comments of obsutil.successorssets to see all 4 cases) where we know successors sets of all direct successors of CURRENT, we were just missing a condition to filter out the case when a cset is pruned. And without this condition (that this patch added) it was making a whole successor set to [] just because of one pruned marker. For e.g:if following is the successors set of a cset A: A -> [a, b, c] if we prune c, we expect A's successors set to be [a, b] but you would get: A -> [] So this patch make sure that we calculate the right successorsset of csets considering the pruned cset (in split case). Differential Revision: https://phab.mercurial-scm.org/D5474

histedit: add user input to warning message on editing tagged commits This is a follow-up patch to D5489. Now, the user will be able to input yes/no(default) on the warning message. Initially, it was the sleep of 1s and histedit window opens. Changes were made as suggested by @mharbison72 and @yuja. Differential Revision: https://phab.mercurial-scm.org/D5494

sparse: fix debugrebuilddirsate when narrow extension is enabled dirstate wrapping in narrow extension converts the manifest object to a list. So let's assume we get a list of files in sparse extension. Differential Revision: https://phab.mercurial-scm.org/D5481

tests: show that debugrebuilddirstate is broken with narrow+sparse This adds a test to show that debugrebuildirstate is broken when both narrow and sparse extensions are enabled. Differential Revision: https://phab.mercurial-scm.org/D5480

sparse: don't enable on clone if it was a narrow clone When both sparse and narrow extensions are enabled and we do a narrow clone, sparse enables itself because it reads --include flag and thinks that user is trying to do a sparse clone. This patch changes that behavior, and now if both extensions are enabled and user passes `--narrow`, then the includes and excludes won't be considered as part of sparse profile. Differential Revision: https://phab.mercurial-scm.org/D5479

tests: add tests for narrow clone when both sparse and narrow are enabled This patch adds tests for doing a narrow clone when client has both sparse and narrow enabled. As shown by this patch, we can see that the narrow clone has sparse enabled which we does not want here. Upcoming patch will fix this. Also while writing tests, I realized we don't have a flag to list the current sparse profile. Differential Revision: https://phab.mercurial-scm.org/D5478

histedit: crashing with a more useful error message on empty defaultrev Before this, `hg --config histedit.defaultrev= histedit` would crash with File "destutil.py", line 385, in desthistedit if revs: UnboundLocalError: local variable 'revs' referenced before assignment Differential Revision: https://phab.mercurial-scm.org/D5543

progress: check what type of progress bar to use only once per topic This seems to have sped up `hg perfprogress` from 1.78 s to 1.41 s. Differential Revision: https://phab.mercurial-scm.org/D5530

progress: split up _print() method in bar-updating and debug-printing I just thought this was clearer, but it turned out to also simplify the next patch. This seems to have sped up `hg perfprogress` from 1.85 s to 1.78 s. Differential Revision: https://phab.mercurial-scm.org/D5529

progress: move cached debug flag from progress.progbar to scmutil.progress It's simpler this way. One possible drawback (and a possisble advantage) is that we now check the debug flag once per topic, so processes that generate new topics all the time will still check the flag frequently. Differential Revision: https://phab.mercurial-scm.org/D5528

progress: write ui.progress() in terms of ui.makeprogress() I think ui.makeprogress() should be the preferred interface and we should deprecate ui.progress(). All in-core callers already use ui.makeprogress(). Moving the logic to the scmutil.progress() will let us make further improvements. This seems to have sped up `hg perfprogress` from 1.92 s to 1.85 s, perhaps because we now skip the indirection of updating the progress bar via ui.progress(). Differential Revision: https://phab.mercurial-scm.org/D5527

testrunner: avoid capturing a regex group we don't care about Differential Revision: https://phab.mercurial-scm.org/D5536

remotefilelog: add newlines to ui.log() invocations The comment at the top of ui.log() says that the message should be a newline-terminated string. When using the blackbox logger, if the string does not end in a newline, the appearance in the file gets quite messy. This sometimes leaves a string with just the newline as the message, these are logged by the blackbox logger just fine. I don't know what other loggers do when logging structured data and a message that is just the newline. Differential Revision: https://phab.mercurial-scm.org/D5533

xdiff: don't attempt to use fuzzer inputs larger than 100k This is the recommended approach from [0], and limiting the input was suggested in https://github.com/google/oss-fuzz/issues/2076 when discussing our broken coverage build. 0: https://github.com/google/oss-fuzz/blob/master/docs/new_project_guide.md#custom-libfuzzer-options-for-clusterfuzz Differential Revision: https://phab.mercurial-scm.org/D5525

tests: support passing testcase after .t paths that have path separators This probably could have been implemented by changing the regex above this bit of code, but I wasn't sure if it would end up handling various OSes correctly, so I decided to go with this version instead. Previously: $ tests/run-tests.py tests/test-ssh.t -l running 2 tests using 2 parallel processes .. # Ran 2 tests, 0 skipped, 0 failed. $ tests/run-tests.py tests/test-ssh.t#sshv1 -l running 0 tests using 0 parallel processes # Ran 0 tests, 0 skipped, 0 failed. Now: $ tests/run-tests.py tests/test-ssh.t -l running 2 tests using 2 parallel processes .. # Ran 2 tests, 0 skipped, 0 failed. $ tests/run-tests.py tests/test-ssh.t#sshv1 -l running 1 tests using 1 parallel processes . # Ran 1 tests, 0 skipped, 0 failed. Differential Revision: https://phab.mercurial-scm.org/D5535

progress: document progress.debug config option I think it was not spotted by test-check-config.t that we had not documented it because no caller refers to the ui object simply as "ui" (it was either "self.ui" or just "self"). Differential Revision: https://phab.mercurial-scm.org/D5526

tests: migrate test-wireproto-serverreactor.py to our internal CBOR This leaves the only client of thirdparty.cbor as test-cbor.py, which appears to be testing the behavior of cborutil against cbor. We should figure out some appropriate test strings and drop thirdparty.cbor, but that's a mission for another day. Differential Revision: https://phab.mercurial-scm.org/D5522

state: update comment about use of CBOR We use our internal cbor library, not the vendored one in thirdparty. Differential Revision: https://phab.mercurial-scm.org/D5518

rust-ancestors: adjust branches and inline comments per previous change Now the top-level "if" can be read as both_visit|revs_visit|bases_visit.

rust-ancestors: remove unreachable conditions from missing_ancestors()

rust-ancestors: duplicate loop that visits parents of revs/bases As the inline comment says, it can't be cleanly implemented in Rust. It's better to duplicate the code instead of inserting "if"s. The loop will be cleaned up by future commits.

rust-ancestors: adjust indent level to make next change easier to follow

rust-cpython: remove invalid __package__ attribute Since mercurial.rustext is a package, its __package__ should be, if set, "mercurial.rustext". AFAIK, we don't have to set this attribute manually as the rustext module will be imported by the system importer. https://stackoverflow.com/a/21233334/10435339

merge with stable

Added signature for changeset 197f092b2cd9

Added tag 4.8.2 for changeset 197f092b2cd9

tests: use more portable flags in test-fix.t `head --lines N` is known to not exist on OSX. Differential Revision: https://phab.mercurial-scm.org/D5517

amend: add -D/--currentdate option It bumps the date field even if there's no other change. The help text is copied from commands.graft(). Unlike graft, --date with --currentdate is disallowed, which I think is saner behavior.

amend: add boolean to ignore date-only change I want to add -D/--currentdate option, which should behave slightly different from the update-timestamp option.

amend: refactor commit date handling There's no need to parse a date tuple. Just do it for user option.

test-amend: remove uninteresting fields from log output to deduplicate tests We aren't testing the behavior of obsolescence-based amend.

pull: use opts.get('bookmark') instead of opts['bookmark'] This is done because at places in hgsubversion, we call the function directly. I expect there might be more instances in extensions out there which calls commands.push() directly. So let's not require explicitly passing of bookmark value. The use of opts['bookmark'] was introduced in bad05a6afdc89cc58a2af320698ab29bd8de62d4. Differential Revision: https://phab.mercurial-scm.org/D5484

amend: add config option to update time to current in hg amend (issue5828) The given config option i.e. `rewrite.update-timestamp` updates date to current when //True//. However when only date is to be updated to current with the working directory clean and no other attributes changing then it does not amend as stated in issue 5828. Further when `--date` flag is specified along with the new config option then `--date` is given priority over the config option. Differential Revision: https://phab.mercurial-scm.org/D5491

tests: correct version check in clientreactor test Experimentally, `(3, 6, 3)` is less than `sys.version_info` on an actual Python 3.6.3 installation. Let's just check `< (3, 6, 4)` instead of `<= (3, 6, 3)` so we stop seeing this bogus failure. Differential Revision: https://phab.mercurial-scm.org/D5502

py3: buildbot spotted more passing tests Differential Revision: https://phab.mercurial-scm.org/D5501

tests: add lots of b prefix goo to test-fastannotate-revmap.py All the paths are now bytes, so now things work correctly. # skip-blame just bytes/str issues in this test Differential Revision: https://phab.mercurial-scm.org/D5500

fastannotate: add a missing b prefix Spotted while debugging a test failure, but this wasn't the problem. # skip-blame b prefix Differential Revision: https://phab.mercurial-scm.org/D5499

rust-cpython: using the new bindings from Python The Python callers detect if we have cpython or direct-ffi bindings and fallback to the Python implementation if none is present. This intermediate state allows to compare the three possibilities. Differential Revision: https://phab.mercurial-scm.org/D5442

rust-cpython: binding for LazyAncestors The `mercurial.rustext.ancestor` module will not in the foreseeable future be a drop-in replacement for the pure `mercurial.ancestor`, because the Rust variants take the index at instantiation whereas the Python ones take a parents function. From the Python side, using the index from `ancestor` would leak internal details out of `mercurial.revlog`, and that's unwanted. Therefore, given that classes defined in `rust-cpython` have the same names in both language, we keep the Rust naming convention (CamelCase). Eventually, though, the ancestor module can be placed under control of `mercurial.policy`, but it will still be up to `revlog` to be aware of that and play the role of a factory for instantiation. Differential Revision: https://phab.mercurial-scm.org/D5441

discovery: move common heads computation inside partialdiscovery object This remove one of the private attribute access. In additions, head tracking and computation is a typical operation we can speed up using Rust.

discovery: introduce a partialdiscovery object This object will ultimately gather the data about common, undecided and missing revs in a single place and deal with most graph related computations. The goal is both to clarify the algorithm and to help provides a simple and clear API that can be reimplemented in Rust. For now, we only moved the `common` set in the object. In this commit, some direct access to the "private" `disco._common` attribute persist. They have not been removed yet because we won't need to expose a full API identical to `incrementalmissingancestors` and it seems simpler to access the attribute directly until the replacement is in place.

discovery: move handling of sampling special case inside sampling function The handling of cases where the number of revisions to sample is smaller than the sample size can be moved with the sample function themselves. This simplifies main logic, preparing a coming refactoring.

discovery: minor fix to some conditionals Since `size` is the upper limit of the sample, we should include it in the check. Otherwize the `more` variable will be zero and the sampling will be useless

delta: reuse _findsnapshot call from previous stage Two different stage of the sparse-revlog logic needs the _findsnapshot data. To avoid recomputing it twice, make it possible to reuse the first computation in the second step. example affected manifest write before: 0.067141s after: 0.064252s (-5%) (total gain since start of series: 95%)

delta: have a native implementation of _findsnapshot The function might traverse a lot of revision, a native implementation get significantly faster. example affected manifest write before: 0.114989 after: 0.067141 (-42%)

transaction: do not overwrite atomic-temp files on error Even though the original files can be restored from the backup, it should be better to not write back a temporary file if we know it can be corrupted.

help: fix typo

rust: use 'impl Trait' in method argument of AncestorsIterator I just didn't know it's stabilized. Let's switch to new convenient syntax. Differential Revision: https://phab.mercurial-scm.org/D5451

rust: use .rev() for reverse range Differential Revision: https://phab.mercurial-scm.org/D5450

histedit: add warning message on editing tagged commits (issue4017) Differential Revision: https://phab.mercurial-scm.org/D5489

strip: extract bookmark movement into a separate function We will need it for the soft-strip case.

strip: compute bookmark target only if we have bookmark to move This is a small change that seems to make sense.

strip: extract code to create strip backup We will reuse this for soft stripping.

strip: clarify comment around bundle create We may create two bundles, one for the changeset actually stripped, and one for the changeset affected by the strip we want to keep. (Changesets with a higher rev number than the striped ones). For soft stripping, we still need to the first one (used for "restoring" the changesets), but not the second one. We start with clarifying which bits of the code is used for what.

strip: move attributes shortcut assigned earlier This series will add support for "soft" stripping using the archived phase in addition to the usual heavy weight revlog stripping. First, we need to refactor the strip function to extract logic common to the "soft-strip" case. We are about to extract bits of the strip function into separate functions. Those blocks of code contain variables for frequently accessed attributes. We move those variables outside of the soon to be moved blocks.

log: fix line wrap on diffstat with -G/--graph (issue5800) Differential Revision: https://phab.mercurial-scm.org/D5485

registrar: minor cleanup to the merge function Pointed out by Yuya.

vfs: raise NotImplementedError in abstractvfs._auditvfs This make sure that auditing is implemented.

vfs: handle _auditpath in proxyvfs Just forward the call to the underlying vfs.

vfs: fix proxyvfs inheritance The proxyvfs class is designed to overwrite some of the vfs logic. Yet, it did not use normal class inheritance. This is becoming an issue as `abstractvfs` method could take precedence over their `proxyvfs` version. We fix the inheritance chain to be as expected.

vfs: also audit rename Renaming through the vfs is not used in many places, and none of them seems to be a security risk. However, it is still worthwhile to run the auditing on rename file to perform developer-warning level checks.

vfs: add a `_auditpath` to abstract vfs We are about to make `rename` audit path. Since rename lives in the `abstractvfs` layer, we need it to be aware of auditing to some extent. The default implementation is no-op because multiple existing vfs are not using auditing at all right now (eg: fncachevfs).

vfs: makes all audited path relative Only auditing relative path helps the vfs warning logic.

debugupgraderepo: add a --no-backup mode The process has been around for a while and is pretty safe now. Having an automated way to clean up the old data is useful when running many different conversion to compare delta algorithm.

upgrade: add '-' in optimization name The older name `redeltaall` was hard to type and read. The newer form should be more user-friendly. We keep backward compatibility with the old form (at least for a while). Having to use different form depending on the version is very impractical and error prone.

revlog: add test case for _findsnapshots Testing the method directly is more robust.

revlog: use the native implementation of issnapshot In some sparserevlog case where a lot of the history has to be searched for a snapshot, the cost of issnashot cost becomes significant. The computation done by the method is fairly low level, a native implementation provide a very significant speedup. example affected manifest write before: 0.490375s after: 0.114989s (-76%)

revlog: add a native implementation of issnapshot This will be used in the next changesets

revlog: more efficient implementation for issnapshot We avoid multiple method calls and tuple creation, this provides a significant speedup in some case: example affected manifest write before: 0.815520s after: 0.487767s (-40%)

revlog: clarify the non sparse-revlog case in `issnapshot` If we are not using sparse-revlog, there is only one type of snapshot, the full one.

exthelper: switch to using the registrar merging method

registrar: add a method to merge registrar instances This provides sanity checking beyond simply merging the underlying dictionaries.

phabricator: ensure that the return of urlopener.open() is closed No problem observed, just an oversight noticed while reading documentation.

revlog: fix pure python slicing test when chain contains nullrev The revlog mock used in the test was not behaving the same as its C counterpart.

merge with stable

remotefilelog: fix bug in maybesparsematch returning alwaysmatcher The description of the method says that it should return None if sparse is not used in this repository; since sparse.matcher() returns alwaysmatcher if sparse is not enabled, I'm using that as the signal to return None here to preserve the previous behavior. Differential Revision: https://phab.mercurial-scm.org/D5487

procutil: correct spelling of uninterruptable -> uninterruptible Differential Revision: https://phab.mercurial-scm.org/D5488

help: show advanced, experimental and deprecated extensions with --verbose I noticed that phabricator and absorb weren't showing up, no matter what. There's a related problem where commands for these extensions *do* show up without --verbose. I'm not sure what the point of hiding the extensions, but not the commands is.

help: remove a duplicate category name entry for TOPIC_CATEGORY_NONE

phabricator: assign commands to help categories `phabsend` and `phabupdate` seem pretty close to `email`. `phabread` is a little less close, but can be trivially used to actually import patches.

largefiles: eliminate an unnecessary import of configitems Thanks to Yuya for pointing this out.

exthelper: add some examples for using registrar aliases Maybe it's my general lack of python knowledge, but how to use these would be way too obscure for me otherwise.

exthelper: reintroduce the ability to register filesets Same mechanism as revsets earlier in this series. The LFS extension is updated to provide test coverage.

exthelper: reintroduce the ability to register templates Same mechanism as revsets earlier in this series. The LFS extension is updated to provide test coverage. This also seems to make the test failure around issue6033 mentioned in 0a7f582f6f1f much less regular on Windows.

extensions: deprecate extsetup without a `ui` argument (API) 9.5 years should be enough time, but there were some tests for the old style still (which are now updated). Exthelper doesn't fallback to the old API, so this is for consistency. .. api:: The extension hook ``extsetup`` without a `ui` argument has been deprecated, and will be removed in the next version. Add a `ui` argument to avoid the deprecation warning.

largefiles: port revset registration to exthelper This tests the merge code that wasn't tested in the previous patch.

exthelper: reintroduce the ability to register revsets I think this is what Yuya and Boris agreed on.[1] This happens *after* the extsetup phase now (and after the _aftercallback notifications). But this is trivial, mergeable between exthelper instances, and doesn't need to have the extension name supplied when registering. The test needed updating so that extsetup() takes a `ui` argument, as exthelper isn't trying to be backward compatible with 1.3.1. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-December/125888.html

largefiles: drop the uisetup module This is small enough to live in the __init__ module.

branches: add -r option to show branch name(s) of a given rev (issue5948) Differential Revision: https://phab.mercurial-scm.org/D5486

progress: avoid ui.configbool() lookup when progress bar is active Profiling revealed that the ui.configbool('progress', 'debug') during progress bar updates was consuming a significant amount of overhead. This commit adds an attribute on progress bar instances that caches this config option. The impact on `hg perfprogress` with default options is significant: before: ! wall 4.641942 comb 4.580000 user 4.210000 sys 0.370000 (best of 3) after: ! wall 1.948626 comb 1.950000 user 1.950000 sys 0.000000 (best of 5) After this change, profiling reveals that progress.progbar.progress() is now consuming ~73% of time. This change does not improve the execution time if the progress bar is disabled. We may want a more comprehensive solution for that case, as the progress bar won't be enabled in a number of scenarios (e.g. servers and processes not attached to an interactive TTY). I also think that overhead of ~2.0s for 1M updates is a bit high. I suspect further refactoring of the progress bar can significantly reduce overhead. I don't have plans to do this, however. Differential Revision: https://phab.mercurial-scm.org/D5408

largefiles: port wrapped functions to exthelper Things get interesting in the commit. I hadn't seen issue6033 on Windows, and yet it is now reproducible 100% of the time on Windows 10 with this commit. I didn't test Linux. (For comparison, after seeing this issue, I tested on the parent with --loop, and it failed 5 times out of over 1300 tests.) The strange thing is that largefiles has nothing to do with that test (it's not even mentioned there). It isn't autoloading run amuck- it occurs even if largefiles is explicitly disabled, and also if the entry in afterhgrcload() is commented out. It's also not the import of lfutil- I disabled that by copying the function into lfs and removing the import, and the problem still occurs. Experimenting further, it seems that the problem is isolated to 3 entries: exchange.pushoperation, hg.clone, and cmdutil.revert. If those decorators are all commented out, the test passes when run in a loop for awhile. (Obviously, some largefiles tests will fail.) But if any one is commented back in, the test fails immediately. I left one method related to wrapping the wire protocol, because it seemed more natural with the TODO. Also, exthelper doesn't support wrapping functions from another extension, only commands in another extension. I didn't try to figure out why rebase is both command wrapped and function wrapped.

largefiles: port commands to exthelper One subtle change here is that the purge, rebase and transplant extensions are wrapped in extsetup() instead of uisetup().

exthelper: support the option argument when registering a command Largefiles uses this 5th argument with push and pull, so this will be tested in the next commit. I assume the reason for unrolling and reforming the tuple in each finalxxxsetup() is to validate that something proper was passed in when registering. But it's better to explode when decorating than during the delayed actual registration.

largefiles: port configitems to exthelper It looks like dynamicdefault was referenced in a non-standard way.

remotefilelog: use repo.local() instead of isinstance()

exthelper: add a cautionary note about adding attributes to containers

exthelper: drop a compatibility shim with '^command' syntax This syntax was dropped in fa88170c10bb.

wireproto: in batch queries, support queries with immediate responses listkeys and pushkey return without querying the remote when the remote doesn't support such queries. Before this change, the batching code didn't handle this convention, resulting in this kind of error: $ hg pull ssh://user@dummy/repo1 -r tip -B a pulling from ssh://user@dummy/repo1 remote: ** Unknown exception encountered with possibly-broken third-party extension disable-lookup remote: ** which supports versions unknown of Mercurial. remote: ** Please disable disable-lookup and try your action again. remote: ** If that fixes the bug please report it to the extension author. remote: ** Python 2.7.15+ (default, Oct 2 2018, 22:12:08) [GCC 8.2.0] remote: ** Mercurial Distributed SCM (version 4.8.1+586-ef54bd33b476+20181224) remote: ** Extensions loaded: disable-lookup remote: Traceback (most recent call last): remote: File "/tmp/hgtests.i66Npc/install/bin/hg", line 43, in <module> remote: dispatch.run() remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 99, in run remote: status = dispatch(req) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 225, in dispatch remote: ret = _runcatch(req) or 0 remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 376, in _runcatch remote: return _callcatch(ui, _runcatchfunc) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 384, in _callcatch remote: return scmutil.callcatch(ui, func) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/scmutil.py", line 166, in callcatch remote: return func() remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 367, in _runcatchfunc remote: return _dispatch(req) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 1021, in _dispatch remote: cmdpats, cmdoptions) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 756, in runcommand remote: ret = _runcommand(ui, options, cmd, d) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 1030, in _runcommand remote: return cmdfunc() remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/dispatch.py", line 1018, in <lambda> remote: d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/util.py", line 1670, in check remote: return func(*args, **kwargs) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/commands.py", line 5257, in serve remote: s.serve_forever() remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/wireprotoserver.py", line 797, in serve_forever remote: self.serveuntil(threading.Event()) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/wireprotoserver.py", line 804, in serveuntil remote: _runsshserver(self._ui, self._repo, self._fin, self._fout, ev) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/wireprotoserver.py", line 656, in _runsshserver remote: rsp = wireprotov1server.dispatch(repo, proto, request) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/wireprotov1server.py", line 74, in dispatch remote: return func(repo, proto, *args) remote: File "/tmp/hgtests.i66Npc/install/lib/python/mercurial/wireprotov1server.py", line 195, in batch remote: data[k] = vals[k] remote: KeyError: 'namespace' abort: unexpected response: empty string [255] Differential Revision: https://phab.mercurial-scm.org/D5482

rust: core implementation for lazyancestors Once exposed through appropriate bindings, this should be able to replace ancestor.lazyancestors entirely. Differential Revision: https://phab.mercurial-scm.org/D5440

rust-cpython: binding for AncestorsIterator It's now reachable from Python as rustext.ancestor.AncestorsIterator Tests are provided in the previously introduced Python testcase: this is much more convenient that writing lengthy Rust code to call into Python. Differential Revision: https://phab.mercurial-scm.org/D5439

rust-cpython: implement Graph using C parents function We introduce the `Index` struct that wraps the C index. It is not intrinsically protected by the GIL (see the lengthy discussion in its docstring). Improving on this seems prematurate at this point. A pointer to the parents function is stored on the parsers C extension module as a capsule object. This is the recommended way to export a C API for consumption from other extensions. See also: https://docs.python.org/2.7/c-api/capsule.html In our case, we use it in cindex.rs, retrieving function pointer from the capsule and storing it within the CIndex struct, alongside with a pointer to the index. From there, the implementation is very close to the one from hg-direct-ffi. The naming convention for the capsule is inspired from the one in datetime: >>> import datetime >>> datetime.datetime_CAPI <capsule object "datetime.datetime_CAPI" at 0x7fb51201ecf0> although in datetime's case, the capsule points to a struct holding several type objects and methods. Differential Revision: https://phab.mercurial-scm.org/D5438

pull: fix inconsistent view of bookmarks during pull (issue4700) I had a share where a pull apparently pulled a bookmark but not the revision pointed to by the bookmark, which I suspect is due to this (and if not, we might as well remove known issues in this area). I do this by combining doing all the queries that could read the bookmarks in one round trip. I had to change the handling of the case where the server doesn't support the lookup query, because if it fails, it would otherwise make fremotebookmark.result() block forever. This is due to wireprotov1peer.peerexecutor.sendcommands's behavior (it fills a single future if any query fails synchronously and leaves all other futures unchanged), but I don't know if the fix is to cancel all other futures, or to keep going with the other queries. Differential Revision: https://phab.mercurial-scm.org/D5449

merge: modify the logical statement Differential Revision: https://phab.mercurial-scm.org/D5476

exthelper: correct a documentation typo

lfs: convert to using exthelper to wrap functions I'm not 100% sure that upgraderequirements() can be double annotated safely, but it seems OK based on printing the address of the function being wrapped. One thing I've noticed is that @eh.reposetup doesn't do the usual check to ensure that it's a local repo. Should that be baked into @eh.reposetup() somehow, possibly with a non-default option to skip the check? It seems like a gaping hole if every function that gets registered needs to add this check.

tests: convert a test extension to use exthelper This provides test coverage to uipopulate().

exthelper: drop fileset/revset/template support for now Yuya raised concerns about duplicating registrar functionality. There are a couple of ideas to work around this, which would allow bringing them back, and then backporting to evolve. For now, I just want to get the subsequent changes landed before the bulk b'' rewrite makes rebasing too hard.

exthelper: simplify configitem registration

extensions: import the exthelper class from evolve 980565468003 (API) This should help make extensions that wrap a lot of stuff more comprehendible. It was copied unmodified, except: - fix up the imports - rename final_xxxsetup() -> finalxxxsetup() to appease checkcode - avoid a [] default arg to wrapcommand() .. api:: Add `exthelper` class to simplify extension writing by allowing functions, commands, and configitems to be registered via annotations. The previous APIs are still available for use.

narrow: detect if narrowspec was changed in a different share With this commit, `hg share` should be usable with narrow repos. Design explained on https://www.mercurial-scm.org/wiki/NarrowSharePlan I was running into cache invalidation problems when updating the narrowspec. After spending a day trying to figure out a good solution, I resorted to just assigning repo.narrowpats and repo._narrowmatch after invalidating them. Differential Revision: https://phab.mercurial-scm.org/D5278

tests: add test for narrow+share For how narrow+share is supposed to work, see https://www.mercurial-scm.org/wiki/NarrowSharePlan. Differential Revision: https://phab.mercurial-scm.org/D5276

narrow: keep narrowspec backup in store As suggested by Yuya in review of D4099. Differential Revision: https://phab.mercurial-scm.org/D5470

tests: update narrowspec when narrowspec, not dirstate, is accessed test-narrow-expanddirstate.t mimics a Google-internal extension that updates the narrowspec whenever the dirstate is accessed. Since 1d09ba0d2ed3 (narrow: move remaining narrow-limited dirstate walks to core, 2018-10-01) and a few commits before it, we no longer restrict repo.dirstate.walk() to the narrowspec. It is instead done at a higher level (e.g. context.status()). We were running into problems with the Google-internal extension when importing those commits. The issue was that the narrowspec was read before the first dirstate access. I believe the right fix is to instead update the narrowspec when trying to read it (not when reading the dirstate), so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D5275

merge: extract helper for creating empty "actions" dict Replicating the set of actions in multiple places is bad. Differential Revision: https://phab.mercurial-scm.org/D5472

manifest: accept narrowmatch into constructor instead of getting from repo The manifest should ideally not know at all about the repo, so this is just a little step towards cleaning that up. Differential Revision: https://phab.mercurial-scm.org/D5469

py3: byteify one more sys.argv in gendoc.py

test: introduce a new flag to display env variable line per line It's easier to conditionalize some of the environment variables per Mercurial version once there is only one value per line. Differential Revision: https://phab.mercurial-scm.org/D5453

revlog: add an explicit test for `issnapshot` We test the method on a real revlog containing "real" data.

revlog: add some direct testing of the slicing logic This test check slicing backed by an actual revlog. It will test the C version of slicing (if the test are run with the C extensions).

revlog: limit base to rev size ratio to 500 instead of 50 While a value of 50 provided large speedup in some case (400s → 7s) it also creates a slow down for a whole class of revision we are seeing in a private repository (0.1s → 3s). A value of 500 makes them disappear improving the total runtime (the slower revision still improve significantly (400s → 21s)). We need to run a wider array of tests on various repositories to see the effect on speed and size of different values for this acceptable constant. However, in the meantime, it seems safer to move back to a less impactful value.

contrib: remove unused version of os Caught by test-check-pyflakes.t Differential Revision: https://phab.mercurial-scm.org/D5471

py3: convert `'{}'.format(foo)` to `'%s' % foo` in the bookflow extension Byte strings don't have the former. Converting these to byte strings is waiting on the mass rewrite.

py3: byteify sys.argv in gendoc.py

repository: update interface signature of narrowmatch() This should have been part of 4fd0fac48922 (localrepo: allow narrowmatch() to accept matcher to intersect with, 2018-09-28) and 41fcdfe3bfeb (narrow: allow repo.narrowmatch(match) to include exact matches from "match", 2018-10-01). Differential Revision: https://phab.mercurial-scm.org/D5466

narrow: when narrowing, write new narrowspec before removing revlogs If revlogs were removed and then the process crashed before the narrowspec was written, the repo would be corrupt. Differential Revision: https://phab.mercurial-scm.org/D5467

narrow: replace "ui.warn(); return 1" by "raise error.Abort()" This is the usual way of doing it and I don't see a reason to do it differently here. Differential Revision: https://phab.mercurial-scm.org/D5468

test-bookmarks-pushpull: add failing test of issue4700 Differential Revision: https://phab.mercurial-scm.org/D5447

tests: update printenv.py argument parsing We are about to introduce a new flag for printing the HG environment variables one per line and it's easier to do when using the argparse module for argument parsing. Differential Revision: https://phab.mercurial-scm.org/D5452

fuzz: new fuzzer for parsers.fm1readmarkers Differential Revision: https://phab.mercurial-scm.org/D5465

server: always close http socket if responding with an error (issue6033) It's possible for hgweb to respond _very_ early with an error if we're catching certain types of errors. When we do, we need to tell the client the socket is toast when there's a POST involved because otherwise there can be lingering POST data on the socket that will confuse any future requests on the socket. This manifested as a flaky failure on Linux in an lfs extension test and a reliable failure on FreeBSD. With this patch applied, test-lfs-serve-access.t now passes for me on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5498

match: fix assertion for fileset with no context (issue6046) A falsy changectx should be allowed.

templatekw: fix documentation typos

update: do not pass in user revspec as default destination (issue6044) When the revsingle() was introduced at 61c0df2b089a, it couldn't handle revspec=0 (not '0') properly. That's probably why the default was set to rev. This is technically BC since "hg update ''" was identical to "hg update '.'" whereas "hg update -r ''" is "hg update", but I believe that's a bug given no test fails with this change.

revlog: cache delta base value under -1 Such base are invalid so we better report them early.

revlog: catch revlog corruption in index_baserev A revision cannot use a base above itself, it can only happens one corrupted repository. Ignoring such corrupted could lead to infinite loop.

phabricator: properly encode boolean types in the request body I tripped over this playing with `hg debugcallconduit` to query for valid reviewers. If the JSON on stdin is written as 'True' or 'False', python complains it isn't valid JSON. If it's written as 'true' or 'false', it made it to the server, but got kicked back with this: abort: Conduit Error (ERR-CONDUIT-CORE): Error while reading "isBot": Expected boolean (true or false), got something else. The test isn't really relevant here (the code can be reverted, and it will pass), but this gives us coverage for the debug command.

parsers: better bounds checking in fm1readmarkers Our Python already calls this with reasonable values consistently, but my upcoming fuzzer is extremely quick to discover the lack of sanity checking here. Differential Revision: https://phab.mercurial-scm.org/D5464

fuzz: new fuzzer for dirstate parser Differential Revision: https://phab.mercurial-scm.org/D5463

fuzz: new fuzzer for revlog's parse_index2 method Differential Revision: https://phab.mercurial-scm.org/D5462

fuzz: extract Python initialization to utility package Avoids code duplication between fuzzers of parsers.so. Differential Revision: https://phab.mercurial-scm.org/D5461

fuzz: remove probably-wrong -fsanitize from fuzzutil.o rule Differential Revision: https://phab.mercurial-scm.org/D5460

parsers: remove long-dead parse_manifest method We haven't used this in years, I think it's fine to ditch it now. We had previously kept it around to ease bisecting with built extensions, but these days we've got a better versioning scheme anyway. Noticed this method kicking around while looking in parsers.so for likely fuzzing targets. Differential Revision: https://phab.mercurial-scm.org/D5459

help: hide default value for default-off flags If we no longer show the "[no-]" for default-off flags, it also seems unnecessary to show the "default: off" for them, since that's quite clearly the default. It's extra confusing for action flags like `hg bookmarks --delete`. Differential Revision: https://phab.mercurial-scm.org/D5455

help: show "[no-]" only for default-on Flags As Anton (av6) pointed out, the "[no-]" is confusing for action flags like `hg bookmark --delete`. We could come up with a way of indicating which flags are action flags (e.g. use None for the default value instead of False). However, it's probably also unlikely that users will want to negate even non-action flags like --hidden. One of the more common flags where the "[no-]" prefix would be useful is `hg evolve --update`. The reason it's helpful there is that it defaults to on. So I think we can simply include "[no-]" only for flags that are on by default (and thus require the user to add the "[no-]" for the option to have any effect). Note that there are use cases for negating flags that already off by default. For example, you may have an alias for `hg log -G --hidden -T foo` and now want to pass "--no-hidden" to that alias. However, I think that users who want that are likely to be advanced enough that they've already learnt about the "no-" prefix by seeing it somewhere else. Differential Revision: https://phab.mercurial-scm.org/D5454

shelve: drop unnecessary backup of narrowspec I mechanically added the backup code everywhere in ad24b581e4d9 (narrow: call narrowspec.{save,restore,clear}backup directly, 2018-08-03), but I can't think of a reason it would be needed in the shelve code, so let's drop it. Differential Revision: https://phab.mercurial-scm.org/D5457

shelve: pass transaction around to clarify where it's used Differential Revision: https://phab.mercurial-scm.org/D5456

py3: byteify docchecker The exception is printed as str because I'm too lazy to convert it and the pieces.

py3: replace hard coded `python` with $PYTHON in check-gendoc

py3: byteify gendoc.py This is mostly b'' prefixing, with some cargoculting of help.py to get around `textwrap.dedent()` and __doc__ string requirements.

py3: use bytes stdout in hghave.py This fixes a failure in test-run-tests.t around notarealhghavefeature. It seems crazy to me that all of this needs to be adjusted in all of these tests, but the line as run-tests.py sees it in _processoutput() before doing anything is already mangled with a trailing '\r'. Switching to normalizenewlines=True for TTest works, but I'm sure that breaks other stuff.

perfrevlogwrite: fix a typo in the option name The submitted patches was apparently damaged, and a 's' was missing.

fuzz: improve makefile per feedback from some folks at work This is a *ton* better. Once this lands, I can remove a nasty kludge from the oss-fuzz build.sh. Bonus: this fixes the coverage build. Differential Revision: https://phab.mercurial-scm.org/D5458

py3: use bytes stdout in test-check-help.t Setting stdout to binary seemed to have no effect on Windows, as it was appending a literal '\r' to each topic keyword. This also stops prepending 'b' to the topic on all platforms as well.

commandserver: preload repository in master server and reuse its file cache This greatly speeds up repository operation with lots of obsolete markers: $ ls -lh .hg/store/obsstore -rw-r--r-- 1 yuya yuya 21M Dec 2 17:55 .hg/store/obsstore $ time hg log -G -l10 --pager no (hg) 1.79s user 0.13s system 99% cpu 1.919 total (chg uncached) 0.00s user 0.01s system 0% cpu 1.328 total (chg cached) 0.00s user 0.00s system 3% cpu 0.180 total As you can see, the implementation of the preloader function is highly experimental. It works, but I'm yet to be sure how things can be organized. So I don't want to formalize the API at this point.

commandserver: add IPC channel to teach repository path on command finished The idea is to load recently-used repositories first in the master process, and fork(). The forked worker can reuse a warm repository if it's preloaded. There are a couple of ways of in-memory repository caching. They have pros and cons: a. "preload by master" pros: can use a single cache dict, maximizing cache hit rate cons: need to reload a repo in master process (because worker process dies per command) b. "prefork" pros: can cache a repo without reloading (as worker processes persist) cons: lower cache hit rate since each worker has to maintain its own cache c. "shared memory" (or separate key-value store server) pros: no need to reload a repo in master process, ideally cons: need to serialize objects to sharable form Since my primary goal is to get rid of the cost of loading obsstore without massive rewrites, (c) doesn't work. (b) isn't ideal since it would require much more SDRAMs than (a). So I take (a). The idea credits to Jun Wu.

upgrade: correct implementation of improvement.__ne__ The "not" operator binds more closely than "==": >>> not False == False False

extensions: use ui.log() interface to provide detailed loading information The output format changes and the messages will be sent to stderr instead of stdout, but I don't think that matters.

mq: implement log() on dummyui Otherwise ui.log() in extensions.py would explode.

ui: install logger that sends debug.extensions messages to stderr This will replace the custom log function introduced at d58958676b3c "extensions: add detailed loading information."

blackbox: resurrect recursion guard If I added ui.log() to hg.repository() function, test-merge-subrepos.t exploded. The problem is that the blackbox may create new repository instance while logging is active, and the created repository owns its new ui derived from the baseui, not from the ui which is processing the active logging. I tried to work around the issue in ui.log(), but that turned out to be not easy. We shouldn't globally lock the ui.log() since there may be more than one active repo/ui instances in threaded environment. We could store the logging state in thread-local storage, but that seems unnecessarily complex. So this patch reintroduces the _inlog flag to per-repository logger instances.

tests: filter out uninteresting log events This helps adding more log()s without updating the tests.

context: error out if basefilectx.cmp() is called without self._filenode The base implementation can't handle such cases because the filelog has no knowledge about the working directory. Loading self._filenode should have no extra cost since self.size() would load it anyway.

context: collapse complex condition to see if filelog have to be compared It's hard to read. I'd rather make the return statement duplicated.

merge with stable

worker: do not swallow exception occurred in main process Before, SystemExit(255) would be most likely raised since the worker processes were terminated by the main process and the status would be set to 255 in response. We should instead re-raise the exception occurred first. It's pretty hard to debug problems like the issue 6035 with no traceback.

run-tests: fix permission to clean up unreadable directories I found many hgtests.* directories left in $TMPDIR, which couldn't be deleted because test-ssh-repoerror.t created some directories with a-rx mode.

context: reimplement memfilectx.cmp() If I added a sanity check to basefilectx, test-context.py exploded. This patch copies the naive implementation from overlayworkingfilectx.

rust-cpython: build and support for Python3 Defined Cargo features for Python3, making them overall simpler to use, hooked them in build and made mercurial.rustext importable. This is tested with Python 3.6.7. Differential Revision: https://phab.mercurial-scm.org/D5446

py3: convert popen() command arguments in hgclient to str on Windows This fixes test-commandserver.t and test-keyword.t, which was previously complaining TypeError("a bytes-like object is required, not 'str'")

py3: byteify tests/pullext.py # skip-blame for b'' prepending

py3: byteify the fakeversion extension in test-http-bad-server.t # skip-blame for b'' prefixing

py3: spawn all python instances with legacy stdio enabled on Windows This fixes 6 more tests. See ef7119cd4965.

setup: avoid attempting to invoke the system-wide hg.exe on Windows On Windows, the executable in the current directory gets priority over anything in $PATH (both for cmd.exe and MSYS). That means, the former code was launching the local hg.exe instead of the system-wide one, if it was previously built. If that failed, it then fell back to the local hg code, but run through python.exe. I'm not sure what it is about ef7119cd4965, but that started throwing up a messagebox that python37.dll couldn't be loaded. (And indeed, python37 is not in $PATH by default.) Invoking the local hg via the current python avoids that.

delta: ignore base whose chains already don't match expectations If we know the existing chain does not match our criteria, there is no point to build a delta to append. This is especially useful when dealing with a full text much smaller than its parent. In that case, the parent chain is probably already too large. example affected manifest write before: 1.421005s after: 0.815520s (-42%)

delta: exclude base candidate much smaller than the target If a revision's full text is that much bigger than a base candidate full text, we no longer consider that candidate. This solves a pathological case we encountered on a very specify repository. It contains a long series of changesets with a very small manifest (one file) co-existing with others changesets using a very large manifest. Without this filtering, we ended up considering a large number of tiny full snapshots as a potential base. It resulted in very large delta (the size of the full text) and mercurial spending 99% of its time compressing these deltas. The timing of a commit moved from about 400s to about 10s (still slow, but not ridiculously slow).

perfrevflogwrite: clear revlog cache between each write We want to measure write time from a cold cache (similar to commit). So we need to clear the cache to prevent computation from rev N-1 to interfere with rev N.

py3: enable legacy stdio mode in exewrapper This drops the test failure count from 166 to 117. The failures were typically in the form of `hg serve -d` spawning but crashing immediately, leaving clients with "bad http status" lines, connection refusals, and so forth. The underlying message on the server side was either "OSError: [WinError 6] The handle is invalid" or "OSError: [WinError 1] Incorrect function". Additionally, no output was rendered if the pager was activated. Thanks to Yuya for diagnosing the problem. The failure count drops to 107 when PYTHONLEGACYWINDOWSSTDIO=1 is defined in the environment. These failures seem to revolve around the dummyssh server process, and dumbhttp.py. So I'll probably add that to the test runner. One small regression here (only in py3) is that if hg.exe is already built, a messagebox appears when building it again saying that python37.dll can't be loaded. Python3 isn't in PATH by default, and setup.py tries running bare `hg` first. But MSYS prepends '.' to PATH, so it runs the local hg, but can't find the library. It falls back to the python used to invoke setup.py, so ultimately it works. I'm not sure if it's better to strip '.' from PATH or just skip right to `sys.executable hg` on Windows. Also, something seems to be wrong with run-tests._usecorrectpython(). I accidentially left off the 'PYTHON="py -3"' when building (thus making py2 stuff), and yet managed to invoke run-tests.py with "py -3". (And that only had 67 failures.)

run-tests: alias hg to hg.exe on Windows To enable legacy stdio mode on Windows, hg.exe needs to be updated. But before that, we actually have to use it when running the tests. I *think* what was happening before was when MSYS invoked `hg`, it looked at the shbang line and ran python.exe found there. The test harness must be updating $PATH to include the python used to launch it, and therefore it ran py3. As a side note, this also fixed `py -3 run-tests.py` (without --local), which complained about the space in the shbang line before this. This should also help in WSL, because the explicit '.exe' is needed to invoke a Windows app instead of invoking the Linux app. I have no idea why this change capitalizes Lib in the tests, as it was previously lowercase for both py2 and py3.

py3: conditionalize the python version in test-install.t

tests: followup on network related errors on Debian 9 First, update test-clonebundles.t which was failing since the introduction of the `$EADDRNOTAVAIL$` common pattern. Also update two tests from 8695fbe17f7c with a more compact condition line. Differential Revision: https://phab.mercurial-scm.org/D5445

py3: unimplement RevlogError.__str__() On Python 2, str(exc) would crash if __str__() returned a unicode containing non-ASCII characters.

py3: use forcebytestr() to stringify hook exception This fixes test-narrow-exchange.t.

color: fix a documentation typo

py3: quote $PYTHON in test-patchbomb.t for Windows I couldn't get the quoting right in the environment variable, so now it's a function.

rust-cpython: testing the bindings from Python This is easier and more convincing than doing the same tests from a Rust tests module. Differential Revision: https://phab.mercurial-scm.org/D5437

rust-cpython: build via HGWITHRUSTEXT=cpython The existing behaviour, building the direct ffi bindings if HGIWTHRUSTEXT is just set is unchanged, but if HGWITHRUSTEXT is cpython, then the cpython bindings (aka mercurial/rustext.so) are built. Differential Revision: https://phab.mercurial-scm.org/D5436

rust: better treatment of cargo/rustc errors Differential Revision: https://phab.mercurial-scm.org/D5435

rust-cpython: start cpython crate bindings This changeset introduces the hg-cpython crate, that compiles as a shared library holding a whole Python package (mercurial.rustext), with only the empty 'ancestor' submodule for now. Such bindings will be easier and safer to develop and maintain that those of `hg-direct-ffi`. They don't involve C code, only unsafe Rust that's mostly isolated within the cpython crate. The long-term goal would be to import the provided modules, such as rustext.ancestor with mercurial.policy.importmod, same as we already do with cext modules. Differential Revision: https://phab.mercurial-scm.org/D5434

rust-cpython: exclude hgcli from workspace hgcli uses a specific rust-cpython commit by indygreg, of which a PR has been derived which is not merged nor released yet. But we can't use several versions of the sys-python2.7 crate in a single workspace: it makes for a build error. Since hgcli does not at the time being need anything from hg-core, whereas the upcoming hg-cpython will. So for now we're moving hgcli aside, hoping we could base all of them on the same version of rust-cpython again in the future. Differential Revision: https://phab.mercurial-scm.org/D5433

sparse-revlog: protect C code against delta chain including nullrev For unclear reasons, some repositories include nullrev (-1). Re-computing delta for such repo remove nullrev from all chain, so some older versions have been creating them. This currently raise an IndexError with the new C code doing chain slicing as it expect all item to be positive. Both python and C code for reading delta chain preserve nullrev, and the Python code for chain slicing handle the case fine. So we take the safe route and make the new C code works fine in that case.

sparse-revlog: handle nullrev in index_get_length The more generic index_get method handle nullrev fine, we apply the same logic here.

sparse-revlog: handle nullrev in index_get_start The more generic index_get method handle nullrev fine, we apply the same logic here.

revlog: introduce a constant for nullrev in `revlog.c` The value is important enough to be explicitly tracked.

rust: translation of missingancestors This is as direct as possible a translation of the ancestor.missingancestors Python class in pure Rust. The goal for this changeset is to make it easy to compare with the Python version. We also add to Python tests the cases that helped us develop and debug this implementation. Some possible optimizations are marked along the way as TODO comments Differential Revision: https://phab.mercurial-scm.org/D5416

contrib: provide a small script that draw performance plot We have been using this script to look into the result of various runs of the `hg perfrevlogwrite` command. It seems useful enough to be shared more widely.

delta: filter nullrev out first When picking a potential candidate, we filter them on various criteria. The "different from nullrev" criteria is very fast to compute and we should process it first.

perf: report more of the higher range in perfrevlogwrite Since the delta chain length is limited to 1000 revisions, we get a new snapshot about every 1000 revisions. If we assume that the snapshot will be most of the slowest revision, the current display (99% and max) are not very precise in their area. We now include more information about this space in the default report.

help: present boolean arguments as "--[no-]foo" This should make it much more discoverable (we document it in `hg help flags`, but most users don't think to look there). Note that flags that default to None (and not False) will not get this new presentation. We can change the defaults to False later for flags where it makes sense (probably almost all boolean flags). Differential Revision: https://phab.mercurial-scm.org/D5432