rawdata: implement the method for `remotefilelog` too This is needed for all storage implementations.

rawdata: implement `rawdata` for `simplestore` too This is needed for all implementation.

rawdata: forward `rawdata` call on `manifestlog` This needs to be sent to the underlying `revlog` too.

rawdata: implement `rawdata` for `sqlitestore` too This is a different store, it needs it declared.

rawdata: add the method to bundlerevlog The bundlerepo logic has its own `revision` method on its own `revlog` object. We need to "implement" `rawdata` there too.

rawdata: forward the method call on `filelog` object We have a new method, we need to expose it.

rawdata: introduce a `rawdata` method on revlog This method aims at replacing `revision(..., raw=True)` call. The purpose of data returned without and without raw are different enough that having two different method would make sense. This split is motivated by other work aiming at storing data on the side of the main revision of a revlog. Having a cleaner API makes it simpler to add this work. The series following this first changesets is organised as follow: 1) add `rawdata` method everywhere it is useful 2) update all caller 3) implement all `rawdata` method without using `revision` 4) deprecate the `rawdata` parameter of `revision`

revlog: split a `_revisiondata` method to file `revision` job We are about to introduce more public method to access revision data (eg: `rawdata`). revset subclass tend to recursively call `revision` which will create all kind of issue with the coming series. To avoid them we introduce an explicit difference between the internal call and the public all. This will be useful for later work anyway (so the subclass issue is just moving it earlier in the series). I am not sure if the subclass are actually doing something sensible. However, I am certain I don't want to be rabbit holed into figuring it out right now.

continue: added support for transplant This creates a seperate function `continuetransplant()` containing logic for resuming transplant from interrupted state. `continuetransplant()` is then registered as `continuefunc` for state detection API. Results are shown in tests. Differential Revision: https://phab.mercurial-scm.org/D6689

merge with stable

branchmap: explicitly warm+write all subsets of the branchmap caches 'full' claims it will warm all of the caches that are known about, but this was not the case - it did not actually warm the branchmap caches for subsets that we haven't requested, or for subsets that are still considered "valid". By explicitly writing them to disk, we can force the subsets for ex: "served" to be written ("immutable" and "base"), making it cheaper to calculate "served" the next time it needs to be updated. Differential Revision: https://phab.mercurial-scm.org/D6710

changectx: extract explicit computechangesetfilesremoved method from context Right now, the logic around changeset centric removed files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

changectx: extract explicit computechangesetfilesadded method from context Right now, the logic around changeset centric added files data are buried into the "changectx" code. We extract this code in a dedicated method (in the scmutil module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

copies: extract an explicit `computechangesetcopie` method from context Right now, the logic around changeset centric copies data are buried into the "changectx" code. We extract this code in a dedicated method (in the copies module) for clarity. This clarity will help to explicitly compute and caches these data in the future.

config: fix fm.data() handling of defaultvalue This is a follow-up patch to rHG51a2e3102db2. This moves `fm.data()` out of the if block in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6720

config: remove pycompat.bytestr() for defaultvalue This is a follow-up patch to 51a2e3102db2. This removes `pycompat.bytestr` to preserve `None` in `commands.config()`. Differential Revision: https://phab.mercurial-scm.org/D6712

unshelve: clear shelvedstate and _finishunshelve() on partial unshelve On a partial unshelve, `shelvedstate` was not cleared and `_finishunshelve()` was not called. Ideally, these should be called on this case. This patch makes `shelvedstate` to delete after a successful partial unshelve and calls `_finishunshelve()` in the same case. Differential Revision: https://phab.mercurial-scm.org/D6708

unshelve: delete shelvedstate after a successful unshelve --continue `unshelve --continue` was preventing the deletion of `shelvedstate` on a partial `unshelve`. Ideally, `shelvedstate` should be deleted after a successful `unshelve`. Now, the behavior of `unshelve --continue` will be as follows in interactive mode: 1] The user tried to `unshelve` changes interactively but, ran into conflicts. 2] They resolved the conflicts and triggered `unshelve --continue` but, unshelved changes partially. 3] Now, on trying to do `unshelve --continue` again will abort as the last `unshelve` was successful and we are deleting the `shelvedstate`. 4] If they want to unshelve the remaining shelved change, they need to trigger `unshelve` without `--continue`. Differential Revision: https://phab.mercurial-scm.org/D6694

unshelve: handle stripping changesets on interactive mode On interactive mode, changesets on `nodestoremove` should be stripped regardless of the shelve is partial or not. This patch modifies `unshelvecontinue()` to do that. Differential Revision: https://phab.mercurial-scm.org/D6686

byteify-strings: add --version argument This is indispensable for automated tools to detect changes in behavior.

byteify-strings: add space in special comments to silence flake8 error This is done soon enough that nobody has had the time to use this feature yet.

revset: drop argument when it's None getstack's definition is `getstack(repo, rev=None)`, so providing None explicitly is unnecessary. Moreover, when x is not None, it's definitely not a revision but a part of a parsed tree of revset arguments. Differential Revision: https://phab.mercurial-scm.org/D6707

stack: remove unnecessary reverse() predicate Stack already sorts revisions, so no need to do it twice. This change was a part of D2400, which didn't land for other reasons. See also D2399, where this change was suggested. Differential Revision: https://phab.mercurial-scm.org/D6706

automation: increase root volume size on Linux It is close to full in the AMI. I actually ran out of space running tests without increasing the volume size! Differential Revision: https://phab.mercurial-scm.org/D6716

automation: install Rust in Linux environment This will install Rust 1.31.1, 1.34.2, and whatever stable is at the time the install runs. We install 1.31.1 as our minimum supported Rust version (I think that's what we're currently targeting) and 1.34 because that's what Debian 10 is shipping. Differential Revision: https://phab.mercurial-scm.org/D6715

automation: update packages in requirements files We like keeping up to date. The content of the autogenerated files changed slightly because I used a newer version of `pip-compile` than what was used previously. Differential Revision: https://phab.mercurial-scm.org/D6714

automation: install latest Python versions This required bumping the pyenv commit so the new versions are available. Differential Revision: https://phab.mercurial-scm.org/D6713

upgrade: introduce the internal code for revlog cloning selection For now we still clone every single revlogs but all the selection mechanism is now in place in the lower layer. The next changesets will introduce the user interface part of the selection.

upgrade: introduce a _copyrevlog method This function copies a revlog from the old store to the new one, without re-adding all deltas manually. This will eventually save a lot of time when some revlog does not needs any conversions. Code actually using this will be introduced in later changesets.

upgrade: rename `_copyrevlogs` to `_clonerevlogs` The underlying revlog method is named `clone`, keeping the naming consistent seems clearer. This is motivated to clarify the difference with an (upcoming) function that simply copy revlog files as is.

upgrade: walk the source store file only once I don't expect this to have a significant performance impact, but it seems simpler and saner to do the operation only once and to keep the result around.

upgrade: always use full text if "full-add" mode is enable We should not be using a delta since the goal is to perform a full addition from scratch in all cases. Without this patch, `hg debugupgraderepo --optimize re-delta-fulladd --run` can crash.

byteify-strings: fix misalignment with multi-line parenthesis This improves the current fix to also take into account cases where the last line ended on the opening `(`, `[` or `{` and adds a regression test.

byteify-strings: add test for byteify-strings.py This tests the basic features expected from this script, some cases may not be covered yet. A future patch will demonstrate an issue with multi-line `(`, `[` and `{` alignment and propose a fix.

merge with stable

byteify-strings: add cli argument to handle `attr*()` when they are methods Certain code bases have useful utils that wrap the builtin functions, and are called like `util.setattr`.

byteify-strings: simplify default value for `--treat-as-kwargs`

byteify-strings: add --treat-as-kwargs argument to handle kwargs-like objects This argument will help extensions move to Python 3 as keyword arguments should not be byte-prefixed. Most of the time, code bases will call this object `kwargs`, but other conventions exist like `opts`, so it should make sense to allow for custom names. This is a best effort solution that does minimal static checking; cases like `options = [o for o in ('a', 'b', 'c') if kwargs.get(o)]` and other just as complicated will not be detected.

byteify-strings: add helpers to check for item access or method call These helpers will be used in a future patch, split for ease of review.

byteify-strings: add support for ignore comments Our simple token analysis is sometimes not clever enough, we need to be able to turn off our script for parts of the code. This change introduces three special comments: - `#no-py3-transform` to tell `byteify-strings` ignore the next line - `#py3-transform: off` to ignore everything until the end of the file - `#py3-transform: on` to stop ignoring The last two can be particularly useful within Python 2/3 compatibility files.

byteify-strings: handle triple quoted strings if they are not docstrings As with anything in this script, this is a best effort approach. Most of the time, when a triple quoted string is assigned to something, it's not a docstring.

byteify-strings: handle multi-line strings in _ensuresysstr The current implementation did not handle calls like `repo.ui.log("first line" "other line")` correctly.

fix: run fixer tools in the repo root as cwd so they can use the working copy This lets fixer tools do things like find configuration files, with the caveat that they'll only see the version of that file in the working copy, regardless of what revisions are being fixed. Differential Revision: https://phab.mercurial-scm.org/D6440

config: add defaultvalue template keyword This patch tries to fix one of the issues mentioned in issue6014. This adds a new `defaultvalue` template keyword to be used with `hg showconfig` to get the default value of the config item. Differential Revision: https://phab.mercurial-scm.org/D6704

merge with stable

module-policy: update rust extension import to use the new module policy Differential Revision: https://phab.mercurial-scm.org/D6677

transaction: leave unfinished without crashing when not properly released I think the transaction.__del__ is there just as a last resort in case we (or an extension) forgot to release the transaction. When that happens, the repo can (or will on Python 3?) get deleted before the transaction. This leads to a crash in test-devel-warnings.t on Python 3 because we tried to access repo.dirstate, where repo was retried from a weak reference. There's not much we can do here, but let's at least avoid the crash. The user will have run `hg recover` afterwards regardless. Differential Revision: https://phab.mercurial-scm.org/D6664

unshelve: add abort on using continue and interactive together `unshelve --continue --interactive` will not work as expected by the user as the mode of in-progress unshelve is preserved and cannot be overwritten. This patch makes `unshelve` to throw an error on using both `--continue` and `--interactive` together with `unshelve`. Differential Revision: https://phab.mercurial-scm.org/D6703

py3: add one more test to list of passing tests Found by buildbot. Differential Revision: https://phab.mercurial-scm.org/D6702

tests: sort imports in test-bookmarks-corner-case.t test-check-module-imports.t breaks on py3 without this change. Differential Revision: https://phab.mercurial-scm.org/D6701

fix: add some new test cases These cover a couple of behaviors we were testing at Google that weren't covered here before. Differential Revision: https://phab.mercurial-scm.org/D6698

unshelve: store information about interactive mode in shelvedstate This is a follow-up patch to 5162753c4c14. This makes `unshelve` stores the information about interactive mode on conflicts. Differential Revision: https://phab.mercurial-scm.org/D6679

unshelve: create a matcher only if required on creating unshelve ctx Differential Revision: https://phab.mercurial-scm.org/D6687

unshelve: changes how date is set on interactive mode On an interactive unshelve, the remaining changes are shelved again for later. This patch modifies the date of remaining shelved change to the time of interactive shelve. Differential Revision: https://phab.mercurial-scm.org/D6685

unshelve: unify logic around creating an unshelve changeset This is a follow-up patch to 5162753c4c14 on addressing reviews on the commit. This unifies the logic around creating an unshelve changeset. Differential Revision: https://phab.mercurial-scm.org/D6683

fix: ignore fixer tool configurations that are missing patterns This is to prevent a crash under the same circumstances. This is also to avoid data loss due to accidental application of a fixer tool to all files, if the matching logic somehow changed to that effect. Affecting all files until otherwise configured would be dangerous, and not very useful. We shouldn't abort because there may be other fixers, and it may still be useful to run them without having to adjust configuration. A user might not feel confident in changing configs, for example. Differential Revision: https://phab.mercurial-scm.org/D6693

fix: add a test case around the effect of cwd on pattern matching This was not covered by previous tests. It is related to a regression encountered at Google due to misconfiguration of [fix]. Differential Revision: https://phab.mercurial-scm.org/D6692

fix: remove support for :fileset sub-config in favor of :pattern Differential Revision: https://phab.mercurial-scm.org/D6691

fsmonitor: add support for extra `hg debuginstall` data This might make some things easier to debug, and for default bug report templates it'll help collect more data from users all at once. I don't actually need fsmonitor in our bug reports (we don't use it), but this demonstrates the utility of the preceding patches without having to add new things to core. Differential Revision: https://phab.mercurial-scm.org/D6682

debugcommands: add support for extensions adding their own debug info We've had a couple of cases where it'd be handy at Google to add data to `hg debuginstall`'s output. We've kludged around that at various times, but it seems reasonable to let extensions add their own data here so extension maintainers can get useful extra data. Differential Revision: https://phab.mercurial-scm.org/D6681

fsmonitor: refactor watchmanclient.client to accept ui and repo path This will make my next patch simpler. Differential Revision: https://phab.mercurial-scm.org/D6680

Added signature for changeset 181e52f2b62f

Added tag 5.1.2 for changeset 181e52f2b62f

merge: back out changeset a4ca0610c754 (parents order when grafting a merge) Turns out it's not enough to just swap parents, because when we do, there are unexpected bad side effects, such as a tracked file becoming untracked. These side effects need more code to be handled properly, but it's not written yet. Let's back this feature out from stable for now and some day implement it on default instead.

merge: respect parents order when using `graft` on a merge, this time for real See a4ca0610c754. potherp1 is a boolean variable that means "pother is ctx.p1", and parents is naturally [ctx.p1, ctx.p2]. pctx is always removed from parents, so if pctx is parents[0], then we end up using parents[1] as pother. To be true to its name, potherp1 should then be True only when pctx is at parents[1].

phabricator: don't abort if property writing fails during amending Currently if one of the writediffproperty calls fails due to network issues during the amending of commit messages to include the Diff. Rev. line, the transaction is aborted and rolled back. This means that the associations between the commits and the DREVs are lost for any already amended commits because the removal of the local tags isn't covered by the rollback. Differential Revision: https://phab.mercurial-scm.org/D6835

merge: respect parents order when using `graft` on a merge The previous code did not record the index of the replaced parent. It was always using the "graft" destination as `p1`. This could switch parents order in some situation (eg: some of the evolve evolving merge case). Recording and using the information fixes the issue in evolve. We are not aware of core commands calling graft in that fashion, so we could not build a simple test case for it using core commands.

tests: register test-merge-combination.t as small but slow run-tests.py use file size as an heuristic for test run time. The new `test-merge-combination.t` is a small file that do a lot of processing. As a result it tend to be scheduled really late but delay the full test run by a lot. On an example test run, the one-before-last test completed 279s after the start of the run, while `test-merge-combination.t` finished 355s after it. A 76s delay. This delay can be avoided since `test-merge-combination.t` only got started 175s after the start of the run.

test: allow different result for zstd compression (issue6188) test-repo-compengines fails on big-endian due to different file size, but the repo doesn't seem broken, so allow both sizes. Differential Revision: https://phab.mercurial-scm.org/D6787

Added signature for changeset a4e32fd539ab

Added tag 5.1.1 for changeset a4e32fd539ab

python-zstandard: apply big-endian fix (issue6188) This is a port of commit d4baf1f95b811f40773f5df0d8780fb2111ba6f5 from the upstream project to fix python-zstandard on 64-bit big-endian.

exchange: abort on pushing bookmarks pointing to secret changesets (issue6159) Until now, if there is a bookmark points to a changeset which is in secret phase, hg will push the bookmark, but not the changeset referenced by that bookmark. This leaves the server bookmarks in a bad state, because that bookmark now points to a revision that does not exist on the server. This patch makes hg to abort on such cases. Differential Revision: https://phab.mercurial-scm.org/D6731

tests: add test to demonstrate issue6159 Differential Revision: https://phab.mercurial-scm.org/D6740

packaging: add Bullseye, remove Jessie Jessie is oldoldstable now, and Bullseye is going to be the next stable (now testing). We're continuing to support the current oldstable, stable and testing releases. Differential Revision: https://phab.mercurial-scm.org/D6762

packaging: add Cosmic and Disco, remove Trusty and Artful - Trusty was publicly supported until 2019-04-30 - Artful was publicly supported until 2018-07-19 - Cosmic was publicly supported until 2019-07-18 - Disco will be publicly supported until 2020-01 Cosmic is officially out-of-date, but since it still may be in use, and because we didn't add it when it first came out, I think it would be nice to support it until the next time somebody decides to update this list of Ubuntu releases. Differential Revision: https://phab.mercurial-scm.org/D6761

makefile: run Rust tests if cargo is installed While no particular minimum toolchain version is targeted as of yet, this serves as a first step to make more people/machines run the Rust tests.

tests: use `tr -d` and not `tr --delete` as the latter is absent on BSD tr(1) Differential Revision: https://phab.mercurial-scm.org/D6729

fncache: make debugrebuildfncache not fail on broken fncache The code reading the fncache changed in 5.0, to complain if the file is not \n terminated. This makes apparent the fact that the fncache gets corrupted. Make it possible to recover, instead of having `hg debugrebuildfncache` failing by saying `(run hg debugrebuildfncache)`. The corruption itself is most likely due to hg not using fsync in general, and so various bad things can happen. Here, the reported problems happened when running out of disk space. So I suspect that because the fncache is much bigger than the average commit/pull, when running out of disk space, the bulk of the pull may succeed, but the new fncache may get half-written and still renamed into place. Differential Revision: https://phab.mercurial-scm.org/D6722

fncache: show that debugrebuildfncache is partly broken Differential Revision: https://phab.mercurial-scm.org/D6721

test: further fixes to matching for run-tests.py bug The fix in bac24a8a095a did not fix the full issue, because the extra number also eat some of the separator space. Since we are already matching arbitrary number of space, we easily fix the matching.

demandimport: explicitly declare `_session` at the module level The `_session` module level variable is set within a function using the `global` keyword. This confuses my `test-check-pyflakes.t`. Explicitly declaring the variable at the top level solves the issue (and seems absolutely reasonable).

tests: give more room for slowness in test-run-tests.t The test expected any run-test.py run to end in less than 10 seconds. On slower loaded CI machine, this gets slower than that. We give a bit more room to the regexp.

relnotes: copy "next" to "5.1" and clear "next" To avoid merge conflicts, we want to avoid modifying the file on multiple branches in parallel. This patch is therefore meant to be applied to the stable branch and then quickly be merged to default (at least before edits are made to relnotes/next there). Another option would have been to copy the file on the stable branch and to clear it on the default branch. However, that still results in conflicts if the copy is edited on the stable branch (Mercurial would try to apply the changes from the default branch to it). We could also delete the file in one commit and recreate it in another commit. However, Mercurial is quite inconsistent in what it considers a break in history (see test-copies-unrelated.t), so I'd like to avoid that. Differential Revision: https://phab.mercurial-scm.org/D6705

automation: push changes affecting .hgtags When I went to build the 5.1 tag using the in-repo automation, the automatic version calculation failed to deduce the clean 5.1 version string because we had only pushed the changeset corresponding to the 5.1 tag and not the changeset containing the 5.1 tag. So from the perspective of the remote repo, the 5.1 tag didn't exist yet and automatic version deduction failed. This commit changes the `hg push` to also push all changesets affecting the .hgtags file, ensuring the remote has up-to-date tags information. I tested this by creating a local draft changeset with a dummy tag value on a different DAG head and instructed the automation to build a revision that didn't have this change to .hgtags. The tag was successfully pushed and the built package had a version number incorporating that tag. Sending this to stable so the 5.1.1 automation hopefully "just works."

bookmarks: actual fix for race condition deleting bookmark This is a simple but efficient fix to prevent the issue tested in `test-bookmarks-corner-case.t`. It might be worth pursuing a more generic approach where filecache learn to depend on each other, but that would not be suitable for stable. The issue is complicated enough that I documented the race and its current solution as inline comment. See this comment for details on the fix.

strip: access bookmark before getting a reference to changelog Bookmark access might invalidate the current changelog (to make sure both are in a reasonable synchronisation state). So we should grab the reference to changelog after we access bookmark. Otherwise we risk using a dead object for the whole strip process. (note: this dead object business probably requires a new layers of checking)

test: use a more verbose output in the test While debugging some test failure, I released the test never checks if the relevant changesets were preserved. So I am updating the test from `hg parents` usage to `hg log -G` with a special template. This increase the area covered by the test and clarify the test failures.

Added signature for changeset e91930d712e8

Added tag 5.1 for changeset e91930d712e8

automation: execute powershell when connecting For some reason, the ability to execute PS scripts appears to come online after the ability to execute regular command scripts. This is creating race conditions when connecting to instances resulting in our wait_for_winrm() returning before PS is available leading to an exception being thrown in other code. Let's change the client connection code to execute a minimal PS script so we can try to trap the exception in wait_for_winrm().

automation: allow exit code of 1 for `hg push` `hg push` exits 1 for no-ops. No-op pushes should be fine in the context of automation.

curses: do not setlocale() at import time (issue5261) setlocale() can break date formatting/parsing functions because they are locale dependent. We should avoid doing setlocale() as possible. This patch moves setlocale() just before curses.wrapper(), which function is documented to "initialize curses." I don't know the details about the curses initialization, but I *think* this would work as well. Maybe we can extract a curses setup function later. https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-February/128788.html

contrib: install Python 3.8b2 instead of 3.8a2 Let's install the most recent Python 3.8 distribution. Differential Revision: https://phab.mercurial-scm.org/D6674

automation: make Windows base image name configurable Since automation broke in the middle of the 5.0 release cycle, there's a good chance it will break again in the future. While a robust solution might be to search for all available images and choose the newest one, it does seem useful to be able to explicitly choose the name of the image to find and use so users can opt in to using a different image. This commit implements that functionality. Differential Revision: https://phab.mercurial-scm.org/D6673

automation: extract strings to constants Differential Revision: https://phab.mercurial-scm.org/D6672