packaging: add Cosmic and Disco, remove Trusty and Artful - Trusty was publicly supported until 2019-04-30 - Artful was publicly supported until 2018-07-19 - Cosmic was publicly supported until 2019-07-18 - Disco will be publicly supported until 2020-01 Cosmic is officially out-of-date, but since it still may be in use, and because we didn't add it when it first came out, I think it would be nice to support it until the next time somebody decides to update this list of Ubuntu releases. Differential Revision: https://phab.mercurial-scm.org/D6761

makefile: run Rust tests if cargo is installed While no particular minimum toolchain version is targeted as of yet, this serves as a first step to make more people/machines run the Rust tests.

tests: use `tr -d` and not `tr --delete` as the latter is absent on BSD tr(1) Differential Revision: https://phab.mercurial-scm.org/D6729

fncache: make debugrebuildfncache not fail on broken fncache The code reading the fncache changed in 5.0, to complain if the file is not \n terminated. This makes apparent the fact that the fncache gets corrupted. Make it possible to recover, instead of having `hg debugrebuildfncache` failing by saying `(run hg debugrebuildfncache)`. The corruption itself is most likely due to hg not using fsync in general, and so various bad things can happen. Here, the reported problems happened when running out of disk space. So I suspect that because the fncache is much bigger than the average commit/pull, when running out of disk space, the bulk of the pull may succeed, but the new fncache may get half-written and still renamed into place. Differential Revision: https://phab.mercurial-scm.org/D6722

fncache: show that debugrebuildfncache is partly broken Differential Revision: https://phab.mercurial-scm.org/D6721

test: further fixes to matching for run-tests.py bug The fix in bac24a8a095a did not fix the full issue, because the extra number also eat some of the separator space. Since we are already matching arbitrary number of space, we easily fix the matching.

demandimport: explicitly declare `_session` at the module level The `_session` module level variable is set within a function using the `global` keyword. This confuses my `test-check-pyflakes.t`. Explicitly declaring the variable at the top level solves the issue (and seems absolutely reasonable).

tests: give more room for slowness in test-run-tests.t The test expected any run-test.py run to end in less than 10 seconds. On slower loaded CI machine, this gets slower than that. We give a bit more room to the regexp.

relnotes: copy "next" to "5.1" and clear "next" To avoid merge conflicts, we want to avoid modifying the file on multiple branches in parallel. This patch is therefore meant to be applied to the stable branch and then quickly be merged to default (at least before edits are made to relnotes/next there). Another option would have been to copy the file on the stable branch and to clear it on the default branch. However, that still results in conflicts if the copy is edited on the stable branch (Mercurial would try to apply the changes from the default branch to it). We could also delete the file in one commit and recreate it in another commit. However, Mercurial is quite inconsistent in what it considers a break in history (see test-copies-unrelated.t), so I'd like to avoid that. Differential Revision: https://phab.mercurial-scm.org/D6705

automation: push changes affecting .hgtags When I went to build the 5.1 tag using the in-repo automation, the automatic version calculation failed to deduce the clean 5.1 version string because we had only pushed the changeset corresponding to the 5.1 tag and not the changeset containing the 5.1 tag. So from the perspective of the remote repo, the 5.1 tag didn't exist yet and automatic version deduction failed. This commit changes the `hg push` to also push all changesets affecting the .hgtags file, ensuring the remote has up-to-date tags information. I tested this by creating a local draft changeset with a dummy tag value on a different DAG head and instructed the automation to build a revision that didn't have this change to .hgtags. The tag was successfully pushed and the built package had a version number incorporating that tag. Sending this to stable so the 5.1.1 automation hopefully "just works."

bookmarks: actual fix for race condition deleting bookmark This is a simple but efficient fix to prevent the issue tested in `test-bookmarks-corner-case.t`. It might be worth pursuing a more generic approach where filecache learn to depend on each other, but that would not be suitable for stable. The issue is complicated enough that I documented the race and its current solution as inline comment. See this comment for details on the fix.

strip: access bookmark before getting a reference to changelog Bookmark access might invalidate the current changelog (to make sure both are in a reasonable synchronisation state). So we should grab the reference to changelog after we access bookmark. Otherwise we risk using a dead object for the whole strip process. (note: this dead object business probably requires a new layers of checking)

test: use a more verbose output in the test While debugging some test failure, I released the test never checks if the relevant changesets were preserved. So I am updating the test from `hg parents` usage to `hg log -G` with a special template. This increase the area covered by the test and clarify the test failures.

Added signature for changeset e91930d712e8

Added tag 5.1 for changeset e91930d712e8

automation: execute powershell when connecting For some reason, the ability to execute PS scripts appears to come online after the ability to execute regular command scripts. This is creating race conditions when connecting to instances resulting in our wait_for_winrm() returning before PS is available leading to an exception being thrown in other code. Let's change the client connection code to execute a minimal PS script so we can try to trap the exception in wait_for_winrm().

automation: allow exit code of 1 for `hg push` `hg push` exits 1 for no-ops. No-op pushes should be fine in the context of automation.

curses: do not setlocale() at import time (issue5261) setlocale() can break date formatting/parsing functions because they are locale dependent. We should avoid doing setlocale() as possible. This patch moves setlocale() just before curses.wrapper(), which function is documented to "initialize curses." I don't know the details about the curses initialization, but I *think* this would work as well. Maybe we can extract a curses setup function later. https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-February/128788.html

contrib: install Python 3.8b2 instead of 3.8a2 Let's install the most recent Python 3.8 distribution. Differential Revision: https://phab.mercurial-scm.org/D6674

automation: make Windows base image name configurable Since automation broke in the middle of the 5.0 release cycle, there's a good chance it will break again in the future. While a robust solution might be to search for all available images and choose the newest one, it does seem useful to be able to explicitly choose the name of the image to find and use so users can opt in to using a different image. This commit implements that functionality. Differential Revision: https://phab.mercurial-scm.org/D6673

automation: extract strings to constants Differential Revision: https://phab.mercurial-scm.org/D6672

automation: use newer Windows base image It looks like the old base image disappeared. Let's use a newer image that exists today. Differential Revision: https://phab.mercurial-scm.org/D6671

copies: fix crash on in changeset-centric tracing from commit to itself When we trace copies from a changeset to itself, the "work" queue ends up empty and we hit the "assert False" after it. It was only the last of the three added tests that failed before this patch. That is because the other two cases have fast paths, so _committedforwardcopies() is never reached. Differential Revision: https://phab.mercurial-scm.org/D6675

unshelve: add help text on --interactive in verbose mode This is a follow-up patch to rHG9eace8d6d537. This modifies the help text of unshelve in verbose mode to mention the details about `--interactive` flag. Differential Revision: https://phab.mercurial-scm.org/D6676

amend: stop committing unrequested file reverts (issue6157) Differential Revision: https://phab.mercurial-scm.org/D6667

amend: add a test for a simplified version of issue6157 Differential Revision: https://phab.mercurial-scm.org/D6666

py: error out if a "skip" character was given with non-dict to util.dirs() util.dirs() keeps track of the directories in its input collection. If a "skip" character is given to it, it will assume the input is a dirstate map and it will skip entries that are in the given "skip" state. I think this is used only for skipping removed entries ("r") in the dirtate. The C implementation of util.dirs() errors out if it was given a skip character and a non-dict was passed. The pure implementation simply ignored the request skip state. Let's make it easier to discover bugs here by erroring out in the pure implementation too. Let's also switch to checking for the dict-ness, to make the C implementation (since that's clearly been sufficient for many years). This last change makes test-issue660.t pass on py3 in pure mode, since the old check was for existence of iteritems(), which doesn't exist on py3. Differential Revision: https://phab.mercurial-scm.org/D6669

py3: fix incorrect fix of test-setdiscovery.t in eb27d9eee2cc Both places should have been changed from 185 to 187. Differential Revision: https://phab.mercurial-scm.org/D6668

Added signature for changeset e386b5f4f836

Added tag 5.1rc0 for changeset e386b5f4f836

merge default into stable for 5.1 release

rust-filepatterns: unescape comment character property There were multiple issues in the original implementation: a. the local variable "line" dropped soon after replace_slice() applied b. replace_slice() was noop since br"\#".len() != b"#" This patch uses bytes::Regex::replace_all() since it seems the simplest way to replace bytes of arbitrary length, and I don't think we have to avoid using Regexp here.

rust-filepatterns: use literal b'#' instead of cast

rust-filepatterns: fix type of warnings tuple to (bytes, bytes) Otherwise warn() in match.py would fail if the warning contains non-ASCII character. We might want to add a thin ByteString wrapper around Vec<u8> to implement ToPyObject<ObjectType = PyBytes>, but I'm not sure.

hgignore: add escape syntax test for glob patterns The last example, [\#], is what the rust implementation fails to parse. The other escapes can be removed by regexp engine or _globre().

hgignore: add a few more weird patterns to test case

hgignore: update \-escape test to reflect actual behavior "\\<char>" is not an escape character but "\\" + <char>.

py3: add a b'' prefix in tests/test-convert-identity.t Differential Revision: https://phab.mercurial-scm.org/D6662

lookup: don't use "00changelog.i@None" when lookup of prefix fails We were shadowing the "node" variable, so we always passed None to the LookupError instead of the node we meant to pass. (This showed up in py3 tests since py3 doesn't like to format None using "%s".) Differential Revision: https://phab.mercurial-scm.org/D6661

py3: fix test-setdiscovery.t on Python 3 by conditionalizing two lines I'm not clear why this behaves very slightly differently on Python 3, but I'm also not concerned about it. Differential Revision: https://phab.mercurial-scm.org/D6658

commands: removed part of description from abort and continue The description for registration of new `continuefunc` or `abortfunc` is removed as it is not required from user perspective. Differential Revision: https://phab.mercurial-scm.org/D6660

tests: glob over some timing numbers in test-shelve.t The Windows bot is slow enough that it was 2s in the first hunk. Differential Revision: https://phab.mercurial-scm.org/D6663

py3: another passing test Differential Revision: https://phab.mercurial-scm.org/D6656

cleanup: remove redundant import For some reason the import checker only caught this on py3. Differential Revision: https://phab.mercurial-scm.org/D6657

shelve: modify help text on --interactive We now have `unshelve --interactive` after rHG5162753c4c14. So, the help text on `shelve --interactive` suggesting that it only works for `shelve` can be removed. Differential Revision: https://phab.mercurial-scm.org/D6654

unshelve: mark unshelve interactive as experimental This is a follow-up patch to rHG5162753c4c14. We have the logic for interactive unshelve under `_rebaserestorecommit()`. So, we might get conflicts even if there are conflicting changes other than selected changes by the user. We should mark unshelve `--interactive` as `EXPERIMENTAL` until we solve this issue. Differential Revision: https://phab.mercurial-scm.org/D6653

commit: improve the files field of changelog for merges Currently, the files list of merge commits repeats all the deletions (either actual deletions, or files that got renamed) that happened between base and p2 of the merge. If p2 is the main branch, the list can easily be much bigger than the change being merged. This results in various problems worth improving: - changelog is bigger than necessary - `hg log directory` lists many unrelated merge commits, and `hg log -v -r commit` frequently fills multiple screens worth of files - it possibly slows down adjustlinkrev, by forcing it to read more manifests, and that function can certainly be a bottleneck - the server side of pulls can waste a lot of time simply opening the filelogs for pointless files (the constant factors for opening even a tiny filelog is apparently pretty bad) So stop listing such files as described in the code. Impacted merge commits and their descendants get a different hash than they would have without this. This doesn't seem problematic, except for convert. The previous commit helped with that in the hg->hg case (but if you do svn->hg twice from scratch, hashes can still change). The rest of the description is numbers. I don't have much to report, because recreating the files list of existing repositories is not easy: - debugupgradeformat and bundle/unbundle don't recreate the list - export/import tends to choke quickly applying patches or on description that contain diffs, - merge commits from the convert extension don't have the right files list for reasons orthogonal to the current commit - replaying the merge with hg update/hg merge/hg revert --all/hg commit can end up failing in hg revert - I wasn't sure that using debugsetparents + debugrebuilddirstate would really build the right thing I measured commit time before and after this change, in a case with no files filtered out, several files filtered out (no difference) and 5k files filtered out (+1% time). Recreating the 100 more recent merges in a private repo, the concatenated uncompressed files lists goes from 1.12MB to 0.52MB. Excluding 3 merges that are not representative, then the size goes from 570k to 15k. I converted part of mozilla-central, and observed file list shrinking quite a bit too, starting at the very first merge, 733641d9feaf, going from 550 files to 10 files (although they have relatively few merges, so they probably wouldn't care). Differential Revision: https://phab.mercurial-scm.org/D6613

convert: add a config option to help doing identity hg->hg conversion I want to change the computation of the list of files modified by a commit. In principle, this would simply change a cache. But since this information is stored in commits rather than a cache, changing it means changing commit hashes (going forward). Some users rely on the convert extension from hg to hg not changing hashes when nothing changes (usually). Allow these users to preserve hashes despite changes to the changelog files computation by reusing these files lists when the manifest is unchanged (since these files list are derived from the manifest). Differential Revision: https://phab.mercurial-scm.org/D6643

tests: show the files fields of changelogs for many merges I don't think there's coverage for many of the subtle cases, and I found it hard to understand what the code is doing by reading it. The test takes 40s to run on a laptop, or 9s with --chg. I have yet to find a description of what the files field is supposed to be for merges. I thought it could be one of: 1. the files added/modified/removed relative to p1 (wouldn't seem useful, but `hg diff -c -r mergerev` has this behavior) 2. the files with filelog nodes not in either parent (i.e., what is needed to create a bundle out of a commit) 3. the files added/removed/modified files by merge itself [1] It's clearly not 1, because file contents merges are symmetric. It's clearly not 2 because removed files and exec bit changes are listed. It's also not 3 but I think it's intended to be 3 and the differences are bugs. Assuming 3, the test shows that, for merges, the list of files both overapproximates and underapproximates. All the cases involve file changes not in the filelog but in the manifest (existence of file at revision, exec bit and file vs symlink). I didn't look at all underapproximations, but they looked minor. The two overapproximations are problematic though because they both cause potentially long lists of files when merging cleanly. [1] even what it means for the merge commit itself to change a file is not completely trivial. A file in the merge being the same as in one of the parent is too lax as it would consider that merges change nothing when they revert all the changes done on one side. The criteria used in the test and in the next commit for "merge didn't touch a file" is: - the parents and the merge all have the same file - or, one parent didn't touch the file and the other parent contains the same file as the merge Differential Revision: https://phab.mercurial-scm.org/D6612

phabricator: handle local:commits time being string or int When setting local:commits arcanist has different behaviour depending on whether the repo is git or hg. With hg it sets the time as a number, since it calls PHP's strtotime on the value, but with git it sets it as a string. Normally this wouldn't be an issue since phabread wouldn't be interacting with Phabricator Revisions for git repos, but Mozilla has a secondary workflow for git users that uses the git-cinnabar tool to interact with their hg repos. When a git-cinnabar user uses the moz-phab tool to submit patches for mozilla-central it makes use of Mozilla's fork of arcanist, which works with their local git version of m-c, and thus sets the local:commit time as a string, and then translates the commit hashes. Currently when encountering such DREVS phabread dies with "TypeError: %d format: a number is required, not str". phabsend also used to set it as a string but wouldn't have encountered the issue with its own DREVs since it would read hg:meta first. Differential Revision: https://phab.mercurial-scm.org/D6650

phabricator: demonstrate broken phabread on string local:commit times Differential Revision: https://phab.mercurial-scm.org/D6649

unshelve: add interactive mode Until now, there is no way to `unshelve` selected changes only from the stored shelve as given in issue6162. This patch makes `unshelve` perform with certain changes only by adding an interactive mode. Differential Revision: https://phab.mercurial-scm.org/D6596

blackbox: disable extremely verbose logging (issue6110) This is maybe not the best way to go about fixing this, but anything is better than the status quo. Differential Revision: https://phab.mercurial-scm.org/D6611

continue: added support for unshelve This patch adds the support for `ushelve` in `hg continue` plan. `hgcontinueunshelve()` has been created for independent calls. In case an interrupted unshelve is resumed via hg continue the shelvedstate needs to be loaded seperately. This has been ensured by `_loadunshelvedstate()` `hgcontinueunshelve()` is then registered as `continuefunc` for state detection API. Results are shown as tests. Differential Revision: https://phab.mercurial-scm.org/D6652

continue: added support for rebase This adds support of rebase to hg continue plan. An independent continue logic for rebase is created under continuerebase() function. For this a seperate rebaseruntime object is created under the function to handle an interrupted rebasestate. Results of tests are shown. Differential Revision: https://phab.mercurial-scm.org/D6646

continue: added logic for hg continue This is part of GSoC19 project `Implement abort and continue commands`. This patch is part of the continue plan. This adds the basic logic for hg continue. This command aborts an multistep operation like graft, histedit, rebase, transplant and unshelve if they are in an unfinished state. The first part of the logic is determining the unfinished operation from the state detection API under statemod. This API is extended to support hg continue by adding a method to register the abort logic as a function (here continuefunc). Once the unfinished operation is determined the registered logic is used to resume the command in case it is interrupted. The benefit of this kind of framework is that any new extension developed can support hg continue by registering the command and logic under statedetection API. hg continue currently supports --dry-run/-n flag only. It is used to dry run hg abort Differential Revision: https://phab.mercurial-scm.org/D6645