tests: demonstrate crash when committing subset of copies to changeset When writing copy metadata to the changeset and not committing all copies in the dirstate, we get a ProgrammingError. This commit adds two tests showing how to trigger this bug. Differential Revision: https://phab.mercurial-scm.org/D6749

bdiff-torture: fix pyflakes warning reporting undefined name 'inst' Looks like I got a latest version of pyflakes somehow or it's running on py3 and it spotted this. Differential Revision: https://phab.mercurial-scm.org/D6757

split: handle partial commit of renames when doing split or record (issue5723) When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This was because the logic for handling partial application of the patches relies on knowing what files are "new with modifications" and it doesn't treat "rename destination" as "new". There was a complicating issue, however. We're relying on the patch header specifying the copy from/to information, which works as long as the 'copy from' file is there. In the case of renames, however, the 'rename from' file is *not* there, so we need to add it back. Differential Revision: https://phab.mercurial-scm.org/D6768

split: handle partial commit of copies when doing split or record When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This appears to be because the logic for handling partial application of the patches relies on knowing what files are "new with modifications", and it doesn't treat "copy destination" as "new". Handling renames correctly is more difficult and will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D6767

py3: use pycompat.sysargv[0] for instead of fsencode(sys.argv[0]) Yuya noted in a recent review that fsencode(sys.argv[0]) could be incorrect on Windows. Differential Revision: https://phab.mercurial-scm.org/D6782

httppeer: use context manager when reading temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6784

httppeer: use context manager when writing temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6783

rust-cpython: mark unsafe functions as such It wasn't trivial to fix leak_immutable() to be safe since we have to allow immutable operations (e.g. iter()) on the leaked reference. So let's mark it unsafe for now. Callers must take care of the returned object to guarantee the memory safety. I'll revisit this later. I think $leaked<T: 'static> could have a function that converts itself into $leaked<U: 'static> with a given FnOnce(&T) -> &U, where T is $inner_struct, and U is $iterator_type for example.

rust-cpython: pair leaked reference with its manager object Still leak_immutable() is unsafe since leak_handle must live longer than the leaked_ref.

rust-cpython: introduce restricted variant of RefCell This should catch invalid borrow_mut() calls. Still the ref-sharing interface is unsafe.

tests: register test-merge-combination.t as small but slow run-tests.py use file size as an heuristic for test run time. The new `test-merge-combination.t` is a small file that do a lot of processing. As a result it tend to be scheduled really late but delay the full test run by a lot. On an example test run, the one-before-last test completed 279s after the start of the run, while `test-merge-combination.t` finished 355s after it. A 76s delay. This delay can be avoided since `test-merge-combination.t` only got started 175s after the start of the run.

test: allow different result for zstd compression (issue6188) test-repo-compengines fails on big-endian due to different file size, but the repo doesn't seem broken, so allow both sizes. Differential Revision: https://phab.mercurial-scm.org/D6787

rust-cpython: fix unsafe inner(py).borrow_mut() calls Since self.inner is managed by PySharedState, it must not be borrowed mutably through the RefCell interface. Otherwise, the underlying object could be mutated while a reference is leaked to Python world.

revlog: deprecate the use of `revision(..., raw=True)` We have an official `rawdata` function now.

Added signature for changeset a4e32fd539ab

Added tag 5.1.1 for changeset a4e32fd539ab

remotefilelog: reduce probability of race-condition in remotefilelog tests ca1014ad3de4 introduced a new parameter `ensurestart` to speed up remotefilelog background processes start. Unfortunately it seems to have increased the possibility of race-conditions in remotefilelog tests testing those background processes. With `ensurestart=False`, it seems that it's more probable to enter in a race condition with `debugwaitonprefetch` and `debugwaitonrepack` in remotefilelog background tests. Our CI seems to have a high probability of triggering this race condition so make it configurable to ensure tests are stable. Differential Revision: https://phab.mercurial-scm.org/D6772

rust: apply more formatting fixes My cargo fmt updated these lines and they look good.

rust-utils: add normalize_case util to mirror Python one While we still don't handle filenames properly cross-platform, this at least sticks closer to the Python behavior. Differential Revision: https://phab.mercurial-scm.org/D6756

rust: fix warnings about trait objects without dyn being deprecated Differential Revision: https://phab.mercurial-scm.org/D6770

py3: convert hg executable path to bytes in missing case in procutil We (Google) noticed this in our tests when we use chg and a hg wrapper script not called 'hg'. The executable then ended up being a native string, which then failed in chgserver when trying to convert the environment dict to a byte string. Differential Revision: https://phab.mercurial-scm.org/D6775

py3: make statprof's chrome output work With this patch, this command works: python3 hg --profile --config profiling.statformat=chrome st (and it works with s/python3/python2/ as well) Differential Revision: https://phab.mercurial-scm.org/D6777

py3: for statprof's Chrome output, write json to string, then encode to bytes `json.dump(obj, fp)` requires `fp.write()` to accept str output, and since the file pointer we have there only accepts bytes, we need to change to json.dumps() and then encode as utf-8. We have already done the same thing for the json (non-Chrome) format in 4b7eb862692e (py3: encode json output to bytes and use write(), 2018-10-12). Differential Revision: https://phab.mercurial-scm.org/D6781

statprof: use context manager for file when writing flame graph Differential Revision: https://phab.mercurial-scm.org/D6780

statprof: use context manager when reading source from file Differential Revision: https://phab.mercurial-scm.org/D6779

statprof: clarify by naming tuple members while enumerate()'ing Differential Revision: https://phab.mercurial-scm.org/D6778

upgrade: make sure we reclone all revlogs when updating to some format Adding or removing some requirement (eg: sparserevlog), requires to reclone revlog to use the new format. We cannot simply copy the original files. In this case, we issue a warning to proceed with clone every revlogs.

upgrade: add an argument to control changelog upgrade Same as for `--manifest` we can now select more selection.

upgrade: add an argument to control manifest upgrade The argument can be used to only "clone" manifest revlog or clone all of them but this one. The selection will make more sense once we have a `--changelog` flag in the next changesets.

unionrepo: drop the custom `rawdata` implementation We can rely on the main one now.

unionrepo: drop `baserevdiff` It has no caller anymore.

unionrepo: use normal inheritance scheme to call revdiff

unionrepo: fix `revdiff` implementation to use `rawdata` The parent code is using rawdata so we should use it here. Before this change, union repo was probably broken with some flag processors.

unionrepo: get rid of `baserevision` The method is not called anywhere anymore, so we can safely drop it. Some of the comment get moved to `baserevdiff` because we did not got rid of it (yet).

unionrepo: use a lower level overide in unionrepo too The unionrepo class also have a strange `baserevision` hack, let's try to get ride of it too.

bundlerepo: drop the custom `rawdata` implementation We can rely on the main one now.

bundlerepo: drop the `baserevision` hack It is not used anywhere anymore, so we can safely drop it.

bundlerepo: simplify code to take advantage of `_rawtext` In the revlog code, the code getting the raw text is now isolated. We take advantage of this to simplify the bundlerepo code.

merge with stable

rust: run cargo fmt

py3: use pycompat.maplist() in chgserver test-chg.t almost passes on py3 after this patch. Differential Revision: https://phab.mercurial-scm.org/D6771

run-tests: handle --local before --with-hg We no longer support them both together, so this is now safe to do. By checking --local first, we avoid error out about an invalid --with-hg script if --local was also given (we instead tell the user that the options are mutually exclusive). I also had to wrap the 'binpath' we pass to setattr() in _strpath() to keep `python3 run-tests.py -l` working. That change also made `python3 run-tests.py -l --chg` work, which was the reason for this series. Differential Revision: https://phab.mercurial-scm.org/D6760

run-tests: error out on `--local --with-[c]hg` I don't see much reason to allow these combinations. You could use --local and override only one of --with-hg or --with-chg, but I don't see much practical use for that. It would be easy to work around anyway by passing both --with-hg and --with-chg. By erroring out, it makes the code a bit easier to reason about to allow the next few patches. Differential Revision: https://phab.mercurial-scm.org/D6759

contrib: simplify the genosxversion.py command to find the hg libraries I forget what problem I ran into while trying to teach the makefile to use a non-system python. (It might have ben missing hg-evolve and/or keyring, but `check_output()` was raising an error.) This still isn't great because it will return non zero for something like the username not being set, even though we aren't asking for it. But I suppose it's still useful to simplify. Differential Revision: https://phab.mercurial-scm.org/D6753

interfaceutil: move to interfaces/ Now that we have a dedicated folder for interfaces, let's move interfaceutil there. Differential Revision: https://phab.mercurial-scm.org/D6742

interfaces: create a new folder for interfaces and move repository.py in it I was trying to understand current interfaces and write new ones and I realized we need to improve how current interfaces are organised. This creates a dedicated folder for defining interfaces and move `repository.py` which defines all the current interfaces inside it. Differential Revision: https://phab.mercurial-scm.org/D6741

python-zstandard: apply big-endian fix (issue6188) This is a port of commit d4baf1f95b811f40773f5df0d8780fb2111ba6f5 from the upstream project to fix python-zstandard on 64-bit big-endian.

exchange: abort on pushing bookmarks pointing to secret changesets (issue6159) Until now, if there is a bookmark points to a changeset which is in secret phase, hg will push the bookmark, but not the changeset referenced by that bookmark. This leaves the server bookmarks in a bad state, because that bookmark now points to a revision that does not exist on the server. This patch makes hg to abort on such cases. Differential Revision: https://phab.mercurial-scm.org/D6731

tests: add test to demonstrate issue6159 Differential Revision: https://phab.mercurial-scm.org/D6740

packaging: add Bullseye, remove Jessie Jessie is oldoldstable now, and Bullseye is going to be the next stable (now testing). We're continuing to support the current oldstable, stable and testing releases. Differential Revision: https://phab.mercurial-scm.org/D6762

packaging: add Cosmic and Disco, remove Trusty and Artful - Trusty was publicly supported until 2019-04-30 - Artful was publicly supported until 2018-07-19 - Cosmic was publicly supported until 2019-07-18 - Disco will be publicly supported until 2020-01 Cosmic is officially out-of-date, but since it still may be in use, and because we didn't add it when it first came out, I think it would be nice to support it until the next time somebody decides to update this list of Ubuntu releases. Differential Revision: https://phab.mercurial-scm.org/D6761

narrow: fix typo "respositories" Differential Revision: https://phab.mercurial-scm.org/D6758

merge with stable

makefile: run Rust tests if cargo is installed While no particular minimum toolchain version is targeted as of yet, this serves as a first step to make more people/machines run the Rust tests.

merge: hint about using `hg resolve` for resolving conflicts This was suggested by one of our users at Google. Makes sense to me. Differential Revision: https://phab.mercurial-scm.org/D6755

rust-dirstate: remove test case for DirsMultiset::new(Manifest, Some) It's no longer possible.

rust-dirstate: split DirsMultiset constructor per input type Since skip_state only applies to dirstate, it doesn't make sense to unify these constructors and dispatch by enum.

rust-dirstate: remove excessive clone() of parameter and return value I think pass-by-ref is preferred in general.

rust-dirstate: handle invalid length of p1/p2 parameters It uses match syntax since map_err() failed to deduce the argument type.

rust: simply use TryInto to convert slice to array Since our rust module depends on TryInto, there's no point to avoid using it. While rewriting copy_into_array(), I noticed CPython interface doesn't check the length of the p1/p2 values, which is marked as TODO.