flagutil: use it in simplestorerepo This remove the other code duplication of the `_processflags` code. To be honest, this code looks very dead since it is not run by either developer nor buildbot. However, we update the code to make the task of reviving this less daunting. Differential Revision: https://phab.mercurial-scm.org/D6799

flagutil: make the error class used by the mixin configurable One of the code duplication use a different error class. So let's make it possible to do so. Differential Revision: https://phab.mercurial-scm.org/D6798

flagutil: use the new mixin use in remotefilelog This remove one of the code duplication. Hooray \o/. Differential Revision: https://phab.mercurial-scm.org/D6797

flagutil: introduce a flagprocessorsmixin class To avoid code duplication, we will provide a simple "ready to use" mixin that carry the appropriate logic. First we use it in standard revlog, we'll remove code duplication in later changesets. Differential Revision: https://phab.mercurial-scm.org/D6796

check-code: allow command substitution with $(command) Both `command` and $(command) are specified by POSIX. The latter nests better. I don't see why we shouldn't allow both (or only the latter). Differential Revision: https://phab.mercurial-scm.org/D6789

run-tests: rename `lcmd` variable to `line_cmd` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: rename `lout` variable to `out_line` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: clarify "l" variable as "out_rawline" More explicit variable name never hurt. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: use symbolic constant instead of arbitrary number line matching (This is a gratuitous cleanup that I made while investigating a bug).

rustfilepatterns: shorter code for concatenating slices Differential Revision: https://phab.mercurial-scm.org/D6765

match: simplify the regexps created for glob patterns For legibility of the resulting regexes, although it may help with performance as well. Differential Revision: https://phab.mercurial-scm.org/D6764

rustfilepatterns: refactor the pattern of removing a prefix from a &[u8] Differential Revision: https://phab.mercurial-scm.org/D6766

tests: show the pattern generated for a relative glob Differential Revision: https://phab.mercurial-scm.org/D6763

copies: remove existing copy info from the changeset on amend (BC) When amending a changeset with copy information in the changeset and the new changeset doesn't have any copy information (or similar for "filesadded" and "filesremoved"), we shouldn't keep it. A drawback of this is that we now unconditionally remove these four entries from the extras, breaking any extensions that happened to write entries with the same names (which seems very unlikely). I think I'd heard that there was list of blacklisted keys that would be removed from the extras when a commit is rewritten, but I couldn't find that. It would make sense to add the keys mentioned above there instead of the custom filtering I've added in this patch. Differential Revision: https://phab.mercurial-scm.org/D6752

tests: show invalid copies when turning off copies-in-changeset If you turn on copies in changesets and write a commit with a copy, then turn it off and amend the commit while undoing the copy, the invalid copy information will remain. The read path doesn't crash in invalid copy data, but it seems better to not produce the invalid data. Differential Revision: https://phab.mercurial-scm.org/D6751

context: filter out invalid copies from workingctx.p[12]copies() workingctx normally gets its lists of modified, added, removed files etc. based on the dirstate status. Its constructor also accepts a "changes" argument to override the status from the dirstate. This is used for partial commits. If a "changed" argument was passed, we should clearly also filter out copies to those paths, which I had previously missed. This patch adds that filtering and fixes the bugs demonstrated in the previous patch. Differential Revision: https://phab.mercurial-scm.org/D6750

tests: demonstrate crash when committing subset of copies to changeset When writing copy metadata to the changeset and not committing all copies in the dirstate, we get a ProgrammingError. This commit adds two tests showing how to trigger this bug. Differential Revision: https://phab.mercurial-scm.org/D6749

bdiff-torture: fix pyflakes warning reporting undefined name 'inst' Looks like I got a latest version of pyflakes somehow or it's running on py3 and it spotted this. Differential Revision: https://phab.mercurial-scm.org/D6757

split: handle partial commit of renames when doing split or record (issue5723) When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This was because the logic for handling partial application of the patches relies on knowing what files are "new with modifications" and it doesn't treat "rename destination" as "new". There was a complicating issue, however. We're relying on the patch header specifying the copy from/to information, which works as long as the 'copy from' file is there. In the case of renames, however, the 'rename from' file is *not* there, so we need to add it back. Differential Revision: https://phab.mercurial-scm.org/D6768

split: handle partial commit of copies when doing split or record When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This appears to be because the logic for handling partial application of the patches relies on knowing what files are "new with modifications", and it doesn't treat "copy destination" as "new". Handling renames correctly is more difficult and will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D6767

py3: use pycompat.sysargv[0] for instead of fsencode(sys.argv[0]) Yuya noted in a recent review that fsencode(sys.argv[0]) could be incorrect on Windows. Differential Revision: https://phab.mercurial-scm.org/D6782

httppeer: use context manager when reading temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6784

httppeer: use context manager when writing temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6783

rust-cpython: mark unsafe functions as such It wasn't trivial to fix leak_immutable() to be safe since we have to allow immutable operations (e.g. iter()) on the leaked reference. So let's mark it unsafe for now. Callers must take care of the returned object to guarantee the memory safety. I'll revisit this later. I think $leaked<T: 'static> could have a function that converts itself into $leaked<U: 'static> with a given FnOnce(&T) -> &U, where T is $inner_struct, and U is $iterator_type for example.

rust-cpython: pair leaked reference with its manager object Still leak_immutable() is unsafe since leak_handle must live longer than the leaked_ref.

rust-cpython: introduce restricted variant of RefCell This should catch invalid borrow_mut() calls. Still the ref-sharing interface is unsafe.

rust-cpython: fix unsafe inner(py).borrow_mut() calls Since self.inner is managed by PySharedState, it must not be borrowed mutably through the RefCell interface. Otherwise, the underlying object could be mutated while a reference is leaked to Python world.

revlog: deprecate the use of `revision(..., raw=True)` We have an official `rawdata` function now.

remotefilelog: reduce probability of race-condition in remotefilelog tests ca1014ad3de4 introduced a new parameter `ensurestart` to speed up remotefilelog background processes start. Unfortunately it seems to have increased the possibility of race-conditions in remotefilelog tests testing those background processes. With `ensurestart=False`, it seems that it's more probable to enter in a race condition with `debugwaitonprefetch` and `debugwaitonrepack` in remotefilelog background tests. Our CI seems to have a high probability of triggering this race condition so make it configurable to ensure tests are stable. Differential Revision: https://phab.mercurial-scm.org/D6772

rust: apply more formatting fixes My cargo fmt updated these lines and they look good.

rust-utils: add normalize_case util to mirror Python one While we still don't handle filenames properly cross-platform, this at least sticks closer to the Python behavior. Differential Revision: https://phab.mercurial-scm.org/D6756

rust: fix warnings about trait objects without dyn being deprecated Differential Revision: https://phab.mercurial-scm.org/D6770

py3: convert hg executable path to bytes in missing case in procutil We (Google) noticed this in our tests when we use chg and a hg wrapper script not called 'hg'. The executable then ended up being a native string, which then failed in chgserver when trying to convert the environment dict to a byte string. Differential Revision: https://phab.mercurial-scm.org/D6775

py3: make statprof's chrome output work With this patch, this command works: python3 hg --profile --config profiling.statformat=chrome st (and it works with s/python3/python2/ as well) Differential Revision: https://phab.mercurial-scm.org/D6777

py3: for statprof's Chrome output, write json to string, then encode to bytes `json.dump(obj, fp)` requires `fp.write()` to accept str output, and since the file pointer we have there only accepts bytes, we need to change to json.dumps() and then encode as utf-8. We have already done the same thing for the json (non-Chrome) format in 4b7eb862692e (py3: encode json output to bytes and use write(), 2018-10-12). Differential Revision: https://phab.mercurial-scm.org/D6781

statprof: use context manager for file when writing flame graph Differential Revision: https://phab.mercurial-scm.org/D6780

statprof: use context manager when reading source from file Differential Revision: https://phab.mercurial-scm.org/D6779

statprof: clarify by naming tuple members while enumerate()'ing Differential Revision: https://phab.mercurial-scm.org/D6778

upgrade: make sure we reclone all revlogs when updating to some format Adding or removing some requirement (eg: sparserevlog), requires to reclone revlog to use the new format. We cannot simply copy the original files. In this case, we issue a warning to proceed with clone every revlogs.

upgrade: add an argument to control changelog upgrade Same as for `--manifest` we can now select more selection.

upgrade: add an argument to control manifest upgrade The argument can be used to only "clone" manifest revlog or clone all of them but this one. The selection will make more sense once we have a `--changelog` flag in the next changesets.

unionrepo: drop the custom `rawdata` implementation We can rely on the main one now.

unionrepo: drop `baserevdiff` It has no caller anymore.

unionrepo: use normal inheritance scheme to call revdiff

unionrepo: fix `revdiff` implementation to use `rawdata` The parent code is using rawdata so we should use it here. Before this change, union repo was probably broken with some flag processors.

unionrepo: get rid of `baserevision` The method is not called anywhere anymore, so we can safely drop it. Some of the comment get moved to `baserevdiff` because we did not got rid of it (yet).

unionrepo: use a lower level overide in unionrepo too The unionrepo class also have a strange `baserevision` hack, let's try to get ride of it too.

bundlerepo: drop the custom `rawdata` implementation We can rely on the main one now.

bundlerepo: drop the `baserevision` hack It is not used anywhere anymore, so we can safely drop it.

bundlerepo: simplify code to take advantage of `_rawtext` In the revlog code, the code getting the raw text is now isolated. We take advantage of this to simplify the bundlerepo code.

merge with stable

rust: run cargo fmt

py3: use pycompat.maplist() in chgserver test-chg.t almost passes on py3 after this patch. Differential Revision: https://phab.mercurial-scm.org/D6771

run-tests: handle --local before --with-hg We no longer support them both together, so this is now safe to do. By checking --local first, we avoid error out about an invalid --with-hg script if --local was also given (we instead tell the user that the options are mutually exclusive). I also had to wrap the 'binpath' we pass to setattr() in _strpath() to keep `python3 run-tests.py -l` working. That change also made `python3 run-tests.py -l --chg` work, which was the reason for this series. Differential Revision: https://phab.mercurial-scm.org/D6760

run-tests: error out on `--local --with-[c]hg` I don't see much reason to allow these combinations. You could use --local and override only one of --with-hg or --with-chg, but I don't see much practical use for that. It would be easy to work around anyway by passing both --with-hg and --with-chg. By erroring out, it makes the code a bit easier to reason about to allow the next few patches. Differential Revision: https://phab.mercurial-scm.org/D6759

contrib: simplify the genosxversion.py command to find the hg libraries I forget what problem I ran into while trying to teach the makefile to use a non-system python. (It might have ben missing hg-evolve and/or keyring, but `check_output()` was raising an error.) This still isn't great because it will return non zero for something like the username not being set, even though we aren't asking for it. But I suppose it's still useful to simplify. Differential Revision: https://phab.mercurial-scm.org/D6753

interfaceutil: move to interfaces/ Now that we have a dedicated folder for interfaces, let's move interfaceutil there. Differential Revision: https://phab.mercurial-scm.org/D6742

interfaces: create a new folder for interfaces and move repository.py in it I was trying to understand current interfaces and write new ones and I realized we need to improve how current interfaces are organised. This creates a dedicated folder for defining interfaces and move `repository.py` which defines all the current interfaces inside it. Differential Revision: https://phab.mercurial-scm.org/D6741

narrow: fix typo "respositories" Differential Revision: https://phab.mercurial-scm.org/D6758

merge with stable