run-tests: refactor filtering logic for --retest flag How I got to this: While re-running failed tests using --retest I noticed that the output: "running x tests using y parallel processes". was not actually correct, because x was the total number of tests present in the directory, but it should be the number of failed tests. Although it would run only the failed tests and later will say that remaining tests were skipped. Changes in test files reflect the fixed behaviour. This patch change and move the logic for filtering failed test for --retest option and make sure that we create instances of class Test only for the tests we need to run. As mentioned in the deleted text (in this patch itself) the logic for --retest should be outside of TestSuite. Differential Revision: https://phab.mercurial-scm.org/D8938

tests: add a --retest test to demonstrate a fix in next patch As you see above the added lines, there was only one test failed. So the output should be: "running 1 tests using 1 parallel processes" Next patch will be fixing this. Differential Revision: https://phab.mercurial-scm.org/D8964

tests: remove a sub-test since we are done with it in test-run-tests.t removing test-failure.t to make sure that in next patch when we re-run failed tests using --retest, it won't get considered unnecessarily. Differential Revision: https://phab.mercurial-scm.org/D8963

storageutil: allow modern hash sizes for fileids Differential Revision: https://phab.mercurial-scm.org/D8783

registrar: fix a documentation typo Differential Revision: https://phab.mercurial-scm.org/D8976

documentation: add `zstd` compression to the internal `revlogs` documentation Differential Revision: https://phab.mercurial-scm.org/D8953

extdiff: move single file handling inside `not per-file` conditional If `--per-file` is passed, we will diffing files one by one, hence there is no need to make this optimization around single file in that case. This optimization is only required when `--per-file` is not passed and we will be opening directory-diff for a single file. Differential Revision: https://phab.mercurial-scm.org/D8971

extdiff: remove dir2root and pass full path as dir2 in _runperfilediff() The only use of `dir2root` was to join with `dir2` to generate the path for other side of diff. Like in previous patch, `dir1a` and `dir1b` are full paths and no longer base names, hence we pass `dir2` as full path too and making `dir2root` unrequired. Differential Revision: https://phab.mercurial-scm.org/D8970

extdiff: pass full paths of `dir1a` and `dir1b` to `_runperfilediff()` Earlier we were passing basename instead of the fullpath and then joining with the tmproot. This is wrong because the user can choose `rev1a` as wdir and in those cases, tmproot should not be joined with the path. I am working on refactoring extdiff logic so that we can have options like `diff --tool` and encountered this. Although this patch in itself makes no difference, however in future when a new caller of `_runperfilediff()` will be added, it will be useful to directly pass on the full paths instead. Differential Revision: https://phab.mercurial-scm.org/D8969

merge with stable

mercurial: force LF endings for *.py, *.{c,h} and *.t in .editorconfig PyCharm on Windows otherwise uses CRLF. Differential Revision: https://phab.mercurial-scm.org/D8947

hooklib: update documentation of changeset_obsoletedfor for changed hook type This updates usage example of changeset_obsoleted to reflect the move from pretxnclose to txnclose made in 04ef381000a8 (hooklib: fix detection of successors for changeset_obsoleted). Differential Revision: https://phab.mercurial-scm.org/D8929

worker: don't expose readinto() on _blockingreader since pickle is picky The `pickle` module expects the input to be buffered and a whole object to be available when `pickle.load()` is called, which is not necessarily true when we send data from workers back to the parent process (i.e., it seems like a bad assumption for the `pickle` module to make). We added a workaround for that in https://phab.mercurial-scm.org/D8076, which made `read()` continue until all the requested bytes have been read. As we found out at work after a lot of investigation (I've spent the last two days on this), the native version of `pickle.load()` has started calling `readinto()` on the input since Python 3.8. That started being called in https://github.com/python/cpython/commit/91f4380cedbae32b49adbea2518014a5624c6523 (and only by the C version of `pickle.load()`)). Before that, it was only `read()` and `readline()` that were called. The problem with that was that `readinto()` on our `_blockingreader` was simply delegating to the underlying, *unbuffered* object. The symptom we saw was that `hg fix` started failing sometimes on Python 3.8 on Mac. It failed very relyable in some cases. I still haven't figured out under what circumstances it fails and I've been unable to reproduce it in test cases (I've tried writing larger amounts of data, using different numbers of workers, and making the formatters sleep). I have, however, been able to reproduce it 3-4 times on Linux, but then it stopped reproducing on the following few hundred attempts. To fix the problem, we can simply remove the implementation of `readinto()`, since the unpickler will then fall back to calling `read()`. The fallback was added a bit later, in https://github.com/python/cpython/commit/b19f7ecfa3adc6ba1544225317b9473649815b38. However, that commit also added checking that what `read()` returns is a `bytes`, so we also need to convert the `bytearray` we use into that. I was able to add a test for that failure at least. Differential Revision: https://phab.mercurial-scm.org/D8928

commit: clear mergestate also with --amend (issue6304) The `hg commit --amend` uses the in-memory code, which naturally doesn't touch the merge state (well, it shouldn't anyway; I think I've fixed bugs in that area recently). We therefore need to clear the mergestate after calling `repo.commitctx()` since we expect that from `hg commit --amend`. Differential Revision: https://phab.mercurial-scm.org/D8932

tests: add test showing that merge state is not cleared by amend This is slightly modified version of the test case I provided in issue6304. Differential Revision: https://phab.mercurial-scm.org/D8931

requirements: introduce constants for `shared` and `relshared` requirements We add them to `WORKING_DIR_REQUIREMENTS` too as they should be stored in `.hg/requires` and have information about the type of working copy. Differential Revision: https://phab.mercurial-scm.org/D8926

mergestate: replace `addmergedother()` with generic `addcommitinfo()` (API) Storing that a file is resolved for the other parent while merging is just one case of things we will like to store in the mergestate. There are more which we will like to store. This patch replaces `addmergedother()` with a much more generic `addcommitinfo()`. Doing this, we also blinding stores the same key value pair generated by the merge code instead of touching them. Differential Revision: https://phab.mercurial-scm.org/D8923

merge: introduce `addcommitinfo()` on mergeresult object This makes code little bit nicer as we directly update information in the mergeresult object instead of building up a dict first and then setting it. Differential Revision: https://phab.mercurial-scm.org/D8922

merge: use collections.defaultdict() for mergeresult.commitinfo We will be storing info from mergeresult.commitinfo to mergestate._stateextras in upcoming patches, let's make them use same structure so that we don't have to make much efforts in transferring info from one to other. Differential Revision: https://phab.mercurial-scm.org/D8921

mergestate: use _stateextras instead of merge records for commit related info There is a set of information related to a merge which is needed on commit. We want to store such information in the mergestate so that we can read it while committing. For this purpose, we are using merge records and introduced a merge entry state for that. However this won't scale and is not clean way to implement this. This patch reworks the existing logic related to this to use _stateextras and read from it. Right now the information stored is not very descriptive but it will be in next patch. Using _stateextras also makes MERGE_RECORD_MERGED_OTHER useless and only to be kept for BC. Differential Revision: https://phab.mercurial-scm.org/D8920

mergestate: use collections.defaultdict(dict) for _stateextras I want to use this _stateextras more in upcoming patches to store some commit time related information. Using defaultdict will help in cleaner code around checking whether a file exists or not. Differential Revision: https://phab.mercurial-scm.org/D8919

hgweb: minimize scope of a try-block in staticfile() I think the exceptions are only relevant for the `os.stat()` and `open()` calls, and maybe to the `fh.read()` call. Differential Revision: https://phab.mercurial-scm.org/D8936

hgweb: ignore web.templates config when guessing mime type for static content Frozen binaries won't have a file-system path for static content, so I'd like to remove dependence on that. From the documentation, it seems like `mimetypes.guess_type()` only cares about the suffix, so I think it should be enough to pass in just path under the `web.templates` directory. Differential Revision: https://phab.mercurial-scm.org/D8935

hgweb: let staticfile() look up path from default location unless provided This reduces duplication between the two callers. Differential Revision: https://phab.mercurial-scm.org/D8934

hgweb: handle None from templatedir() equally bad in webcommands.py The following paragraph is based just on my reading of the code; I have not tried to test it. Before my recent work on templates in frozen binaries, it seems both `hgwebdir_mod.py` and `webcommands.py` would pass in an empty list into `staticfile()` when running in a frozen binary. That would then result in a variable in that function (`path`) not getting bound before its first use. I then changed that without thinking in D8786 so we passed a `None` value into the function, which made it break in another way (trying to iterate over `None`). Then I tried to fix it up in D8810, but I only changed `hgwebdir_mod.py` for some reason, and it still doesn't actually work in frozen binaries (which seems fair, since was broken before my changes too). This patch just replicates the half-assed "fix" from D8810 in `webcommands.py`, so they look more similar so I can start refactoring them in the same way. Differential Revision: https://phab.mercurial-scm.org/D8933

posixworker: avoid creating workers that end up getting no work If `workers` (the detected or configured number of CPUs) is greater than the number of work items, then some of the workers end up getting 0 work items. Let's not create such workers. Differential Revision: https://phab.mercurial-scm.org/D8927

revert: remove dangerous `parents` argument from `cmdutil.revert()` As we found out the hard way (thanks to spectral@ for figuring it out!), `cmdutil.revert()`'s `parents` argument must be `repo.dirstate.parents()` or things may go wrong. We had an extension that passed in the target commit as the first parent. The `hg split` command from the evolve extension seems to have made the same mistake, but I haven't looked carefully. The problem is that `cmdutil._performrevert()` calls `dirstate.normal()` on reverted files if the commit to revert to equals the first parent. So if you pass in `ctx=foo` and `parents=(foo.node(), nullid)`, then `dirstate.normal()` will be called for the revert files, even though they might not be clean in the working copy. There doesn't seem to be any reason, other than a tiny performance benefit, to passing the `parents` around instead of looking them up again in `cmdutil._performrevert()`, so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D8925

requirements: introduce a set of working directory specific requirements Some requirements like the SPARSE_REQUIREMENT is working directory specific and cannot be shared. We add a set which will contain all these requirements. This is not the best we can do, I think having a rich requirement class will be much better but that will be out of scope for this series. Differential Revision: https://phab.mercurial-scm.org/D8924

localrepo: move requirements constant to requirements module We now have a dedicated module for requirements constant, let's move the ones in localrepo there. Differential Revision: https://phab.mercurial-scm.org/D8918

requirements: introduce new requirements related module It was not clear where all requirements should and related APIs should be, this patch introduces a requirements module which will have all exitsing requirements and related APIs. Differential Revision: https://phab.mercurial-scm.org/D8917

repository: introduce constant for treemanifest requirement and use it In future we will like to much cleaner logic around which requirement is for working copy and which can go in store. To start with that, we first need to de-clutter the requirement values spread around and replace them with constants. Differential Revision: https://phab.mercurial-scm.org/D8916

tests: make flag parsing test more future safe The revlog format contains a number of feature flags, e.g. if general deltas are active. When testing that unknown flags are rejected, use bits that are not immediately following the currently used bits. Differential Revision: https://phab.mercurial-scm.org/D8782

rename: add support for --at-rev, which marks as copy and removes the source I had previously only added support for `--at-rev` to `hg cp`, but not to `hg mv`. This patch adds that support. Just like for `hg cp`, it marks the destination as copied from the source, and doesn't care if the source file still exists (because it only supports the `-A` mode, aka "don't touch files" mode). It works whether or not the source file still exists. This matches the behavior of `hg mv -A` in the working copy. Differential Revision: https://phab.mercurial-scm.org/D8840

tests: make test-install.t work on debian systems Debian systems, at least as of their version of python3.8 on my machine, have rewritten some logic in ensurepip to make it not use the wheels in pip._bundled, but instead to use wheels installed in /usr/share/python-wheels. It copies these wheels into the virtual environment when it's created, and installenv/bin/pip is able to see them and use them, so it thinks that 'wheel' is installed, and that it can build the mercurial wheel instead of just installing it. For some reason, when it subprocesses to run `python3 setup.py bdist_wheel`, it setup.py does *not* have the 'wheel' wheel available, and we get an error message. Differential Revision: https://phab.mercurial-scm.org/D8813

rhg: handle broken pipe error for stderr Differential Revision: https://phab.mercurial-scm.org/D8871

rhg: extract function handle_stdout_error Avoid repeating the logic of handling stdout write errors. Differential Revision: https://phab.mercurial-scm.org/D8870

rhg: add a limited `rhg files` subcommand Differential Revision: https://phab.mercurial-scm.org/D8869

rhg: add a `Files` `Command` to prepare the `rhg files` subcommand Differential Revision: https://phab.mercurial-scm.org/D8868

rhg: simplify `FindRootError` handling Differential Revision: https://phab.mercurial-scm.org/D8867

rhg: add buffered stdout writing possibility Improve batch stdout writing performance. At some point line buffered output should be introduced. Differential Revision: https://phab.mercurial-scm.org/D8866

rhg: ask the error message from `CommandError` Avoid repeating the display of the same error messages in different commands. Differential Revision: https://phab.mercurial-scm.org/D8865

rhg: Do not return error when when we really mean ok in commands Before when a command was successfull `Err(CommandErrorKind::Ok.into())` was returned which is an oxymoron. Using `Ok(())` when everything is ok seems more appropriate. Differential Revision: https://phab.mercurial-scm.org/D8864

hg-core: define a `ListTrackedFiles` `Operation` List files under Mercurial control in the working directory. Differential Revision: https://phab.mercurial-scm.org/D8863

hg-core: remove the `Operation` trait There is no way to currently define a trait which can both return references to `self` and to passed data, which is what we would need. Generic Associated Types may fix this and allow us to have a unified interface. See: rust #44265 Differential Revision: https://phab.mercurial-scm.org/D8862

hg-core: make parse_dirstate return references rather than update hashmaps Returing a vec is faster than updating a hashmap when the hashmap is not needed like in `hg files` which just list tracked files. Returning references avoid copying data when not needed improving performence for large repositories. Differential Revision: https://phab.mercurial-scm.org/D8861

repository: introduce constant for internal phase repo requirement and use it In future we will like to much cleaner logic around which requirement is for working copy and which can go in store. To start with that, we first need to de-clutter the requirement values spread around and replace them with constants. Differential Revision: https://phab.mercurial-scm.org/D8912

repository: introduce constant for sparse repo requirement and use it In future we will like to much cleaner logic around which requirement is for working copy and which can go in store. To start with that, we first need to de-clutter the requirement values spread around and replace them with constants. Differential Revision: https://phab.mercurial-scm.org/D8911

localrepo: refactor `.hg/requires` reading logic in separate function In an upcoming patch, we will be reusing this to read `.hg/store/requires`, so let's separate it in a different function before. Differential Revision: https://phab.mercurial-scm.org/D8910

localrepo: refactor logic to calculate sharedvfs in separate fn We will be reusing this in an upcoming patch, so better to refactor it into a separate function. Also the underlying handling deserves a function of it's own. Differential Revision: https://phab.mercurial-scm.org/D8909

store: refactor space delimited list to proper data structure There is no good reason why are having a space delimited list and then using `.split()` to get the actual list. Let's convert this into a proper collection. Differential Revision: https://phab.mercurial-scm.org/D8908

makefile: stop setting unused HGEXTDIR variable during osx build The variable was added in a38ed42cd23c (osx: include chg by default, 2017-03-20), but I can't find any others references to the variable in that commit or in any other commits. Differential Revision: https://phab.mercurial-scm.org/D8915

merge: drop commitinfo argument to applyupdates (API) We now pass the mergeresult object and hence there is no need to have a separate commitinfo argument as the required info is present in mergeresult object. Differential Revision: https://phab.mercurial-scm.org/D8904

merge: remove emptyactions() and use collections.defaultdict(list) instead emptyactions() used to return a dict which was populated and passed into applyupdates(). However, with recent changes, we no longer pass a plain dict, instead we pass the mergeresult object. There was only one usage of emptyactions and that too inside mergeresult object. That usage is replaced with collections.defaultdict(list) instead. Not sure why we were not using collections.defaultdict(list) from the beginning. Differential Revision: https://phab.mercurial-scm.org/D8903

merge: pass mergeresult obj in _forgetremoved() (API) Instead of returning a dict of actions and then updating it, let's pass the object directly and update it there. This makes `updateactions()` on mergeresult unused and this patch removes that. After this patch, we have couple of methods left on mergeresult class which still exposes the internal dict based action storage. Differential Revision: https://phab.mercurial-scm.org/D8889

mergeresult: introduce filemap() which yields filename based mapping We wanted to remove `actions` as this was leaking how we store things internally and was direct access to one of the member. This introduces filemap() which yields a map of `filename` -> `action, args, msg`. `mergeresult.actions` has been deleted as it's no longer required. Differential Revision: https://phab.mercurial-scm.org/D8888

mergeresult: add `files()` and use it `files()` will return a list of files on which an action needs to be performed. This is a step to stop exposing the underlying map to the user of this object. Differential Revision: https://phab.mercurial-scm.org/D8887

mergeresult: introduce getfile() and use it where required We want to hide the underlying dictionary from the users and provide API for valid and sane use cases. Differential Revision: https://phab.mercurial-scm.org/D8886

merge: use ACTION_* constants instead of values in _filternarrowactions() It makes easier to check what noconflicttypes are and which are not included. I hope we can have a state where we always use ACTION_* constants instead of these values which are very hard to understand. Differential Revision: https://phab.mercurial-scm.org/D8885

merge: rework iteration over mergeresult object in checkpathconflicts() Instead of following pattern: ``` for f, (m, args, msg) in mresult.actions.items(): if m == mergestatemod.ACTION_*: ... elif m == mergestatemod.ACTION_*: ... .... ``` We do: ``` for (f, args, msg) in mresult.getaction((mergestatemod.ACTION_*,)): ... for (f, args, msg) in mresult.getaction((mergestatemod.ACTION_*,)): ... .... ``` This makes code bit easier to understand and prevent iterating over actions which we don't need. Differential Revision: https://phab.mercurial-scm.org/D8884

applyupdates: simplfy calculation of number of updated files Instead of increasing the `updated` variable each time in a loop, let's use the mergeresult object API to calculate the updated value in one call. Differential Revision: https://phab.mercurial-scm.org/D8883