dirstate-tree: Add add_file, remove_file, and drop_file Again, various counters need to be kept up to date. Differential Revision: https://phab.mercurial-scm.org/D10491

dirstate-tree: Add has_dir and has_tracked_dir A node without a `DirstateMap` entry represents a directory. Only some values of `EntryState` represent tracked files. A directory is considered "tracked" if it contains any descendant file that is tracked. To avoid a sub-tree traversal in `has_tracked_dir` we add a counter for this. A boolean flag would become insufficent when we implement remove_file and drop_file. `add_file_node` is more general than needed here, in anticipation of adding the `add_file` and `remove_file` methods. Differential Revision: https://phab.mercurial-scm.org/D10490

dirstate-tree: Add clear_ambiguous_times in the new DirstateMap Also drive-by refactor it in the other DirstateMap Differential Revision: https://phab.mercurial-scm.org/D10489

dirstate-tree: Add copy_map_insert and copy_map_remove Differential Revision: https://phab.mercurial-scm.org/D10488

dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources This allows implementing __len__ for DirstateMap and CopyMap efficiently, without traversing the tree. Differential Revision: https://phab.mercurial-scm.org/D10487

dirstate-tree: Serialize to disk The existing `pack_dirstate` function relies on implementation details of `DirstateMap`, so extract some parts of it as separate functions for us in the tree-based `DirstateMap`. The `bytes-cast` crate is updated to a version that has an `as_bytes` method, not just `from_bytes`: https://docs.rs/bytes-cast/0.2.0/bytes_cast/trait.BytesCast.html#method.as_bytes Drive-by refactor `clear_ambiguous_times` which does part of the same thing. Differential Revision: https://phab.mercurial-scm.org/D10486

rust: Add a Timestamp struct instead of abusing Duration `SystemTime` would be the standard library type semantically appropriate instead of `Duration`. But since the value is coming from Python as a plain integer and used in dirstate packing code as an integer, let’s make a type that contains a single integer instead of using one with sub-second precision. Differential Revision: https://phab.mercurial-scm.org/D10485

dirstate-tree: Add tree traversal/iteration Like Python’s, Rust’s iterators are "external" in that they are driven by a caller who calls a `next` method. This is as opposed to "internal" iterators who drive themselves and call a callback for each item. Writing an internal iterator traversing a tree is easy with recursion, but internal iterators cannot rely on the call stack in that way, they must save in an explicit object all state that they need to be preserved across two `next` calls. This algorithm uses a `Vec` as a stack that contains what would be local variables on the call stack if we could use recursion. Differential Revision: https://phab.mercurial-scm.org/D10370

dirstate-tree: Add map `get` and `contains_key` methods Differential Revision: https://phab.mercurial-scm.org/D10369

dirstate-tree: Add parsing only dirstate parents from disk Differential Revision: https://phab.mercurial-scm.org/D10368

dirstate-tree: Implement DirstateMap::read This reads the on-disk dirstate in its current format (a flat sequence of entries) and creates a tree in memory. Differential Revision: https://phab.mercurial-scm.org/D10367

dirstate-tree: Add `WithBasename` wrapper for `HgPath` In the tree-shaped dirstate we want to have nodes representing files or directories, where directory nodes contain a map associating "base" names to child nodes for child files and directories. Many dirstate operations expect a full path from the repository root, but re-concatenating string from nested map keys all the time might be expensive. Instead, `WithBasename` stores a full path for these operations but behaves as its base name (last path component) for equality and comparison. Additionally `inclusive_ancestors` provides the successive map keys that are needed when inserting a new dirstate node at a given full path. Differential Revision: https://phab.mercurial-scm.org/D10365

dirstate-tree: Empty shell for a second Rust DirstateMap implementation For background see description of the previous changeset "Make Rust DirstateMap bindings go through a trait object". Add an empty shell for a opt-in second Rust implementation of the `DirstateMap` type and the `status` function. For now all methods panic. This can be seen in "action" with: ./hg status --config experimental.dirstate-tree.in-memory=1 Differential Revision: https://phab.mercurial-scm.org/D10364

dirstate-tree: Abstract "non-normal" and "other parent" sets Instead of exposing `HashSet`s directly, have slightly higher-level methods for the operations that Python bindings need on them. Differential Revision: https://phab.mercurial-scm.org/D10363

dirstate-tree: Make Rust DirstateMap bindings go through a trait object This changeset starts a series that adds an experiment to make status faster by changing the dirstate (first only in memory and later also on disk) to be shaped as a tree matching the directory structure, instead of the current flat collection of entries. The status algorithm can then traverse this tree dirstate at the same time as it traverses the filesystem. We (Octobus) have made prototypes that show promising results but are prone to bitrot. We would like to start upstreaming some experimental Rust code that goes in this direction, but to avoid disrupting users it should only be enabled by some run-time opt-in while keeping the existing dirstate structure and status algorithm as-is. The `DirstateMap` type and `status` function look like the appropriate boundary. This adds a new trait that abstracts everything Python bindings need and makes those bindings go through a `dyn` trait object. Later we’ll have two implementations of this trait, and the same bindings can use either. Differential Revision: https://phab.mercurial-scm.org/D10362

remotefilelog: use the correct capability when using getfilestype threaded The functon was overlooked when the capability was renamed Differential Revision: https://phab.mercurial-scm.org/D10673

test-copies: test that copies' sidedata can get computed during push If the source of the push does not have the necessary sidedata but the server needs them, the client should compute them on the fly while pushing. Differential Revision: https://phab.mercurial-scm.org/D10350

test-copies: test that copies' sidedata can get computed during pull If the source does not have the data, the pulling client should compute the necessary side-data while pulling. Differential Revision: https://phab.mercurial-scm.org/D10349

test-copies: test that copies' sidedata does not get corrupted during push This is an important usecase. Differential Revision: https://phab.mercurial-scm.org/D10348

test-copies: test that copies' sidedata does not get corrupted during pull This is an important usecase. Differential Revision: https://phab.mercurial-scm.org/D10347

test-copies: simplify some conditional output Now that all computation using sidedata give the same result, we can simplify conditional Differential Revision: https://phab.mercurial-scm.org/D10346

sidedata: move documentation about sidedata helpers to sidedata module Differential Revision: https://phab.mercurial-scm.org/D10361

sidedata: move sidedata-related utils to the dedicated module Differential Revision: https://phab.mercurial-scm.org/D10360

sidedata: replace sidedata upgrade mechanism with the new one Note: this is split into a separate change (like some other patches in this series) because it's not easy to have all patches work 100% and this seemed easier for reviewers. When cloning or upgrading a repo, we may need to compute (or remove) sidedata. This is the same mechanism that is used in exchange, so we re-use the new system to simplify the code and fix the remaining issues (correctly dropping flags and handling partial removal, etc.). This also highlighted an issue with `test-copies-in-changeset.t` that kept sidedata categories that are not relevant anymore. They should probably be dropped entirely, but that would be for another patch. Differential Revision: https://phab.mercurial-scm.org/D10359

sidedata: add a way of replacing an existing sidedata computer This will be useful in a future patch to replace a sequential computer with a parallel computer. We only allow for explicit replacement, to force the users to think about overriding computers. Differential Revision: https://phab.mercurial-scm.org/D10358

bundle2: remove restriction around sidedata We are now capable of generating the missing sidedata on-the-fly. Differential Revision: https://phab.mercurial-scm.org/D10345

heptapod-ci: enable pytype checking The trigger is manual for now, until we get a better idea about stability, resource usage, etc. Differential Revision: https://phab.mercurial-scm.org/D10462

hghave: fix the definition of `python3` to work on Windows Both py2 and py3 executables are named `python.exe`, and may or may not be on PATH. So use the dispatcher executable that comes with py3 to fetch the version of the latest py3 executable. This allows at least one relnotes test to run on Windows. Differential Revision: https://phab.mercurial-scm.org/D10694

util: avoid echoing the password to the console on Windows py3 (issue6446) The `getpass.getpass()` implementation on Windows first checks if `sys.stdin` and `sys.__stdin__` are the same object. It's not on py3 because the former is replaced in dispatch.py with something that doesn't normalize '\n' to '\r\n'. When they aren't the same object, it simply calls `sys.stdin.readline()` instead of the mscvrt functions that read the input characters before they are echoed. This simply copies the `getpass.win_getpass()` implementation without the stdin check, and byteifies around the edges. I'm not sure if there's a reasonable replacement for the check that we could implement. When echoing input into the hg command, the `ui.interactive()` check causes `ui.getpass()` to bail before getting here. If the proper config switches are used to bypass that and call this, the process stalls until '\n' is input into the console. So there could be a deadlock here when run by another command if the wrong config settings are applied. Differential Revision: https://phab.mercurial-scm.org/D10708

sidedata: enable sidedata computers to optionally rewrite flags Sidedata computers may want to influence the flags of the revision they touch. For example, the computer for changelog-based copytracing can add a flag to signify that this revision might affect copytracing, inversely removing said flag if the information is no longer applicable. See inline documentation in `storageutil` for more details. Differential Revision: https://phab.mercurial-scm.org/D10344