verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10809

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10808

verify: use some intermediate variables instead of a multi-liner This is shorter and easier to read as the indentation remains the same. Differential Revision: https://phab.mercurial-scm.org/D10807

verify: expand a one liner into explicit commands The result is not longer, but it is more edible. Differential Revision: https://phab.mercurial-scm.org/D10806

verify: align a comment line This argument description is back with it comrade. Differential Revision: https://phab.mercurial-scm.org/D10805

dirstate-v2: Drop parent directory cache when removing a dirstate node The premise of the directory cache is that the dirstate contains child nodes for every entry that `read_dir` would return. When removing nodes, that may not be the case anymore so the cache should be invalidated. Differential Revision: https://phab.mercurial-scm.org/D10829

dirstate-v2: Add --dirs to debugdirstate command `hg debugdirstate --dirs` also shows information stored in the dirstate (for `read_dir` caching) about directories. Differential Revision: https://phab.mercurial-scm.org/D10828

dirstate-v2: Write .hg/dirstate back to disk on directory cache changes Differential Revision: https://phab.mercurial-scm.org/D10827

dirstate-v2: Skip readdir in status based on directory mtime When calling `read_dir` during `status` and the directory is found to be eligible for caching (see code comments), write the directory’s mtime to the dirstate. The presence of a directory mtime in the dirstate is meaningful and indicates eligibility. When an eligible directory mtime is found in the dirstate and `stat()` shows that the mtime has not changed, `status` can skip calling `read_dir` again and instead rely on the names of child nodes in the dirstate tree. The `tempfile` crate is used to create a temporary file in order to use its modification time as "current time" with the same truncation as other files and directories would have in their own modification time. Differential Revision: https://phab.mercurial-scm.org/D10826

dirstate-v2: Allow tree nodes without an entry to store a timestamp Timestamps are stored on 96 bits: * 64 bits for the signed number of seconds since the Unix epoch * 32 bits for the nanoseconds in the `0 <= ns < 1_000_000_000` range For now timestamps are not used or set yet. Differential Revision: https://phab.mercurial-scm.org/D10825

dirstate-tree: Change status() results to not borrow DirstateMap The `status` function takes a `&'tree mut DirstateMap<'on_disk>` parameter. `'on_disk` borrows a read-only byte buffer with the contents of the `.hg/dirstate` file. `DirstateMap` internally uses represents file paths as `std::borrow::Cow<'on_disk, HgPath>`, which borrows the byte buffer when possible and allocates an owned string if not, such as for files added to the dirstate after it was loaded from disk. Previously the return type of of `status` has a `'tree` lifetime, meaning it could borrow all paths from the `DirstateMap`. With this changeset, that lifetime is changed to `'on_disk` meaning that only paths from the byte buffer can be borrowed, and paths allocated by `DirstateMap` must be copied. Usually most paths are in the byte buffer, and most paths are not part of the return value of `status`, so the number of extra copies should be small. This change will enable `status` to mutate the `DirstateMap` after it has finished constructing its return value. Previously such mutation would be prevented by possible on-going borrows. Differential Revision: https://phab.mercurial-scm.org/D10824

dirstate-tree: Fix status algorithm with unreadable directory When reading a directory fails such as because of insufficient permissions, it should be treated as empty by status instead of skipped entirely. Differential Revision: https://phab.mercurial-scm.org/D10823

docket: make compatible with py3.6, where Struct.format is bytes Differential Revision: https://phab.mercurial-scm.org/D10770

packaging: disable rust extensions again on CentOS Backed out changeset eccbfa7e19c0 We're seeing (very rarely) crashes of 'hg purge' on some of our machines (see https://bz.mercurial-scm.org/show_bug.cgi?id=6509 ). Unfortunately, I haven't been able to find out much more about what is going wrong. To avoid further impact on our users and CI, I would prefer to disable the rust extensions for now. Differential Revision: https://phab.mercurial-scm.org/D10877

cext: fix memory leak in phases computation Without this a buffer whose size in bytes is the number of changesets in the repository is leaked each time the repository is opened and changeset phases are computed. Impact: the current code in hgwebdir creates a new `localrepository` instance for each HTTP request. Since any pull or push is made of several requests, a team of 100 people can easily produce thousands of such requests per day. Being a low-level malloc, this leak can't be seen with the gc module and tools relying on that, but was spotted by valgrind immediately. Reproduction ------------ for i in range(cl_args.iterations): repo = hg.repository(baseui, repo_path) rev = repo.revs(rev).first() ctx = repo[rev] del ctx del repo # avoid any pollution by other type of leak # (that should be fixed in 5.8) repoview._filteredrepotypes.clear() gc.collect() Measurements ------------ Resident Set Size (RSS), taken on a clone of mozilla-central for performance analysis (440 000 changesets). before: 5.8+hg19.5ac0f2a8ba72 1000 iterations: 1606MB 5.8+hg19.5ac0f2a8ba72 10000 iterations: 5723MB after: 5.8+hg20.e2084d39e145 1000 iterations: 555MB 5.8+hg20.e2084d39e145 10000 iterations: 555MB (double checked, not a copy/paste error) (e2084d39e14 is the present changeset, before amendment of the message to add the measurements)

revlogv2: make sure bundling pick a compatible bundle format Before this change, revlog-v2 repository where bundled using the incompatible "v1" format. Differential Revision: https://phab.mercurial-scm.org/D10802

censor: do not process sidedata of censored revision while bundling The revision is censored, we should ignore it. Differential Revision: https://phab.mercurial-scm.org/D10801

changegroup: fix deltachunk API to be consistent from one class to another Depending of the subclass the 8th index of `chunkdata` items was either a sidedata dict of a proto_flags integer. We have not fixed the inconsistency and we already return fixed "delta" items from `deltaiter`. Differential Revision: https://phab.mercurial-scm.org/D10778

fuzz: add hg to sys.path when constructing mpatch corpus Differential Revision: https://phab.mercurial-scm.org/D10777

dirstate-tree: Skip readdir() in `hg status -mard` When running the status algorithm in a mode where we don’t list unknown or ignored files, all we care about are files that are listed in the dirstate. We can there for skip making expensive calls to readdir() to list the contents of filesystem directories, and instead only run stat() to get the filesystem state of files listed in the dirstate. (This state may be an error for files that don’t exist anymore on the filesystem.) On 16 CPU threads, this reduces the time spent in the `status()` function for `hg status -mard` on an old snapshot of mozilla-central from ~70ms to ~50ms. Differential Revision: https://phab.mercurial-scm.org/D10752

dirstate-v2: Parse the dirstate lazily, with copy-on-write nodes TODO: more description Differential Revision: https://phab.mercurial-scm.org/D10751

dirstate-v2: Make the dirstate bytes buffer available in more places Differential Revision: https://phab.mercurial-scm.org/D10750

dirstate-v2: Make more APIs fallible, returning Result When parsing becomes lazy, parse error will potentially happen in more places. This propagates such errors to callers. Differential Revision: https://phab.mercurial-scm.org/D10749

dirstate-v2: Add a zero-size error type for dirstate v2 parse errors This error should only happen if Mercurial is buggy or the file is corrupted. It indicates for example that: * A part of the file refers to another part, and the byte offset or item count would cause reading out of bounds, beyond the end of the file. * The byte for an entry state has an invalid value When parsing becomes lazy, many more functions will return a `Result` with this error. Making it zero-size reduces the work that the `?` operator needs to do to pass around the error value. Differential Revision: https://phab.mercurial-scm.org/D10748

dirstate-tree: Add `NodeRef` and `ChildNodesRef` enums They are used instead of `&Node` and `&ChildNodes` respectively. The `ChildNodes` type alias also becomes a similar enum. For now they only have one variant each, to be extended later. Adding enums now forces various use sites go through new methods instead of manipulating the underlying data structure directly. Differential Revision: https://phab.mercurial-scm.org/D10747

rust: Return owned instead of borrowed DirstateEntry in DirstateMap APIs This will enable the tree-based DirstateMap to not always have an actual DirstateEntry in memory for all nodes, but construct it on demand. Differential Revision: https://phab.mercurial-scm.org/D10746

dirstate-tree: Downgrade `&mut Node` to `&Node` in status and serialization Mutable access is not used, and upcoming changes will make it more costly (with copy-on-write nodes that can be read from disk representation) Differential Revision: https://phab.mercurial-scm.org/D10745

dirstate-tree: Remove DirstateMap::iter_node_data_mut In an upcoming changeset we want DirstateMap to be able to work directly with nodes in their "on disk" representation, without always allocating corresponding in-memory data structures. Nodes would have two possible representations: one immutable "on disk" refering to the bytes buffer of the contents of the .hg/dirstate file, and one mutable with HashMap like the curren data structure. These nodes would have copy-on-write semantics: when an immutable node would need to be mutated, instead we allocate new mutable node for it and its ancestors. A mutable iterator of the entire tree would still be possible, but it would become much more expensive since we’d need to allocate mutable nodes for everything. Instead, remove this iterator. It was only used to clear ambiguous mtimes while serializing the `DirstateMap`. Instead clearing and serialization are now two separate passes. Clearing first uses an immutable iterator to collect the paths of nodes that need to be cleared, then accesses only those nodes mutably. Differential Revision: https://phab.mercurial-scm.org/D10744

merge with stable

revlog: close the index file handle after the data one This make sure the data file is flushed before the index. preventing the index to reference unflushed data. Differential Revision: https://phab.mercurial-scm.org/D10776