Simon Sapin <simon.sapin@octobus.net> [Fri, 07 May 2021 16:41:07 +0200] rev 47144
dirstate-tree: Add a test showing that issue 6335 is fixed
… when using the new status algorithm and the tree-based dirstate.
The previous algorithm still has this bug.
Differential Revision: https://phab.mercurial-scm.org/D10699
Simon Sapin <simon.sapin@octobus.net> [Mon, 03 May 2021 20:04:19 +0200] rev 47143
dirstate-tree: Add a dirstate-v1-tree variant of some tests
The `dirstate-v1` variant has the previous behavior.
`dirstate-v1-tree` uses the same format on disk, but uses the new
`DirstateMap` with a tree data structure and the new `status` algorithm.
These were untested so far.
Differential Revision: https://phab.mercurial-scm.org/D10698
Matt Harbison <matt_harbison@yahoo.com> [Fri, 07 May 2021 22:06:25 -0400] rev 47142
merge with stable
Martin von Zweigbergk <martinvonz@google.com> [Fri, 07 May 2021 08:38:17 -0700] rev 47141
rename: add hint about --at-rev if source file doesn't exist
It's quite common that users want to record copy (rename) information
after committing the working copy changes (i.e. an added and a deleted
file). When they try `hg mv [--after] <src> <dst>`, that just fails
because the source file doesn't exist. It seems helpful if we can
point them to `--at-rev=.` in this case.
Differential Revision: https://phab.mercurial-scm.org/D10697
Simon Sapin <simon.sapin@octobus.net> [Fri, 30 Apr 2021 20:21:56 +0200] rev 47140
dirstate-tree: Borrow paths from the "on disk" bytes
Use std::borrow::Cow to avoid some memory allocations and copying.
Differential Revision: https://phab.mercurial-scm.org/D10560
Simon Sapin <simon.sapin@octobus.net> [Fri, 30 Apr 2021 19:33:04 +0200] rev 47139
dirstate-tree: Borrow copy source paths from the "on disk" bytes
Use std::borrow::Cow to avoid some memory allocations and copying.
These particular allocations are not visible when profiling (as many files
in a typical repo don’t have a copy source). This change is "warm up"
for doing the same with paths of files themselves, which is more involved
since those paths are used as `HashMap` keys. This gets of the way the
addition of a lifetime parameter to several types.
Differential Revision: https://phab.mercurial-scm.org/D10559
Simon Sapin <simon.sapin@octobus.net> [Fri, 30 Apr 2021 19:57:46 +0200] rev 47138
rust: Use `&HgPath` instead of `&HgPathBuf` in may APIs
Getting the former (through `Deref`) is almost the only useful thing one can
do with the latter anyway. With this changes, API become more flexible for the
"provider" of these paths which may store something else that Deref’s to HgPath,
such as `std::borrow::Cow<HgPath>`. Using `Cow` can help reduce memory alloactions
and copying.
Differential Revision: https://phab.mercurial-scm.org/D10558
Simon Sapin <simon.sapin@octobus.net> [Fri, 30 Apr 2021 18:24:54 +0200] rev 47137
dirstate-tree: Make `DirstateMap` borrow from a bytes buffer
… that has the contents of the `.hg/dirstate` file.
This only applies to the tree-based flavor of `DirstateMap`.
For now only the entire `&[u8]` slice is stored, so this is not useful yet.
Adding a lifetime parameter to the `DirstateMap` struct (in hg-core) makes
Python bindings non-trivial because we keep that struct in a Python object
that has a dynamic lifetime tied to Python’s reference-counting and GC.
As long as we keep the `PyBytes` that owns the borrowed bytes buffer next to
the borrowing struct, the buffer will live long enough for the borrows to stay
valid. However this relationship cannot be expressed in safe Rust code in a
way that would statisfy they borrow-checker. We use `unsafe` code to erase
that lifetime parameter, and encapsulate it in a safe abstraction similar to
the owning-ref crate: https://docs.rs/owning_ref/
Differential Revision: https://phab.mercurial-scm.org/D10557
Simon Sapin <simon.sapin@octobus.net> [Fri, 30 Apr 2021 18:13:31 +0200] rev 47136
rust: Read dirstate from disk in DirstateMap constructor
Before this changeset, Python code first creates an empty `DirstateMap` Rust
object, then immediately calls its `read` method with a byte string of the
contents of the `.hg/dirstate` file.
This makes that byte string available to the constructor of `DirstateMap`
in the hg-cpython crate. This is a first step towards enabling parts of
`DirstateMap` in the hg-core crate to borrow from this buffer without copying.
Differential Revision: https://phab.mercurial-scm.org/D10556
Simon Sapin <simon.sapin@octobus.net> [Fri, 30 Apr 2021 15:40:11 +0200] rev 47135
rust: Remove handling of `parents` in `DirstateMap`
The Python wrapper class `dirstatemap` can take care of it.
This removes the need to have both `_rustmap` and `_inner_rustmap`.
Differential Revision: https://phab.mercurial-scm.org/D10555
Simon Sapin <simon.sapin@octobus.net> [Fri, 30 Apr 2021 14:22:14 +0200] rev 47134
dirstate-tree: Fold "tracked descendants" counter update in main walk
For the purpose of implementing `has_tracked_dir` (which means "has tracked
descendants) without an expensive sub-tree traversal, we maintaing a counter
of tracked descendants on each "directory" node of the tree-shaped dirstate.
Before this changeset, mutating or inserting a node at a given path would
involve:
* Walking the tree from root through ancestors to find the node or the spot
where to insert it
* Looking at the previous node if any to decide what counter update is needed
* Performing any node mutation
* Walking the tree *again* to update counters in ancestor nodes
When profiling `hg status` on a large repo, this second walk takes times
while loading a the dirstate from disk.
It turns out we have enough information to decide before he first tree walk
what counter update is needed. This changeset merges the two walks, gaining
~10% of the total time for `hg update` (in the same hyperfine benchmark as
the previous changeset).
---
Profiling was done by compiling with this `.cargo/config`:
[profile.release]
debug = true
then running with:
py-spy record -r 500 -n -o /tmp/hg.json --format speedscope -- \
./hg status -R $REPO --config experimental.dirstate-tree.in-memory=1
then visualizing the recorded JSON file in https://www.speedscope.app/
Differential Revision: https://phab.mercurial-scm.org/D10554
Simon Sapin <simon.sapin@octobus.net> [Thu, 29 Apr 2021 11:32:57 +0200] rev 47133
dirstate-tree: Use HashMap instead of BTreeMap
BTreeMap has the advantage of its "natural" iteration order being the one we need
in the status algorithm. With HashMap however, iteration order is undefined so
we need to allocate a Vec and sort it explicitly.
Unfortunately many BTreeMap operations are slower than in HashMap, and skipping
that extra allocation and sort is not enough to compensate.
Switching to HashMap + sort makes `hg status` 17% faster in one test case,
as measure with hyperfine:
```
Benchmark #1: ../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1
Time (mean ± σ): 765.0 ms ± 8.8 ms [User: 1.352 s, System: 0.747 s]
Range (min … max): 751.8 ms … 778.7 ms 10 runs
Benchmark #2: ./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1
Time (mean ± σ): 651.8 ms ± 9.9 ms [User: 1.251 s, System: 0.799 s]
Range (min … max): 642.2 ms … 671.8 ms 10 runs
Summary
'./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ran
1.17 ± 0.02 times faster than '../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1'
```
* ./hg is this revision
* ../hg2/hg is its parent
* $REPO is an old snapshot of mozilla-central
Differential Revision: https://phab.mercurial-scm.org/D10553
Simon Sapin <simon.sapin@octobus.net> [Tue, 27 Apr 2021 17:49:38 +0200] rev 47132
dirstate-tree: Add #[timed] attribute to `status` and `DirstateMap::read`
When running with a `RUST_LOG=trace` environment variable, the `micro_timer`
crate prints the duration taken by each call to functions with that attribute.
Differential Revision: https://phab.mercurial-scm.org/D10552
Simon Sapin <simon.sapin@octobus.net> [Tue, 27 Apr 2021 14:20:48 +0200] rev 47131
dirstate-tree: Paralellize the status algorithm with Rayon
The `rayon` crate exposes "parallel iterators" that work like normal iterators
but dispatch work on different items to an implicit global thread pool.
Differential Revision: https://phab.mercurial-scm.org/D10551
Simon Sapin <simon.sapin@octobus.net> [Tue, 27 Apr 2021 12:42:21 +0200] rev 47130
dirstate-tree: Avoid BTreeMap double-lookup when inserting a dirstate entry
The child nodes of a given node in the tree-shaped dirstate are kept in a
`BTreeMap` where keys are file names as strings. Finding or inserting a value
in the map takes `O(log(n))` string comparisons, which adds up when constructing
the tree.
The `entry` API allows finding a "spot" in the map that may or may not be
occupied and then access that value or insert a new one without doing map
lookup again. However the current API is limited in that calling `entry`
requires an owned key (and so a memory allocation), even if it ends up not
being used in the case where the map already has a value with an equal key.
This is still a win, with 4% better end-to-end time for `hg status` measured
here with hyperfine:
```
Benchmark #1: ../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1
Time (mean ± σ): 1.337 s ± 0.018 s [User: 892.9 ms, System: 437.5 ms]
Range (min … max): 1.316 s … 1.373 s 10 runs
Benchmark #2: ./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1
Time (mean ± σ): 1.291 s ± 0.008 s [User: 853.4 ms, System: 431.1 ms]
Range (min … max): 1.283 s … 1.309 s 10 runs
Summary
'./hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1' ran
1.04 ± 0.02 times faster than '../hg2/hg status -R $REPO --config=experimental.dirstate-tree.in-memory=1'
```
* ./hg is this revision
* ../hg2/hg is its parent
* $REPO is an old snapshot of mozilla-central
Differential Revision: https://phab.mercurial-scm.org/D10550
Simon Sapin <simon.sapin@octobus.net> [Mon, 26 Apr 2021 19:28:56 +0200] rev 47129
dirstate-tree: Handle I/O errors in status
Errors such as insufficient permissions when listing a directory are logged,
and the algorithm continues without considering that directory.
Differential Revision: https://phab.mercurial-scm.org/D10549
Simon Sapin <simon.sapin@octobus.net> [Mon, 26 Apr 2021 19:16:23 +0200] rev 47128
dirstate-tree: Ignore FIFOs etc. in the status algorithm
If a filesystem directory contains anything that is not:
* a "normal" file
* a symbolic link
* or a directory
… act as if that directory entry was not there. For example, if that path was
previously a tracked file, mark it as deleted or removed.
Differential Revision: https://phab.mercurial-scm.org/D10548
Simon Sapin <simon.sapin@octobus.net> [Fri, 16 Apr 2021 12:12:41 +0200] rev 47127
dirstate-tree: Add the new `status()` algorithm
With the dirstate organized in a tree that mirrors the structure of the
filesystem tree, we can traverse both trees at the same time in order to
compare them. This is hopefully more efficient that building multiple
big hashmaps for all of the repository’s contents.
Differential Revision: https://phab.mercurial-scm.org/D10547
Simon Sapin <simon.sapin@octobus.net> [Fri, 16 Apr 2021 12:12:04 +0200] rev 47126
dirstate-tree: Give to `status()` mutable access to the `DirstateMap`
Differential Revision: https://phab.mercurial-scm.org/D10546
Simon Sapin <simon.sapin@octobus.net> [Tue, 06 Apr 2021 15:49:01 +0200] rev 47125
rust: Add doc-comments to DirstateStatus fields
Differential Revision: https://phab.mercurial-scm.org/D10495
Simon Sapin <simon.sapin@octobus.net> [Tue, 06 Apr 2021 15:14:19 +0200] rev 47124
rust: Move "lookup" a.k.a. "unsure" paths into `DirstateStatus` struct
Instead of having `status()` returning a tuple of those paths and
`DirstateStatus`.
Differential Revision: https://phab.mercurial-scm.org/D10494
Simon Sapin <simon.sapin@octobus.net> [Tue, 13 Apr 2021 17:02:58 +0200] rev 47123
rust: Remove DirstateMap::file_fold_map
This was a HashMap constructed on demand and then cached in the DirstateMap
struct to avoid reconstructing at the next access. However the only use is
in Python bindings converting it to a PyDict. That method in turn is wrapped
in a @cachedproperty in Python code.
This was two redudant layers of caching. This changeset removes the Rust-level
one to keep the Python dict cache, and have bindings create a PyDict by
iterating.
Differential Revision: https://phab.mercurial-scm.org/D10493
Simon Sapin <simon.sapin@octobus.net> [Fri, 09 Apr 2021 13:13:19 +0200] rev 47122
dirstate-tree: Add "non normal" and "from other parent" sets
Unlike the other DirstateMap implementation, these sets are not materialized
separately in memory. Instead we traverse the main tree.
Differential Revision: https://phab.mercurial-scm.org/D10492
Simon Sapin <simon.sapin@octobus.net> [Fri, 09 Apr 2021 12:55:35 +0200] rev 47121
dirstate-tree: Add add_file, remove_file, and drop_file
Again, various counters need to be kept up to date.
Differential Revision: https://phab.mercurial-scm.org/D10491
Simon Sapin <simon.sapin@octobus.net> [Mon, 12 Apr 2021 19:46:24 +0200] rev 47120
dirstate-tree: Add has_dir and has_tracked_dir
A node without a `DirstateMap` entry represents a directory.
Only some values of `EntryState` represent tracked files.
A directory is considered "tracked" if it contains any descendant file that
is tracked. To avoid a sub-tree traversal in `has_tracked_dir` we add a
counter for this. A boolean flag would become insufficent when we implement
remove_file and drop_file.
`add_file_node` is more general than needed here, in anticipation of adding
the `add_file` and `remove_file` methods.
Differential Revision: https://phab.mercurial-scm.org/D10490
Simon Sapin <simon.sapin@octobus.net> [Mon, 12 Apr 2021 18:42:51 +0200] rev 47119
dirstate-tree: Add clear_ambiguous_times in the new DirstateMap
Also drive-by refactor it in the other DirstateMap
Differential Revision: https://phab.mercurial-scm.org/D10489
Simon Sapin <simon.sapin@octobus.net> [Mon, 12 Apr 2021 17:53:37 +0200] rev 47118
dirstate-tree: Add copy_map_insert and copy_map_remove
Differential Revision: https://phab.mercurial-scm.org/D10488
Simon Sapin <simon.sapin@octobus.net> [Mon, 12 Apr 2021 17:29:55 +0200] rev 47117
dirstate-tree: Maintain a counter of DirstateEntry’s and copy sources
This allows implementing __len__ for DirstateMap and CopyMap efficiently,
without traversing the tree.
Differential Revision: https://phab.mercurial-scm.org/D10487
Simon Sapin <simon.sapin@octobus.net> [Mon, 12 Apr 2021 14:21:47 +0200] rev 47116
dirstate-tree: Serialize to disk
The existing `pack_dirstate` function relies on implementation details
of `DirstateMap`, so extract some parts of it as separate functions
for us in the tree-based `DirstateMap`.
The `bytes-cast` crate is updated to a version that has an `as_bytes` method,
not just `from_bytes`:
https://docs.rs/bytes-cast/0.2.0/bytes_cast/trait.BytesCast.html#method.as_bytes
Drive-by refactor `clear_ambiguous_times` which does part of the same thing.
Differential Revision: https://phab.mercurial-scm.org/D10486
Simon Sapin <simon.sapin@octobus.net> [Mon, 12 Apr 2021 14:43:45 +0200] rev 47115
rust: Add a Timestamp struct instead of abusing Duration
`SystemTime` would be the standard library type semantically appropriate
instead of `Duration`.
But since the value is coming from Python as a plain integer and used in
dirstate packing code as an integer, let’s make a type that contains a single
integer instead of using one with sub-second precision.
Differential Revision: https://phab.mercurial-scm.org/D10485
Simon Sapin <simon.sapin@octobus.net> [Tue, 06 Apr 2021 21:07:12 +0200] rev 47114
dirstate-tree: Add tree traversal/iteration
Like Python’s, Rust’s iterators are "external" in that they are driven
by a caller who calls a `next` method. This is as opposed to "internal"
iterators who drive themselves and call a callback for each item.
Writing an internal iterator traversing a tree is easy with recursion,
but internal iterators cannot rely on the call stack in that way,
they must save in an explicit object all state that they need to be
preserved across two `next` calls.
This algorithm uses a `Vec` as a stack that contains what would be
local variables on the call stack if we could use recursion.
Differential Revision: https://phab.mercurial-scm.org/D10370
Simon Sapin <simon.sapin@octobus.net> [Tue, 06 Apr 2021 14:35:39 +0200] rev 47113
dirstate-tree: Add map `get` and `contains_key` methods
Differential Revision: https://phab.mercurial-scm.org/D10369
Simon Sapin <simon.sapin@octobus.net> [Tue, 06 Apr 2021 14:29:05 +0200] rev 47112
dirstate-tree: Add parsing only dirstate parents from disk
Differential Revision: https://phab.mercurial-scm.org/D10368
Simon Sapin <simon.sapin@octobus.net> [Wed, 31 Mar 2021 18:59:49 +0200] rev 47111
dirstate-tree: Implement DirstateMap::read
This reads the on-disk dirstate in its current format (a flat sequence of
entries) and creates a tree in memory.
Differential Revision: https://phab.mercurial-scm.org/D10367
Simon Sapin <simon.sapin@octobus.net> [Thu, 08 Apr 2021 20:12:24 +0200] rev 47110
dirstate-tree: Add `WithBasename` wrapper for `HgPath`
In the tree-shaped dirstate we want to have nodes representing files or
directories, where directory nodes contain a map associating "base" names
to child nodes for child files and directories.
Many dirstate operations expect a full path from the repository root, but
re-concatenating string from nested map keys all the time might be expensive.
Instead, `WithBasename` stores a full path for these operations but
behaves as its base name (last path component) for equality and comparison.
Additionally `inclusive_ancestors` provides the successive map keys
that are needed when inserting a new dirstate node at a given full path.
Differential Revision: https://phab.mercurial-scm.org/D10365
Simon Sapin <simon.sapin@octobus.net> [Tue, 30 Mar 2021 09:56:04 +0200] rev 47109
dirstate-tree: Empty shell for a second Rust DirstateMap implementation
For background see description of the previous changeset
"Make Rust DirstateMap bindings go through a trait object".
Add an empty shell for a opt-in second Rust implementation of the
`DirstateMap` type and the `status` function. For now all methods panic.
This can be seen in "action" with:
./hg status --config experimental.dirstate-tree.in-memory=1
Differential Revision: https://phab.mercurial-scm.org/D10364
Simon Sapin <simon.sapin@octobus.net> [Thu, 08 Apr 2021 14:58:44 +0200] rev 47108
dirstate-tree: Abstract "non-normal" and "other parent" sets
Instead of exposing `HashSet`s directly, have slightly higher-level
methods for the operations that Python bindings need on them.
Differential Revision: https://phab.mercurial-scm.org/D10363
Simon Sapin <simon.sapin@octobus.net> [Tue, 30 Mar 2021 14:15:23 +0200] rev 47107
dirstate-tree: Make Rust DirstateMap bindings go through a trait object
This changeset starts a series that adds an experiment to make status faster
by changing the dirstate (first only in memory and later also on disk) to
be shaped as a tree matching the directory structure, instead of the current
flat collection of entries. The status algorithm can then traverse this tree
dirstate at the same time as it traverses the filesystem.
We (Octobus) have made prototypes that show promising results but are prone
to bitrot. We would like to start upstreaming some experimental Rust code that
goes in this direction, but to avoid disrupting users it should only be
enabled by some run-time opt-in while keeping the existing dirstate structure
and status algorithm as-is.
The `DirstateMap` type and `status` function look like the appropriate
boundary. This adds a new trait that abstracts everything Python bindings need
and makes those bindings go through a `dyn` trait object. Later we’ll have two
implementations of this trait, and the same bindings can use either.
Differential Revision: https://phab.mercurial-scm.org/D10362
Kévin Lévesque <klevesque@innovmetric.com> [Wed, 05 May 2021 18:26:04 -0400] rev 47106
remotefilelog: use the correct capability when using getfilestype threaded
The functon was overlooked when the capability was renamed
Differential Revision: https://phab.mercurial-scm.org/D10673
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47105
test-copies: test that copies' sidedata can get computed during push
If the source of the push does not have the necessary sidedata but the server
needs them, the client should compute them on the fly while pushing.
Differential Revision: https://phab.mercurial-scm.org/D10350
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47104
test-copies: test that copies' sidedata can get computed during pull
If the source does not have the data, the pulling client should compute the necessary side-data while pulling.
Differential Revision: https://phab.mercurial-scm.org/D10349
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47103
test-copies: test that copies' sidedata does not get corrupted during push
This is an important usecase.
Differential Revision: https://phab.mercurial-scm.org/D10348
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47102
test-copies: test that copies' sidedata does not get corrupted during pull
This is an important usecase.
Differential Revision: https://phab.mercurial-scm.org/D10347
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47101
test-copies: simplify some conditional output
Now that all computation using sidedata give the same result, we can
simplify conditional
Differential Revision: https://phab.mercurial-scm.org/D10346
Raphaël Gomès <rgomes@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47100
sidedata: move documentation about sidedata helpers to sidedata module
Differential Revision: https://phab.mercurial-scm.org/D10361
Raphaël Gomès <rgomes@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47099
sidedata: move sidedata-related utils to the dedicated module
Differential Revision: https://phab.mercurial-scm.org/D10360
Raphaël Gomès <rgomes@octobus.net> [Mon, 19 Apr 2021 11:22:24 +0200] rev 47098
sidedata: replace sidedata upgrade mechanism with the new one
Note: this is split into a separate change (like some other patches in this
series) because it's not easy to have all patches work 100% and this seemed
easier for reviewers.
When cloning or upgrading a repo, we may need to compute (or remove) sidedata.
This is the same mechanism that is used in exchange, so we re-use the new
system to simplify the code and fix the remaining issues (correctly dropping
flags and handling partial removal, etc.).
This also highlighted an issue with `test-copies-in-changeset.t` that kept
sidedata categories that are not relevant anymore. They should probably be
dropped entirely, but that would be for another patch.
Differential Revision: https://phab.mercurial-scm.org/D10359
Raphaël Gomès <rgomes@octobus.net> [Mon, 19 Apr 2021 11:22:21 +0200] rev 47097
sidedata: add a way of replacing an existing sidedata computer
This will be useful in a future patch to replace a sequential computer with
a parallel computer. We only allow for explicit replacement, to force the users
to think about overriding computers.
Differential Revision: https://phab.mercurial-scm.org/D10358
Raphaël Gomès <rgomes@octobus.net> [Thu, 08 Apr 2021 16:30:10 +0200] rev 47096
bundle2: remove restriction around sidedata
We are now capable of generating the missing sidedata on-the-fly.
Differential Revision: https://phab.mercurial-scm.org/D10345
Raphaël Gomès <rgomes@octobus.net> [Thu, 08 Apr 2021 16:55:17 +0200] rev 47095
sidedata: enable sidedata computers to optionally rewrite flags
Sidedata computers may want to influence the flags of the revision they touch.
For example, the computer for changelog-based copytracing can add a flag to
signify that this revision might affect copytracing, inversely removing said
flag if the information is no longer applicable.
See inline documentation in `storageutil` for more details.
Differential Revision: https://phab.mercurial-scm.org/D10344
Raphaël Gomès <rgomes@octobus.net> [Sat, 10 Apr 2021 11:27:40 +0200] rev 47094
cg4: introduce protocol flag to signify the presence of sidedata
We need a way of signaling whether the current revision has sidedata or not,
and re-using the revision flags would waste potential revlog flags and mix two
normally independent layers.
In this change, we add a single byte at the start of the ch4 delta header to
set potential protocol flags. We also reclaim the revlog flag for sidedata,
since it is no longer used, in its place now lives the (also experimental)
copytracing flag.
When generating deltas, apply the `CG_FLAG_SIDEDATA` flag if there is sidedata.
When applying the deltas, if said flag is present, the next chunk contains the
sidedata.
Differential Revision: https://phab.mercurial-scm.org/D10343
Raphaël Gomès <rgomes@octobus.net> [Thu, 08 Apr 2021 16:34:11 +0200] rev 47093
changegroup: don't limit cgv4 to revlogv2 repos
To help the transition from revlogv1 to revlogv2, we need to be able to enable
cgv4 for revlogv1 repos, so that revlogv2 clients can handle adding/removing
sidedata over the wire.
Differential Revision: https://phab.mercurial-scm.org/D10342
Raphaël Gomès <rgomes@octobus.net> [Thu, 08 Apr 2021 16:39:39 +0200] rev 47092
sidedata: gate sidedata functionality to revlogv2 in more places
Since revlogv1 is not capable of storing sidedata, we prevent sidedata
mechanisms around the revlog layer from doing anything. We however keep the
ones that allow a revlogv1 repo to generate sidedata on-the-fly on a push, the
pull case simply does not add the sidedata to the revlog.
Differential Revision: https://phab.mercurial-scm.org/D10341
Raphaël Gomès <rgomes@octobus.net> [Tue, 30 Mar 2021 17:03:02 +0200] rev 47091
sidedata: register copies sidedata computer regardless of the revlog version
Repositories should not gate their sidedata computers based on any requirement,
only their wanted sidedata. A repository might need to generate sidedata wanted
by the peer that it itself does not want.
Differential Revision: https://phab.mercurial-scm.org/D10340
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Apr 2021 19:00:21 +0200] rev 47090
revlog: replace the old `revlog_kind` approach with the new `target` one
The new `target` attribute supersedes the previous one.
Differential Revision: https://phab.mercurial-scm.org/D10353
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Apr 2021 05:20:24 +0200] rev 47089
revlog: introduce an explicit tracking of what the revlog is about
Since the dawn of time, people have been forced to rely to lossy introspection
of the index filename to determine what the purpose and role of the revlog they
encounter is. This is hacky, error prone, inflexible, abstraction-leaky,
<insert-your-own-complaints-here>.
In f63299ee7e4d Raphaël introduced a new attribute to track this information:
`revlog_kind`. However it is initialized in an odd place and various instances
end up not having it set. In addition is only tracking some of the information
we end up having to introspect in various pieces of code.
So we add a new attribute that holds more data and is more strictly enforced.
This work is done in collaboration with Raphaël.
The `revlog_kind` one will be removed/adapted in the next changeset. We expect
to be able to clean up various existing piece of code and to simplify coming
work around the newer revlog format.
Differential Revision: https://phab.mercurial-scm.org/D10352
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 May 2021 08:54:28 -0700] rev 47088
config: add --source option to include source of value
Showing the source of each config option is quite useful and not
something the user should have to reach for the `--debug` flag for.
I updates documentation and tests, except for one place in
`test-hgrc.t` where I thought the test might have been intended to
also test that `--debug` results in `ui.quiet` etc being test.
Differential Revision: https://phab.mercurial-scm.org/D10668
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 May 2021 10:49:32 -0700] rev 47087
rewriteutil: say how many commits would become orphan if commit is rewritten
This copies the message from the evolve extension, but modifies it a
bit to work with the grammar (in particular with the use of "change
branch of" as `action`). I don't know why it doesn't use the same
`_formatrevs()` as for public commmits.
Differential Revision: https://phab.mercurial-scm.org/D10671
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 May 2021 10:16:34 -0700] rev 47086
rewriteutil: give examples of public changesets that can't be rewritten
This patch copies the feature from the evolve extension.
Differential Revision: https://phab.mercurial-scm.org/D10670
Augie Fackler <augie@google.com> [Mon, 03 May 2021 15:14:09 -0400] rev 47085
merge: with stable
Raphaël Gomès <rgomes@octobus.net> [Mon, 03 May 2021 18:55:19 +0200] rev 47084
branching: merge stable into default
Joerg Sonnenberger <joerg@bec.de> [Fri, 30 Apr 2021 02:11:58 +0200] rev 47083
manifests: push down expected node length into the parser
This strictly enforces the node length in the manifest lines according
to what the repository expects. One test case moves large hash testing
into the non-treemanifest part as treemanifests don't provide an
interface for overriding just the node length for now.
Differential Revision: https://phab.mercurial-scm.org/D10533
Joerg Sonnenberger <joerg@bec.de> [Fri, 30 Apr 2021 03:19:45 +0200] rev 47082
core: don't hard-code node length
Differential Revision: https://phab.mercurial-scm.org/D10536
Joerg Sonnenberger <joerg@bec.de> [Fri, 30 Apr 2021 03:09:16 +0200] rev 47081
core: don't hard-code hex node lengths
Differential Revision: https://phab.mercurial-scm.org/D10535
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 02:33:00 +0200] rev 47080
enforcesinglehead-test: add the expected node output next to the error
this make it simpler to validate that the test is correct.
Differential Revision: https://phab.mercurial-scm.org/D10545
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Apr 2021 00:34:16 +0200] rev 47079
revlog: code for `revlogv0` in its own module
This code is mostly unused compatiblity code. Yet it take a prohiminent place in
the `revlog.py` module. That module is already quite big, so we move all that
code in a dedicated module.
Differential Revision: https://phab.mercurial-scm.org/D10511
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 01 Apr 2021 11:31:54 +0200] rev 47078
revlog: have an explicit "pack_header" method
Having to pass the version header when retrieving the binary version of every
single entry is a bit silly. So we extract that special logic in its own method.
This also prepare the move to newer revlog format, not storing the header within
an actual entry…
Differential Revision: https://phab.mercurial-scm.org/D10510
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 01 May 2021 14:47:39 +0200] rev 47077
revlog: remove the revlogio class
The class only contains a single `parseindex` method. Lets just make it a
function and remove the `revlogio` class. It served us well but became thinner
and thinner overtime.
Differential Revision: https://phab.mercurial-scm.org/D10509
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 01 May 2021 14:47:33 +0200] rev 47076
revlog: fix some comment style
They displease check-code.
Differential Revision: https://phab.mercurial-scm.org/D10542
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Apr 2021 00:01:11 +0200] rev 47075
revlog: add a `entry_binary` method on index
The revlog index is already responsible for unpacking the binary entry, it would be
simpler to make it responsible for packing them. In practice the C version of
the index is already doing this internally.
We introduce a "entry_binary" method that return the binary version of an
existing revision. The method currently need to also take the revlog header to
deal with the "first revision" special case. We will introduce further refactor
in a later changeset to split that logic out.
Differential Revision: https://phab.mercurial-scm.org/D10508
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 12:08:34 +0200] rev 47074
template: make an explicit closure for formatting entry in peerurls
This is about to be become significantly more complicated as `ui.path[x]` will
become a list.
Differential Revision: https://phab.mercurial-scm.org/D10443
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 11:50:08 +0200] rev 47073
template: use `list_paths` in `peerurls`
Using common code will make it simpler to update the logic behind the path
definition and storage.
Differential Revision: https://phab.mercurial-scm.org/D10442
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 11:48:29 +0200] rev 47072
paths: use `list_paths` in `hg paths`
Using common code will make it simpler to update the logic behind the path
definition and storage.
Differential Revision: https://phab.mercurial-scm.org/D10441
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 11:46:31 +0200] rev 47071
urlutil: introduce a new `list_paths` function
This function will be useful for command and template that wants to display path
related information.
Differential Revision: https://phab.mercurial-scm.org/D10440
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 10:05:51 +0200] rev 47070
urlutil: deprecate `getpath`
There as no remaining user of that function.
Differential Revision: https://phab.mercurial-scm.org/D10439
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 10:01:44 +0200] rev 47069
urlutil: inline the relevant part of `getpath` in `get_push_paths`
The part that `get_push_paths` needs is quite simple, inclining will help us
to deprecated `getpath`.
Differential Revision: https://phab.mercurial-scm.org/D10438
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 09:50:56 +0200] rev 47068
url_util: introduce a `try_path` function
That function… try a build a path, returning None on failure. This helps us to simplify various part of the existing code.
Differential Revision: https://phab.mercurial-scm.org/D10437
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Apr 2021 11:22:35 -0700] rev 47067
narrow: add more status messages when narrowing
Each of the steps I added status messages for in this patch frequently
take minutes or tens of minutes for our internal users.
It would be nice to also have a progress bar but that will have to
come later.
Differential Revision: https://phab.mercurial-scm.org/D10503
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Apr 2021 10:24:03 -0700] rev 47066
narrow: add progress-reporting when looking for local changes in `hg tracked`
Looking for local changes (changes not on the given remote) can take a
long time, so we should have progress-reporting for it.
Differential Revision: https://phab.mercurial-scm.org/D10501
Kyle Lippincott <spectral@google.com> [Fri, 16 Apr 2021 16:21:26 -0700] rev 47065
chg: pass --no-profile to disable profiling when starting hg serve
If profiling is enabled via global/user config (as far as I can tell, this
doesn't affect use of the --profile flag, but it probably does affect --config
profiling.enabled=1), then the profiling data can be *cumulative* for the
lifetime of the chg process.
This leads to some "interesting" results where hg claims the walltime is
something like 200s on a command that took only a second or two to run. Worse,
however, is that with at least some profilers (such as the default "stat"
profiler), this can cause a large slowdown while generating the profiler output.
Differential Revision: https://phab.mercurial-scm.org/D10470
Kyle Lippincott <spectral@google.com> [Fri, 16 Apr 2021 15:31:05 -0700] rev 47064
profiling: add --no-profile to disable profiling enabled via config
Differential Revision: https://phab.mercurial-scm.org/D10469
Kyle Lippincott <spectral@google.com> [Fri, 16 Apr 2021 18:56:26 -0700] rev 47063
tests: fix test-chg to ignore a warning about being unable to set locale
This is apparently coming from bash when bash is providing the sh that we're
using to execute the .sh file generated by run-tests for this test.
Bash on my machine:
```
$ sh --version
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ sh
sh-5.1$ LC_CTYPE=unsupported_value echo hi
sh: warning: setlocale: LC_CTYPE: cannot change locale (unsupported_value): No such file or directory
hi
```
Differential Revision: https://phab.mercurial-scm.org/D10468
Martin von Zweigbergk <martinvonz@google.com> [Tue, 27 Apr 2021 14:36:52 -0700] rev 47062
branch: delete obsolete message about changing branch of obsolete commit
We now rely on `rewriteutil.precheck()` to check for divergence, so we
don't need the extra check in `cmdutil.changebranch()`. The former
check is a little less strict in that it allows you to rewrite a
commit without non-obsolete successors.
Differential Revision: https://phab.mercurial-scm.org/D10519
Martin von Zweigbergk <martinvonz@google.com> [Tue, 23 Feb 2021 10:28:42 -0800] rev 47061
rewriteutil: check for divergence
This code is adapted from the code in the evolve extension. It seems
to be equivalent as far as the evolve extension's test suite can tell
(the only impact when making their `precheck()` delegate to our
version is that error messages are less detailed).
I had to change the error message to work with "change branch of"
being inserted as the action.
Differential Revision: https://phab.mercurial-scm.org/D10518
Martin von Zweigbergk <martinvonz@google.com> [Wed, 28 Apr 2021 08:48:10 -0700] rev 47060
rewriteutil: adapt "cannot %s while merging" to work with "change branch of"
`rewriteutil.precheck()` creates error messages by inserting a given
verb into a sentence. The `hg branch -r` command passes in "change
branch of" as the verb. That doesn't work well with "cannot %s while
merging" (making it "cannot change branch of while merging"). Let's
insert a "changeset" there to make it work better.
Building sentences like this seems obviously bad for i18n, but fixing
that is out of scope for this series, IMO.
Differential Revision: https://phab.mercurial-scm.org/D10530
Martin von Zweigbergk <martinvonz@google.com> [Thu, 11 Feb 2021 15:11:10 -0800] rev 47059
rewriteutil: point to help about instability when rewriting creates orphan
This replicates the message from the evolve extension.
Differential Revision: https://phab.mercurial-scm.org/D10517
Martin von Zweigbergk <martinvonz@google.com> [Tue, 27 Apr 2021 14:59:45 -0700] rev 47058
help: add topic about evolution, based on text from evolve extension
I've taken the text produced by `hg help evolution` when the evolve
extension is enabled and made that available by the same command with
just hg core. Changes I've made:
* Added "(EXPERIMENTAL)" to the title. (That doesn't hide the topic
from `hg help`, though.)
* Replaced old-style `experimental.evolution=<names>` config by
new-style `experimental.evolution.<name>=true`.
* Replaces a "obsolete markers" by "obsolescence markers".
* Removed most content from "Current feature status".
When the evolve extension is enabled, its help text takes precedence.
Differential Revision: https://phab.mercurial-scm.org/D10516
Martin von Zweigbergk <martinvonz@google.com> [Thu, 11 Feb 2021 15:02:57 -0800] rev 47057
rewriteutil: add devel warning if precheck is called with contexts
I'm trying to upstream parts from the evolve extension. This check
exists there.
Differential Revision: https://phab.mercurial-scm.org/D10515
Martin von Zweigbergk <martinvonz@google.com> [Tue, 27 Apr 2021 11:02:41 -0700] rev 47056
rewriteutil: replace "null changeset" by "the null revision" in error message
The evolve extension uses "the null revision" and we seem to use that
term much more frequently in core too.
Differential Revision: https://phab.mercurial-scm.org/D10514
Joerg Sonnenberger <joerg@bec.de> [Mon, 29 Mar 2021 01:52:06 +0200] rev 47055
node: replace nullid and friends with nodeconstants class
The introduction of 256bit hashes require changes to nullid and other
constant magic values. Start pushing them down from repository and
revlog where sensible.
Differential Revision: https://phab.mercurial-scm.org/D9465
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 09 Jul 2021 00:25:14 +0530] rev 47054
Added signature for changeset 411dc27fd9fd
Pulkit Goyal <7895pulkit@gmail.com> [Fri, 09 Jul 2021 00:25:07 +0530] rev 47053
Added tag 5.8.1 for changeset 411dc27fd9fd
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jul 2021 16:12:09 +0200] rev 47052
corruption: backout changeset 49fd21f32695 (issue6528)
inverting the parent is masking copy information leading to bad content being
fetched and bad status result.
Since 49fd21f32695, exchange can actively swap these parent corrupting existing
changesets and triggering the corruption.
Data corruption are considered critical so backing this out and doing and
unscheduled release seems in order.
Differential Revision: https://phab.mercurial-scm.org/D10995
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Jul 2021 19:22:50 +0200] rev 47051
corruption: add a test for issue6528
The initial reproduction script was provided by Charles Chamberlain from Jane
Street.
Differential Revision: https://phab.mercurial-scm.org/D10996
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 24 Jun 2021 03:22:03 +0200] rev 47050
stream: double check that self.vfs is *not* in the vfsmap
The stream clone logic allows for writing any content to any file under various
vfs. This is *not* suitable for *vfs*, since writing in `.hg/` directly allow to
modify the configuration and is a great and simple gateway for remote code
execution.
Differential Revision: https://phab.mercurial-scm.org/D10905
Mathias De Mare <mathias.de_mare@nokia.com> [Wed, 30 Jun 2021 14:17:28 +0200] rev 47049
Backed out changeset 9b8f326731ac
Unfortunately, disabling the rust extensions means newer
Mercurial versions no longer have the persistent-nodemap
feature enabled.
This means a Mercurial 5.8.1 on RPM-based Linux distributions
will no longer be able to read repositories created
by a Mercurial 5.8 on RPM-based Linux distributions.
This violates the compatibility rules
(see https://www.mercurial-scm.org/wiki/CompatibilityRules ).
For this reason, I have to backout this change.
I'll try to find another solution to the 'hg purge' crashes.
Differential Revision: https://phab.mercurial-scm.org/D10918