Arseniy Alekseyev <aalekseyev@janestreet.com> [Thu, 18 May 2023 17:18:54 +0100] rev 50540
rust: speed up zstd decompression by re-using the decompression context
Admittedly, zstd is already pretty fast, but this change makes it
a bit faster yet: it saves ~5% of time it takes to read our large repo.
The actual motivating use case is treemanifest: in treemanifest
we end up reading *lots* of small directories, and many of them
need decompression, and there the saving for [rhg files] is >10%.
(which also seems unreasonable, we should probably keep things uncompressed more)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 16 May 2023 10:44:25 +0200] rev 50539
store: rename `topfiles` to `top_entries`
The method is now returning StoreEntries let us rename the method for clarity.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 16 May 2023 10:43:36 +0200] rev 50538
store: rename `datafiles` to `data_entries`
The method is now returning StoreEntries let us rename the method for clarity.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 22:03:39 +0200] rev 50537
store: use boolean property for upgrade's matchrevlog
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 22:11:27 +0200] rev 50536
store: use the new boolean property in `upgrade`
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 22:11:02 +0200] rev 50535
store: use the new boolean property in `remotefilelogserver`
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 22:10:33 +0200] rev 50534
store: use the boolean property in `repair_issue6528`
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 22:10:04 +0200] rev 50533
store: use the new boolean property in `narrow`
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 22:09:43 +0200] rev 50532
store: use the boolean property in `store`
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 22:09:15 +0200] rev 50531
store: introduce boolean property for revlog type
This will avoid exposing implementation details to more generic code.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:03:15 +0200] rev 50530
store: issue a single entry for each revlog
We now yield a single entry, with data about each files involved. This help to
simplify multiple code using this and it will help to further simplify and
fixes the streaming code.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:02:59 +0200] rev 50529
store: rename `unencoded_path` to `entry_path` for StoreEntry
This remove the ambiguity with StoreFile and make sure use code will be using
the right API.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:02:43 +0200] rev 50528
store: do the revlog matching on entry directly
This is the last blocker to safely merge the revlog files in a single entry.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:02:26 +0200] rev 50527
store: split the wrapping of encodedstore between _wrap and datafiles
The `datafiles` method of `basicstore` is doing a lot of work that should be
done on decoded filename. So we now wrap `_walk` to do the decoding, and less
work in `datafiles`.
This is necessary to make sure file from the same revlog can be grouped
together.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:02:09 +0200] rev 50526
store: introduce a main_file_path method for revlog
This help code that need to point revlog to an index file. This is put to use
in the upgrade code.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:01:53 +0200] rev 50525
upgrade: actually use StoreEntry API to create revlog
Lets make use of the semanctic of the object we are passed.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:01:36 +0200] rev 50524
upgrade: use StoreEntry object in upgrade
We will make more use of the API in the next changeset, but just moving to use
entry is a significant change for the engine codebase.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:01:18 +0200] rev 50523
upgrade: drop a quick fix that is not longer necessary
We won't issue bad revlog from topfile anymore.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:01:02 +0200] rev 50522
store: use StoreEntry API instead of parsing filename in largefile
This is more explicit and more robust.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:00:46 +0200] rev 50521
store: use StoreEntry API instead of parsing filename when listing manifestlog
This is more explicit and more robust.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:00:28 +0200] rev 50520
store: use StoreEntry API instead of parsing filename when fixing issue6528
This is more explicit and more robust.
We also introduce a small output change as it make things simpler and this is a
affecting a debug-command.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 09:00:13 +0200] rev 50519
store: use StoreEntry API instead of parsing filename in remotefilelog
This is more explicit and more robust.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:59:56 +0200] rev 50518
store: use StoreEntry API instead of parsing filename in narrow
This is more explicit and more robust.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:59:38 +0200] rev 50517
store: add a `target_id` attribute on RevlogStoreEntry
This hold the "target" (file, directory, etc) of a revlog. Having this
available will help a lot of code to avoid direct file path access.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:59:22 +0200] rev 50516
store: actually tag tree manifest revlogs as manifest revlogs
It turn out we have been mislabeling these for a long while. This is now fixed.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:59:06 +0200] rev 50515
store: also gather files per revlog in `topfiles`
This conclude out revlog gathering.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:58:49 +0200] rev 50514
store: also group files by revlog in fncache version of datafiles
One more step.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:58:33 +0200] rev 50513
store: add logic to group revlog file together
For now each file get its own entry, this will help stopping this, soon™.
We use such gathering in the `basicstore` code.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:58:16 +0200] rev 50512
store: change `_walk` return to `(filename, (type, size))`
If we are to group file per revlog, having the filename as the "main key" will be useful. This change will make the following changes clearer.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:58:01 +0200] rev 50511
store: lazily get file size on demand for the fncache case
We don't have the information in the first place, so we can avoid querying the
file system inconditionnaly for use case we don't needs it.
This change requires the StoreFile class to be passed a vfs to retrieve the
file_size if needed.
In the non-fncache case, we already have the information from file system
walking, so we keep it and use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:57:45 +0200] rev 50510
store: only access is_volatile information through the file object
This make sure other code only access this information through the proper API,
and it prepare for store entries to be able to agregate multiple files.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:57:30 +0200] rev 50509
store: only access file_size information through the file object
This make sure other code only access this information through the proper API,
and it prepare for store entries to be able to agregate multiple files.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:57:14 +0200] rev 50508
store: have custom init for entries class
This will get useful to add special processing later in this series.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:56:56 +0200] rev 50507
store: use specialized class for store entries
We introduce two different classes for revlog and other entries. For now, we
still have multiple entry for the same revlog, but we will work toward grouping
the different file in a single entry in this series.
Having the distinction is a step toward this goal.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:56:40 +0200] rev 50506
store: introduce a EntryFile object to actually access file info
For now a StoreEntry match a single file, but the goal is to eventually combine
multiple file in a higher level Entry, so we need to introduce this distinction
and use it first.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:56:23 +0200] rev 50505
store: use a StoreEntry object instead of tuple for store files
We want to make the store return more semantic information instead of a stream
of file path. To achieve this, we start with adding a simple object that hold
the same information as the tuple it replace, and do a simple update to the
user code to fetch and use the same information.
From there, we will be able to iteratively upgrade the codebase toward better
objects.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:56:08 +0200] rev 50504
store: no longer explicitly mark the splitted index as ignored
Since the fncache is going to automatically recognised it as non "normal"
revlog file, we don't need to explicitly make it ignored.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:55:52 +0200] rev 50503
store: cleanup what is recognized as a revlog file
The temporary file from `censor` are not relevant to the store as they are just
temporary. So let `fncache` and `datafiles` forget about them.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:55:34 +0200] rev 50502
store: only inclure revlog related file in `datafiles`
This is a "reading" equivalent of the previous patches (see that patch for
details).
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:55:19 +0200] rev 50501
store: do not record file that are not part of a revlog in fncache
The fncache exist to list file to copy/stream when cloning a repository, it
should only contains file that are relevant for a revlog in such case. For
example, temporary file are not relevant.
So we now skip the addiction of non-relevant file to the fn-cache in the first place.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:55:02 +0200] rev 50500
revlog-split: expand the some test to cover the success case too
It seems useful to double check the case of a transaction successfully
committed before trying the one with transaction abort.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:54:47 +0200] rev 50499
transaction: use a ".bck" extension for all backup file
There are too much code in Mercurial that assume file function from extension.
Keeping the original extension when doing backup is just creating confusion
(for exemple, backup of revlog's confused with new version revlogs).
So we change the backup name from now on to be explicilty backup file and
prevent this kind of problem.
(note that repository using fncache is less affected (but still affected) by
this kind of problem as the backup are not listed in the fncache).
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 15 May 2023 08:54:33 +0200] rev 50498
repair-issue6528: remove a now useless continue
The condition for that continue have been excplicly filtered out of the `files`
list a handful of line earlier.
Raphaël Gomès <rgomes@octobus.net> [Tue, 09 May 2023 11:35:50 +0200] rev 50497
branching: merge stable into default
pacien <pacien.trangirard@pacien.net> [Fri, 28 Apr 2023 12:12:42 +0200] rev 50496
stabletailgraph: clarify omission of linear parts in test sorts
Those are not interesting and are collapsed using globs.
Only the tagged nodes are interesting.
pacien <pacien.trangirard@pacien.net> [Fri, 28 Apr 2023 11:50:49 +0200] rev 50495
stabletailgraph: omit uninteresting linear parts in test log
The test uses tags for the nodes of interest.
There is no need to print the linear parts which exist only to force the rank.
pacien <pacien.trangirard@pacien.net> [Sun, 16 Apr 2023 22:49:42 +0200] rev 50494
stabletailgraph: fix typo in test
pacien <pacien.trangirard@pacien.net> [Sun, 16 Apr 2023 22:17:30 +0200] rev 50493
stabletailgraph: fix terminology in doc
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 28 Apr 2023 16:28:43 +0200] rev 50492
perf: introduce a `perf::stream-locked-section` command
This command benchmark the initial part of a stream clone, where the repository
is locked.
Anton Shestakov <av6@dwimlabs.net> [Wed, 03 May 2023 18:17:32 -0300] rev 50491
transaction: tr._names are actually bytes, use byte string to join them
Looks like this is never checked in the tests? But trying e.g. `print(tr)` by
hand before this change would give:
TypeError: sequence item 0: expected str instance, bytes found
Looks like tr._names are always bytes (although this isn't actually enforced,
but maybe at some point it will at least be type checked e.g. by pytype).
pacien <pacien.trangirard@pacien.net> [Mon, 17 Apr 2023 00:27:46 +0200] rev 50490
nix: add nix-defined package and devel env (flake)
This adds a Nix Flake which defines:
- package recipies to build and run Mercurial (C, C+Rust variants),
- a meta package to run the test suite in an isolated sandbox,
- a development environment with some tools
(notably Python and the Black formatter, pinned to match the CI).
Python is pinned to the recommended version.
The Rust toolchain is a fairly recent one provided by the Nixpkgs channel.
It is not yet pinned to the same version as the "reference" Debian package,
but this does not seem to cause any issue.
Example usage of local commands are provided in the `flake.nix` file.
Once merged, it should also be possible to pull and run directly from the
source repository directly with something like:
`nix run hg+https://foss.heptapod.net/mercurial/mercurial-devel?ref=TOPIC&dir=contrib/nix' -- version`
pacien <pacien.trangirard@pacien.net> [Thu, 20 Apr 2023 16:55:19 +0200] rev 50489
hgignore: ignore nix result symlink
pacien <pacien.trangirard@pacien.net> [Sun, 16 Apr 2023 23:48:24 +0200] rev 50488
tests: add test ignore list for nix
Some tests cannot run properly in the nix sandbox.
This adds a list of tests to ignore when running in that context.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 21 Mar 2023 14:29:35 +0000] rev 50487
ui: keep the progress bar around when writing if stdout is not a tty
Arseniy Alekseyev <aalekseyev@janestreet.com> [Fri, 10 Mar 2023 11:20:18 +0000] rev 50486
tests: in filterpyflakes, tolerate non-ascii file contents
Arseniy Alekseyev <aalekseyev@janestreet.com> [Mon, 30 Jan 2023 13:37:48 +0000] rev 50485
comments: fix spelling