Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:38:13 +0100] rev 50678
store: directly pass the filesize in the `details` of revlog
The dictionary only contains 1 (or 0) entries, we can directly store that
information (or None).
Moving to a simpler argument passing result in a noticable speedup (because
Python)
The number below use `hg perf::stream-locked-section` to measure the time spend
in the locked section of the streaming clone. Number are run on various
repository.
### mercurial-2018-08-01-zstd-sparse-revlog
before: 0.031247 seconds
after: 0.030246 seconds (-3.20%)
### mozilla-central-2018-08-01-zstd-sparse-revlog
before: 6.718968 seconds
after: 6.304179 seconds (-6.17%)
### mozilla-try-2019-02-18-zstd-sparse-revlog
before: 14.631343 seconds
after: 14.142687 seconds (-3.34%)
### netbeans-2018-08-01-zstd-sparse-revlog
before: 2.895584 seconds
after: 2.719939 seconds (-6.07%)
### pypy-2018-08-01-zstd-sparse-revlog
before: 0.561843 seconds
after: 0.543034 seconds (-3.35%)
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:35:10 +0100] rev 50677
store: explicitly pass file_size when creating StoreFile
A small cleanup before large cleanup in the next patch.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 30 May 2023 16:33:28 +0100] rev 50676
store: have the revlog determine which files are volatile itself
This is a first step toward simplifying the walk step.
Mathias De Mare <mathias.de_mare@nokia.com> [Wed, 08 Mar 2023 14:23:43 +0100] rev 50675
clonebundles: add support for inline (streaming) clonebundles
The idea behind inline clonebundles is to send them through
the ssh or https connection to the Mercurial server.
We've been using this specifically for streaming clonebundles,
although it works for 'regular' clonebundles as well
(but is less relevant, since pullbundles exist).
We've had this enabled for around 9 months for a part
of our users.
A few benefits are:
- no need to secure an external system,
since everything goes through the same Mercurial server
- easier scaling (in our case: no risk of inconsistencies
between multiple mercurial-server mirrors and nginx clonebundles hosts)
Remaining topics/questions right now:
- The inline clonebundles don't work for https yet.
This is because httppeer doesn't seem to support sending client
capabilities.
I didn't focus on that as my main goal was to get this working
for ssh.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 31 May 2023 18:08:56 +0100] rev 50674
tree-manifest: allow `debugupgraderepo` to run on tree manifest repo
There does not seems to be anything wrong with running the current logic on
them. So we remove the limitation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 31 May 2023 16:04:16 +0100] rev 50673
stream-clone: update debugcreatestreamclonebundle helps
People to stop using streamv1, so we should point them to alternative in the
place where people might find it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 25 May 2023 00:23:05 +0200] rev 50672
rewrite: simplify the `retained_extras` extra logic
First, we move the definition of value outside of the rebase extensions, as this
apply to all rebase-like operation and some live in other place (like evolve).
Second we make it a simple set, so that it is easy for an extension to add a new
value in it.
Third, we move the associated logic in core too. That make it easily available
to other extensions.
Fourth we simplify it usage, as the verbose version of the filtering is just a
handful on line long, we are just going to test all the value for updates, so
the Projection overlay is not bringing much here.
Note that, we make it a module level set, is a key is worth preserving it is
probably worth preserving in all cases. This was already the behavior prior to
this change.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 18:41:58 +0200] rev 50671
stream-clone: smoothly detect and handle a case were a revlog is split
This detect and handle the most common case for a race condition around stream
and revlog splitting. The one were the revlog is split between the initial
collection of data and the time were we start considering stream that data.
In such case, we repatch an inlined version of that revlog together when this
happens. This is necessary as stream-v2 promised a specific number of bytes and
a specific number of files to the client. In stream-v3, we will have the
opportunity to just send a split revlog instead.
Getting a better version of the protocol for stream-v3 is still useful, but it
is no longer a blocket to fix that race condition.
Note that another, rarer race condition exist, were the revlog is split while
we creating the revlog and extracing content from it. This can be dealt with
later.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 14:07:58 +0200] rev 50670
stream-clone: implement decidated `get_streams` method for revlog
For revlog, we can do better using the maximum linkrev expected. This approach
open the way to dealing with a much larger set of non-trivial changes, like
splitting of inline revlogs.
We will actually tackle this issue in the next changesets (thanks to this one).
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 05:52:58 +0200] rev 50669
stream-clone: make it the responsability of the store entry to stream content
The store entry has more context, this will especially be true when it comes to
revlogs. So we move the details of how to retrieve binary content to the
StoreEntry. The stream clone code now focus on the protocol bits.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 11:42:16 +0200] rev 50668
store: declare a `files` method on BaseStoreEntry
This will help pytype to type check. We have to move `StoreFile` earlier in the
file to use it in the type declaration.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 05:23:46 +0200] rev 50667
revlog: add a `get_revlog` method
This might seen weird, but I actually thing we have been needing this for a
long time. There is multiple object that kind of pretend being revlogs while
actually wrapping the actual revlog. Since multiple code needs to access the
actuel revlog. See documentation for more details.
Expect cleanup of various places one the current series is done.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:26:39 +0200] rev 50666
stream-clone: drop the _emit_v2 function
It has no user left.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:24:39 +0200] rev 50665
stream-clone: directly use `_entries_walk` to generate stream-v2
This does not requires that much changes and will give us much more
flexibility, like improving revlog handling to gracefully handle race
situation.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:12:30 +0200] rev 50664
stream-clone: pre-indent some code
This make the next changeset clearer.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 04:12:10 +0200] rev 50663
local-clone: perform the hardlink/copy based from _entries_walk returns
We previously used `_v2_walk`. However it is not bringing us much. So lets use
the higher level function instead. This will offer us more flexibility with the
`_v2_walk` function… like deleting it eventually.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 04:24:29 +0200] rev 50662
store: cache the file_size when we get it from disk
The point of caching `files` is to ensure consistency and avoiding redoing
expensive work. So we cache the file_size once retrieved.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 28 May 2023 03:46:48 +0200] rev 50661
store: cache the `files()` return for store entries
This make it more efficient to directly use the entries list to retrieve data
in various location. It also make the entry record the file size it previously
promissed to user code, especially the stream clone code.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 27 May 2023 04:22:18 +0200] rev 50660
stream-clone: introduce a richer TempCopyManager object
This replace the previous `copy` callable with a richer object that allow access
to the backup path. This will simplify the user code as they won't need to keep
and pass around the backup path explicitly.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 13:29:01 +0200] rev 50659
store: properly compute the targer_id of manifestlog in no-fncache walk
Creating RevlogStoreEntry is good, but we need to drop the final `00manifest`
part to create something correct.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 13:28:33 +0200] rev 50658
store: do not drop the final `/` when creating manifestlog instance
This bug, inherited from the upgrade code leads to the acces/creation of broken
revlog with name `DIRECTORY00manifest.i` instead of `DIRECTORY/00manifest.i`
We fix it in its own changeset to preserve the "pure code movement" aspect of
the previous changesets.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 27 May 2023 04:01:17 +0200] rev 50657
store: add a `get_revlog_instance` method on revlog entries
The upgrade code needs this a lot, and the stream code is about to needs it
too. So we start by moving the upgrade code in a more generic location.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 02:22:20 +0200] rev 50656
stream-clone: add a test that highlight crash on revlog splitting
This has been a long running problem, we should have a tests for it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 01:38:59 +0200] rev 50655
stream-clone: remove unused code in test-clone-stream.t
We are not using the extension we create inline, we are using
`tests/testlib/ext-stream-clone-steps.py`. So let us deleted the unused version.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 01:38:34 +0200] rev 50654
stream-clone: document the ext-stream-clone-steps.py utility extension
This extension is useful, let us clarify how to use it.
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 29 May 2023 12:15:10 +0200] rev 50653
test-treemanifest: cleanup the test to more easily show server side error
This made my life easier debugging.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:26:10 +0100] rev 50652
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:51 +0100] rev 50651
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 02:57:49 +0100] rev 50650
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 01:05:41 +0100] rev 50649
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 01:04:22 +0100] rev 50648
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 03 Feb 2023 01:03:45 +0100] rev 50647
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:38:14 +0100] rev 50646
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:38:04 +0100] rev 50645
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:44 +0100] rev 50644
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:37 +0100] rev 50643
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:31 +0100] rev 50642
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:25 +0100] rev 50641
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:11 +0100] rev 50640
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:37:02 +0100] rev 50639
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:36:55 +0100] rev 50638
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:36:47 +0100] rev 50637
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:36:38 +0100] rev 50636
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:36:32 +0100] rev 50635
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:36:25 +0100] rev 50634
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:36:15 +0100] rev 50633
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:36:07 +0100] rev 50632
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:35:47 +0100] rev 50631
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:35:38 +0100] rev 50630
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:35:31 +0100] rev 50629
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:35:09 +0100] rev 50628
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:35:01 +0100] rev 50627
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:34:48 +0100] rev 50626
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:34:33 +0100] rev 50625
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:34:25 +0100] rev 50624
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 02 Feb 2023 17:34:17 +0100] rev 50623
safehasattr: pass attribute name as string instead of bytes
This is a step toward replacing `util.safehasattr` usage with plain `hasattr`.
The builtin function behave poorly in Python2 but this was fixed in Python3.
These change are done one by one as they tend to have a small odd to trigger
puzzling breackage.