Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:34:41 +0200] rev 47248
revlogv2: also test transactionality of revlog v2
Without surprise, this is broken :-)
Now that it is tested, we can fix it.
Differential Revision: https://phab.mercurial-scm.org/D10627
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:34:31 +0200] rev 47247
revlog: add a new test file focussed on testing transactionally issue
What we test here is currently also covered by `test-hooks.t`. However having
our own test file allow for more focussed/deep testing and to cover more variant
when relevant. So we create a new test file.
Differential Revision: https://phab.mercurial-scm.org/D10626
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:34:21 +0200] rev 47246
revlogv2: store version information in the docket only
Having it duplicated in the index was both useless and a risk of discrepancy.
Differential Revision: https://phab.mercurial-scm.org/D10625
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:34:11 +0200] rev 47245
revlogv2: introduce a very basic docket file
This is the first stone toward using a docket file in revlogv2. Right now the
docket is very basic and only store the version number (which is -also- stored
into the index file…) and the other files have fixed name. This new
implementation break transactionally… but they are no test checking
transactionally for revlogv2… So I take this as an opportunity to start small.
They are no usage of revlogv2 outside of tests anyway.
The docket keeps the `.i` naming used by previous version index to preserve a
unique entry point. We could decide to use a different name and look it up
first, or to fully rework this in a future "store" version. However that does
not seems necessary right now.
We will re-introduces transactionality (and associated testing…) in a later
changesets.
A long list of TODOs have been added to the relevant comment.
Differential Revision: https://phab.mercurial-scm.org/D10624
Raphaël Gomès <rgomes@octobus.net> [Mon, 17 May 2021 15:05:24 +0200] rev 47244
branching: merge stable into default
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:34:01 +0200] rev 47243
revlogv2: mark revlogv2 as requires a full upgrade
This was overlooked previously.
Differential Revision: https://phab.mercurial-scm.org/D10623
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:30:46 +0200] rev 47242
revlog: unify flag processing when loading index
The new code use a simple declaration to do centralised processing. This is
clearer, shorter and less error prone. This will be especially useful as we plan
to add a fourth format: changelog-v2.
Differential Revision: https://phab.mercurial-scm.org/D10622
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:30:35 +0200] rev 47241
revlog: unify checks for supported flag
The new code use a simple declaration to do centralised checking. This is
clearer, shorter and less error prone. This will be especially useful as we plan
to add a fourth format: changelog-v2.
Differential Revision: https://phab.mercurial-scm.org/D10621
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:30:24 +0200] rev 47240
side-data: drop the associated config and requirements
This is no longer and extensions of revlog v1, but a part of revlog v2. We no
longer needs independent config and requirement management for that.
Differential Revision: https://phab.mercurial-scm.org/D10620
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:30:14 +0200] rev 47239
revlog: fix capitalisation of an error
We don't start error message with capital letters.
Differential Revision: https://phab.mercurial-scm.org/D10619
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:30:03 +0200] rev 47238
test: no longer directly use the sidedata config and requirements
They are on their way out, so we simply use revlogv2 in these cases.
Some test does not requires explicite request for the revlogv2 format since the
upgrade to a side-data-capable format is impled by
`exp-use-copies-side-data-changeset`.
Differential Revision: https://phab.mercurial-scm.org/D10618
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:29:52 +0200] rev 47237
sidedata: add a 'side-data' repository feature and use it
Most code don't really care how sidedata support is added, but it needs to know
if it is present. To achieve this. we use the `repo.features` attributes with a
new dedicated features".
Having such centralised information is more robust and will help us to remove the
sidedata requirements in a later changesets.
Differential Revision: https://phab.mercurial-scm.org/D10617
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:29:41 +0200] rev 47236
statichttp: add the missing `features` attribute
This make statichttp closer to the interface it is "supposed" to follow.
Differential Revision: https://phab.mercurial-scm.org/D10616
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:29:30 +0200] rev 47235
sidedata: use revlogv2 requirement in the test helper
We are about to drop the requirement dedicated to sidedata (since revlogv2 will
support them unconditionally and previous version will not.).
To prepare this more we adapt the test code to not use that requirements.
Differential Revision: https://phab.mercurial-scm.org/D10615
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:29:19 +0200] rev 47234
upgrade: display sidedata before upgrade
This is a small gratuitous change that help making the test simpler to
understand on its own.
Differential Revision: https://phab.mercurial-scm.org/D10614
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:29:09 +0200] rev 47233
revlog: determine sidedata support based on the revlog version
Revlog version "2" support sidedata, previous version does not. So lets make the
initialization simpler.
Differential Revision: https://phab.mercurial-scm.org/D10613
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:28:58 +0200] rev 47232
requirements: no longer drop `generaldelta` requirement with revlogv2
A repository could use a mix of revlogv1 and revlogv2, making the requirements
still necessary. Overall we should move away from the "requirements" file being
used a way to configure the repository and stick to it "what do you need to
access this repository". However this is a wider work for another time.
In addition the logic we just dropped was confusing the `hg debugformat`
command, breaking the upgrade code and inconsistent (eg: `sparse-revlog` is also
implied by `revlogv2`).
Finally, multiple other config option would imply the use of the `revlogv2`
requirements, without drop the `generaldelta` one, leading to more
inconsistency.
Differential Revision: https://phab.mercurial-scm.org/D10612
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:28:47 +0200] rev 47231
config: drop the `format.exp-revlogv2.2` option
It is oddly named and not used at all. The one used by the code is still
`experimental.revlogv2`. So we drop that one option for consistency. We move
associate documentation to the actual option.
Differential Revision: https://phab.mercurial-scm.org/D10611
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:28:36 +0200] rev 47230
revlog: only use the `_indexfp` method for read operation
This will avoid "other" code to not overlook `_writing` usage. We introduces
private method dedicated to writing to make use the right option are always
used.
Differential Revision: https://phab.mercurial-scm.org/D10610
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:28:26 +0200] rev 47229
revlog: use `_writing` in `rewrite_sidedata`
Special cases aren't special enough to break the rules.
Differential Revision: https://phab.mercurial-scm.org/D10609
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:28:15 +0200] rev 47228
revlog: open files in 'r+' instead of 'a+'
The code doing actual writing is already doing the necessary seeking, so we
could safely use 'r+'. This make the file objecs usable in more situation, like
updating the sidedata information during pull.
revlog: forcibly move the file cursor at the right location before writing
This is a paranoid change in case the changelog computation moved the cursors
under our feets.
This is not known to happens right now.
Differential Revision: https://phab.mercurial-scm.org/D10608
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:28:05 +0200] rev 47227
revlog: pass a transaction object to `rewrite_sidedata`
The `_writing` context need one, so we update the function signature before considering using `_writing` in rewrite_sidedata.
Differential Revision: https://phab.mercurial-scm.org/D10607
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:27:53 +0200] rev 47226
revlog: rename variable in `rewrite_sidedata` to match other code
Let's call the index file object and `ifh` and the data file object `dfh` as the
rest of the revlog code. This will make future change clearer.
Differential Revision: https://phab.mercurial-scm.org/D10606
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:27:42 +0200] rev 47225
revlog: introduce a mandatory `_writing` context to update revlog content
Before this change, various revlog methods where managing the opening and
closing of the revlog files manually and passing the file descriptor alors the
call path. To simplify the tracking of the write operation by a future docket,
we need something more organised. As a result, we introduce a `revlog._writing`
context manager that will wrap each revlog update operation. The file
descriptor are kept in the existing `revlog._writinghandles` parameter that
was already used by the `addgroup` logic.
All this change is internal to the revlog only, the "public" interface is not
affected. The `addrevision` and `addgroup` logic are still responsible for
setup up this context. However this new context give us multiple benefits:
* all writer use a same, unified, logic,
* this context is programmatically enforced,
* each write "session" as a clearly identified start and end.
The post-pull sidedata update logic is still doing writing by end and will be
adjusted in a later changesets.
This change affect the concurrency checker test, because register the state of
the file in the transaction sooner in `addrevision` (about as early as what
`addgroup` would do), so the abort is rollbacking the other commit. I don't want
to weaken the current main logic.
Differential Revision: https://phab.mercurial-scm.org/D10605
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:27:31 +0200] rev 47224
revlog: preindent some code in _enforceinlinesize
Indenting this beforehand will make a future changeset much simpler.
Differential Revision: https://phab.mercurial-scm.org/D10604
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 May 2021 12:27:20 +0200] rev 47223
revlog: preindent some code in addgroup
Indenting this beforehand will make a future changeset much simpler.
Differential Revision: https://phab.mercurial-scm.org/D10603
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 09:27:22 -0700] rev 47222
pyoxidizer: use Python 3.9 (BC)
Newer versions of PyOxidizer use Python 3.9 by default. We
previously pinned the version to 3.8 to facilitate porting to
a new PyOxidizer version and diffing results.
Now that the porting work is complete, let's bump Python
to Python 3.9.
This will effectively change our Windows Inno and WiX Python 3
installers from Python 3.8 to 3.9.
.. bc::
Windows .msi and .exe installers now use Python 3.9 instead of
Python 3.8.
Differential Revision: https://phab.mercurial-scm.org/D10689
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:11:13 -0700] rev 47221
packaging: use PyOxidizer for producing WiX MSI installer
We recently taught our in-tree PyOxidizer configuration file to
produce MSI installers with WiX using PyOxidizer's built-in
support for doing so.
This commit changes our WiX + PyOxidizer installer generation
code to use this functionality.
After this change, all the Python packaging code is doing is the
following:
* Building HTML documentation
* Making gettext available to the build process.
* Munging CLI arguments to variables for the `pyoxidizer` execution.
* Invoking `pyoxidizer build`.
* Copying the produced `.msi` to the `dist/` directory.
Applying this stack on stable and rebuilding the 5.8 MSI installer
produced the following differences from the official 5.8 installer:
* .exe and .pyd files aren't byte identical (this is expected).
* Various .dist-info/ directories have different names due to older
versions of PyOxidizer being buggy and not properly normalizing
package names. (The new behavior is correct.)
* Various *.dist-info/RECORD files are different due to content
divergence of files (this is expected).
* The python38.dll differs due to newer PyOxidizer shipping a newer
version of Python 3.8.
* We now ship python3.dll because PyOxidizer now includes this file
by default.
* The vcruntime140.dll differs because newer PyOxidizer installs a newer
version. We also now ship a vcruntime140_1.dll because newer versions
of the redistributable ship 2 files now.
The WiX GUIDs and IDs of installed files have likely changed as a
result of PyOxidizer's different mechanism for generating those
identifiers. This means that an upgrade install of the MSI will
replace files instead of doing an incremental update. This is
likely harmless and we've incurred this kind of breakage before.
As far as I can tell, the new PyOxidizer-built MSI is functionally
equivalent to the old method. Once we drop support for Python 2.7
MSI installers, we can delete the WiX code from the repository.
This commit temporarily drops support for extra `.wxs` files. We
raise an exception instead of silently not using them, which I think
is appropriate. We should be able to add support back in by injecting
state into pyoxidizer.bzl via `--var`. I just didn't want to expend
cognitive load to think about the solution as part of this series.
Differential Revision: https://phab.mercurial-scm.org/D10688
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:07:01 -0700] rev 47220
packaging: extract invocation of pyoxidizer to own function
I'll be refactoring how the WiX installer creation calls into
pyoxidizer and will need a lower level function for facilitating
that.
The new `run_pyoxidizer()` builds our execution environment
(with gettext available) and invokes `pyoxidizer`.
Differential Revision: https://phab.mercurial-scm.org/D10687
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:06:20 -0700] rev 47219
packaging: rename run_pyoxidizer()
I'm going to split this function up in a future commit and I'll
want the name "run_pyoxidizer()" for a lower-level function for
invoking `pyoxidizer`.
Differential Revision: https://phab.mercurial-scm.org/D10686
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 08:37:40 -0700] rev 47218
packaging: move documentation HTML building to own function
This is part of some light refactoring to enable us to use
PyOxidizer for WiX MSI installer generation.
Differential Revision: https://phab.mercurial-scm.org/D10685
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:04:24 -0700] rev 47217
pyoxidizer: support code signing
Newer versions of PyOxidizer feature built-in support for
code signing. You simply declare a code signer in the Starlark
configuration file, activate it for automatic signing, and
PyOxidizer will add code signatures to signable files as it
encounters them.
This commit teaches our Starlark configuration file to enable
automatic code signing. But only on Windows for the moment, as our
immediate goal is to overhaul the Windows packaging.
The feature is opt-in: you must pass variables to PyOxidizer's
build context via `pyoxidizer build --var` or
`pyoxidizer build --var-env` to activate code signing.
Differential Revision: https://phab.mercurial-scm.org/D10684
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:03:43 -0700] rev 47216
pyoxidizer: support producing MSI installers
Newer versions of PyOxidizer have support for building WiX MSI
installers "natively." Essentially, you can script the definition
of your WiX installer via Starlark and PyOxidizer can invoke
WiX tools to produce the installer.
This commit teaches our PyOxidizer config file to produce MSI
installers similarly to how
`contrib/packaging/packging.py wix` would do it.
We had to make a very minor change to `mercurial.wxs` to reflect
different paths depending on who builds. This is because when
PyOxidizer builds WiX installers, it does so from an isolated
directory, not Mercurial's source directory. We simply copy the
files into the build environment so they are accessible.
After this change, running `pyoxidizer build msi` produces a nearly
identical install layout to what the previous method produces.
When I applied this series on top of the 5.8 tag, here is the
list of differences and explanations:
* docs/*.html files are missing from the new installer because the
Python build environment doesn't have docutils.
* .pyd and .exe files differ, likely because I'm using a different
Visual Studio toolchain on my local computer than the official build
environment.
* Various .dist-info/ directories have different names. This is
because older versions of PyOxidizer had buggy behavior and weren't
properly normalizing package names in .dist-info/ directories. e.g.
we went from `cached-property-1.5.2.dist-info` to
`cached_property-1.5.2.dist-info`.
* Translations (.mo files) may be missing if gettext isn't in %Path%.
This is because the packaging.py code installs gettext and ensures
it can be found.
* Some *.dist-info/RECORD files vary due to SHA-256 content digest
divergence due to build environment differences. (This should be
harmless.)
* The new install layout ships a python3.dll because newer versions
of PyOxidizer ship this file.
* The new install layout has a different vcruntime140.dll and also a
vcruntime140_1.dll because newer versions of PyOxidizer ship a
newer version of the Visual C++ Redistributable Runtime.
The new PyOxidizer functionality is not yet integrated with
packaging.py. This will come in a subsequent commit. So for now, the
new functionality introduced here is unused.
Differential Revision: https://phab.mercurial-scm.org/D10683
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:16:21 -0700] rev 47215
pyoxidizer: use allocator_backend instead of raw_allocator
The name of this attribute changed in PyOxidizer 0.11.
Differential Revision: https://phab.mercurial-scm.org/D10682
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 15:58:37 -0700] rev 47214
pyoxidizer: simplify targets
The split targets existed to enable the use of a non-default
distribution flavor on Windows. Modern versions of PyOxidizer
use the "standalone_dynamic" distribution flavor by default.
So our split brain workaround is no longer needed.
Here, we unify the targets. We also remove an unreferenced
target function to create a resources file.
Differential Revision: https://phab.mercurial-scm.org/D10681
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 15:56:04 -0700] rev 47213
contrib: install PyOxidizer 0.16.0
PyOxidizer now provides MSI installers and pre-built Linux binaries.
So we install that way. This significantly reduces the time to bootstrap
a new machine in automation, as building PyOxidizer from source on a low
core count machine takes several minutes.
This change temporarily breaks the ability of the automated environment
to use the in-repo pyoxidizer.bzl configuration file, as there are
backwards-incompatible changes with the upgrade that need to be
reflected. We'll handle those in subsequent commits.
Differential Revision: https://phab.mercurial-scm.org/D10680
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:13:33 -0700] rev 47212
pyoxidizer: pin Python to 3.8
Newer versions of PyOxidizer use version 3.9 by default. As part of
upgrading PyOxidizer, we want to pin the version at 3.8 so we can
compare differences more easily.
Differential Revision: https://phab.mercurial-scm.org/D10679
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 16:00:44 -0700] rev 47211
pyoxidizer: remove some boilerplate in file
This isn't used for anything and can safely be removed.
Differential Revision: https://phab.mercurial-scm.org/D10678
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 15:45:04 -0700] rev 47210
contrib: upgrade Rust in Windows automation
I'm not sure why we don't install the minimum required Rust version
here like we do for Linux. Whatever: that's unrelated to wanting to
stay modern.
Differential Revision: https://phab.mercurial-scm.org/D10677
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 15:44:29 -0700] rev 47209
contrib: update Python versions in Windows automation
Let's keep our Python versions modern.
Differential Revision: https://phab.mercurial-scm.org/D10676
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 15:41:52 -0700] rev 47208
automation: update rust in Linux environment
Our minimum supported Rust is 1.41.1 per rust/README.rst. We also
bump the modern Rust version to latest stable to stay current.
Differential Revision: https://phab.mercurial-scm.org/D10675
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 15:40:47 -0700] rev 47207
automation: install latest Python versions in Linux environment
Let's keep our Linux environment up to date.
Differential Revision: https://phab.mercurial-scm.org/D10674
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 May 2021 17:46:57 -0700] rev 47206
automation: create Python 3.5 variant of requirements.txt
The automation environment is refusing to build with the previous
file because some dependencies won't install on Python 3.5.
I couldn't find an easy way to salvage the situation with a single
requirements.txt file. So, I decided to introduce a variant
for Python 3.5.
As part of this, we update packages to latest versions.
(I do question why we are still supporting Python 3.5...)
Differential Revision: https://phab.mercurial-scm.org/D10690
Karthikeyan Singaravelan <tir.karthi@gmail.com> [Sat, 15 May 2021 09:45:10 +0000] rev 47205
hgweb: Fix deprecation warning in Python 3.10 (issue6520)
Differential Revision: https://phab.mercurial-scm.org/D10711
Simon Sapin <simon.sapin@octobus.net> [Mon, 10 May 2021 21:59:13 +0200] rev 47204
dirstate-tree: Remove newly-empty nodes after removing a `DirstateEntry`
This is actually necessary to make `DirstateMap::has_dir` correct, since it
assumes that a node without a `DirstateEntry` has at least one descedant node
with a `DirstateEntry`.
This bug would become apparent when a later changeset persists tree nodes
on disk in the "dirstate-v2" format.
Differential Revision: https://phab.mercurial-scm.org/D10706
Simon Sapin <simon.sapin@octobus.net> [Mon, 10 May 2021 21:31:05 +0200] rev 47203
dirstate-tree: Refactor DirstateMap::drop_file to be recursive
It should behave the same as before. This will enable the next changeset
to run code on the way "down" (in order to removing newly-empty nodes).
Differential Revision: https://phab.mercurial-scm.org/D10705
Martin von Zweigbergk <martinvonz@google.com> [Tue, 11 May 2021 12:22:26 -0700] rev 47202
templates: fix `revset('parents()') % ...` in amend message template
I don't understand why, but putting `revset('parents()') % {desc}` in
the commit message template for amend resulted in a crash because
`memctx.hex()` did `hex(self.node())` and its node was None. This
patch fixes that.
Martin von Zweigbergk <martinvonz@google.com>
Differential Revision: https://phab.mercurial-scm.org/D10707
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 15 Apr 2021 19:54:03 +0200] rev 47201
path: teach the `hg path` command to display boolean sub-option
The next changeset introduce one.
Differential Revision: https://phab.mercurial-scm.org/D10450