Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 31 Mar 2021 17:54:02 -0400] rev 46892
blackbox: fix type error on log rotation on read-only filesystem
Grepping around, the code uses either encoding.strtolocal or
stringutil.forcebytestr in this situation. No idea which is best.
Differential Revision: https://phab.mercurial-scm.org/D10293
Simon Sapin <simon.sapin@octobus.net> [Thu, 08 Apr 2021 14:38:27 +0200] rev 46891
rust: Remove use of `py.eval()`
The previous Rust code allocated an intermediate `Vec`, converted that
to a Python list, then used `eval` to run Python code that converts that
list to a Python set.
rust-cpython exposes Rust bindings for Python sets, let’s use that instead
to construct a set directly.
Differential Revision: https://phab.mercurial-scm.org/D10328
Simon Sapin <simon.sapin@octobus.net> [Thu, 08 Apr 2021 21:46:54 +0200] rev 46890
rust: Remove the compile-time 'dirstate-tree' feature flag
This code has compiler errors since it is not built on CI and nobody has been
working on it for some time.
We (Octobus) are still pursuing status optimizations based on a tree data
structure for the dirstate, but upcoming patches will use a run-time opt-in
instead of compile-time, so that at least corresponding Rust code keeps
compiling when other changes are made.
Differential Revision: https://phab.mercurial-scm.org/D10329
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sun, 13 Sep 2020 22:14:25 -0400] rev 46889
procutil: avoid using os.fork() to implement runbgcommand
We ran into the following deadlock:
- some command creates an ssh peer, then raises without explicitly
closing the peer (hg id + extension in our case)
- dispatch catches the exception, calls ui.log('commandfinish', ..)
(the sshpeer is still not closed), which calls logtoprocess, which
calls procutil.runbgcommand.
- in the child of runbgcommand's fork(), between the fork and the
exec, the opening of file descriptors triggers a gc which runs the
destructor for sshpeer, which waits on ssh's stderr being closed,
which never happens since ssh's stderr is held open by the parent of
the fork where said destructor hasn't run
Remotefilelog appears to have a hack around this deadlock as well.
I don't know if there's more subtlety to it, because even though the
problem is determistic, it is very fragile, so I didn't manage to
reduce it.
I can imagine three ways of tackling this problem:
1. don't run any python between fork and exec in runbgcommand
2. make the finalizer harmless after the fork
3. close the peer without relying on gc behavior
This commit goes with 1, as forking without exec'ing is tricky in
general in a language with gc finalizers. And maybe it's better in the
presence of rust threads. A future commit will try 2 or 3.
Performance wise: at low memory usage, it's an improvement. At higher
memory usage, it's about 2x faster than before when ensurestart=True,
but 2x slower when ensurestart=False. Not sure if that matters. The
reason for that last bit is that the subprocess.Popen always waits for
the execve to finish, and at high memory usage, execve is slow because
it deallocates the large page table. Numbers and script:
before after
mem=1.0GB, ensurestart=True 52.1ms 26.0ms
mem=1.0GB, ensurestart=False 14.7ms 26.0ms
mem=0.5GB, ensurestart=True 23.2ms 11.2ms
mem=0.5GB, ensurestart=False 6.2ms 11.3ms
mem=0.2GB, ensurestart=True 15.7ms 7.4ms
mem=0.2GB, ensurestart=False 4.3ms 8.1ms
mem=0.0GB, ensurestart=True 2.3ms 0.7ms
mem=0.0GB, ensurestart=False 0.8ms 0.8ms
import time
for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]:
mem = 'a' * memsize
for ensurestart in [True, False]:
now = time.time()
n = 100
for i in range(n):
procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart)
after = time.time()
ms = (after - now) / float(n) * 1000
print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms')
Differential Revision: https://phab.mercurial-scm.org/D9019
Matt Harbison <matt_harbison@yahoo.com> [Thu, 08 Apr 2021 18:43:08 -0400] rev 46888
share: store relative share paths with '/' separators
I created a relative share in Windows and tried to use it in WSL, and it failed:
abort: .hg/sharedpath points to nonexistent directory
/mnt/c/Users/Matt/hg-review/.hg/..\..\hg\.hg
Use `normpath` on the read side so that the code has the usual Windows style
paths it always had (I don't think that matters much), but it also eliminates
the directory escaping path components in the case where the path is printed.
This will not fix repositories that have already been created, but it's trivial
enough to hand edit the file to correct it.
Differential Revision: https://phab.mercurial-scm.org/D10330
Simon Sapin <simon.sapin@octobus.net> [Fri, 09 Apr 2021 12:02:51 +0200] rev 46887
unit-tests: Fix `cargo test` on 32-bit platforms
Fixes https://bz.mercurial-scm.org/show_bug.cgi?id=6506
This makes `IndexEntryBuilder::build`, which is only used in unit tests, use
`u32` or `u64` instead of platform-dependent `usize` when packing binary data
to be used at test input.
To run Rust unit tests in 32-bit mode in a x86-64 environment, use:
rustup target add i686-unknown-linux-gnu # Once
(cd rust && cargo test --target i686-unknown-linux-gnu)
Differential Revision: https://phab.mercurial-scm.org/D10351
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Apr 2021 08:46:40 -0700] rev 46886
rename: add --forget option and stop suggesting `hg revert` for undoing
Differential Revision: https://phab.mercurial-scm.org/D10355
Matt Harbison <matt_harbison@yahoo.com> [Fri, 09 Apr 2021 11:32:19 -0400] rev 46885
win32: enable legacy I/O mode to fix missing pager output on Windows with py3
The equivalent interpreter option is set by wrapper.exe, but this *.bat file is
what gets installed in a venv. Without this mode, any command that spins up a
pager has no output, unless the pager is explicitly disabled. The variable is
set inside the `setlocal` scope to keep it from leaking into the environment
after the bat file exits.
We should probably still figure out how to ship a compiled hg.exe when
installing with `pip`, because the binary does other things like enable long
filename support. But this avoids the dangerous and confusing lack of output in
the meantime.
Differential Revision: https://phab.mercurial-scm.org/D10354
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 14 Jan 2021 04:58:20 +0100] rev 46884
persistent-nodemap: enable the feature by default when using Rust
As discussed at the 5.6 sprint, we can make it enabled by default, but only for
Rust installation.
Differential Revision: https://phab.mercurial-scm.org/D9765
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 03:36:24 +0200] rev 46883
persistent-nodemap: disable it for test-wireproto-content-redirects.t
This is not relevant for the test and this will avoid a lot of variations.
Differential Revision: https://phab.mercurial-scm.org/D10338
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 03:30:18 +0200] rev 46882
persistent-nodemap: disable it for test-wireproto-command-capabilities.t
This is not relevant for the test and this will avoid a lot of variations.
Differential Revision: https://phab.mercurial-scm.org/D10337
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 03:36:07 +0200] rev 46881
persistent-nodemap: disable it unconditionally for test-wireproto-caching.t
This is not relevant for the test and this will avoid a lot of variations.
Differential Revision: https://phab.mercurial-scm.org/D10336
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 03:51:25 +0200] rev 46880
persistent-nodemap: disable it unconditionally for test-ssh-proto-unbundle.t
This is not relevant for the test and this will avoid a lot of variations.
Differential Revision: https://phab.mercurial-scm.org/D10335
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 03:51:35 +0200] rev 46879
persistent-nodemap: disable it unconditionally for test-ssh-proto.t
This is not relevant for the test and this will avoid a lot of variations.
Differential Revision: https://phab.mercurial-scm.org/D10334
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 04:13:07 +0200] rev 46878
persistent-nodemap: disable it unconditionally for test-share-safe.t
This is not relevant for the test and this will avoid a lot of variations.
Differential Revision: https://phab.mercurial-scm.org/D10333
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 03:44:45 +0200] rev 46877
persistent-nodemap: disable it unconditionally for test-http-protocol.t
This is not relevant for the test and this will avoid a lot of variations.
Differential Revision: https://phab.mercurial-scm.org/D10332
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 09 Apr 2021 03:15:07 +0200] rev 46876
persistent-nodemap: disable it unconditionally for test-http-bad-server.t
This is not relevant for the test and this will avoid a lot of variations
Differential Revision: https://phab.mercurial-scm.org/D10331
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 08 Apr 2021 20:05:59 +0200] rev 46875
clang-format: run the formatter on mercurial/cext/revlog.c
This fix `test-check-clang-format.t` that has been complaining for a while.
Differential Revision: https://phab.mercurial-scm.org/D10327
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Apr 2021 18:55:19 +0200] rev 46874
revlog-compression: use zstd by default (if available)
As see in changeset
bb271ec2fbfb, zstd is 20% to 50% faster for reading and
writing.
Use take advantage of the new config behavior to try zstd by default, falling
back to zlib is zstd is not available on that plateform.
Differential Revision: https://phab.mercurial-scm.org/D10326
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 07 Apr 2021 13:31:15 +0200] rev 46873
rhg: make rhg recognise it supports zstd compression for revlogs
It already did, but was not aware of it.
Differential Revision: https://phab.mercurial-scm.org/D10324
Kyle Lippincott <spectral@google.com> [Wed, 31 Mar 2021 12:46:54 -0700] rev 46872
match: convert O(n) to O(log n) in exactmatcher.visitchildrenset
When using narrow, during rebase this is called (at least) once per directory in
the set of files in the commit being rebased. Every time it's called, we did the
set arithmetic (now extracted and cached), which was probably pretty cheap but
not necessary to repeat each time, looped over every item in the matcher and
kept things that started with the directory we were querying.
With very large narrowspecs, and a commit that touched a file in a large number
of directories, this was slow. In a pathological repo, the rebase of a single
commit (that touched over 17k files, I believe in approximately as many
directories) with a narrowspec that had >32k entries took 8,246s of profiled
time, with 5,007s of that spent in visitchildrenset (transitively). With this
change, the time spent in visitchildrenset is less than 34s (which is where my
profile cut off). Most of the remaining time was network access due to our
custom remotefilelog-based setup not properly prefetching.
Differential Revision: https://phab.mercurial-scm.org/D10294
Kyle Lippincott <spectral@google.com> [Tue, 30 Mar 2021 13:05:22 -0700] rev 46871
exthelper: improve docs to indicate what module vars are needed
I recently tried creating an extension "from scratch" using exthelper, and it
wasn't obvious that you needed these. I believe that a careful reading of one of
the comments would tell you that they were required, but it's easy to miss and
having the examples be "complete" is helpful.
Differential Revision: https://phab.mercurial-scm.org/D10295
Matt Harbison <matt_harbison@yahoo.com> [Mon, 05 Apr 2021 12:44:33 -0400] rev 46870
tests: restore the ability to run `black` on Windows
The hghave test for black silently stopped working with the change in
08fd76a553c9. This was the output of what it was hitting when run in the test
environment:
Traceback (most recent call last):\r (esc)
File "c:\\users\\matt\\appdata\\local\\programs\\python\\python39\\lib\\runpy.py", line 197, in _run_module_as_main\r (esc)
return _run_code(code, main_globals, None,\r (esc)
File "c:\\users\\matt\\appdata\\local\\programs\\python\\python39\\lib\\runpy.py", line 87, in _run_code\r (esc)
exec(code, run_globals)\r (esc)
File "c:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\Scripts\\black.exe\\__main__.py", line 4, in <module>\r (esc)
File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\black\\__init__.py", line 70, in <module>\r (esc)
CACHE_DIR = Path(user_cache_dir("black", version=__version__))\r (esc)
File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\appdirs.py", line 293, in user_cache_dir\r (esc)
path = os.path.normpath(_get_win_folder("CSIDL_LOCAL_APPDATA"))\r (esc)
File "C:\\Users\\Matt\\AppData\\Roaming\\Python\\Python39\\site-packages\\appdirs.py", line 481, in _get_win_folder_with_pywin32\r (esc)
dir = shell.SHGetFolderPath(0, getattr(shellcon, csidl_name), 0, 0)\r (esc)
pywintypes.com_error: (-
2147024893, '$ENOTDIR$.', None, None)\r (esc)
[1]
Differential Revision: https://phab.mercurial-scm.org/D10310
Matt Harbison <matt_harbison@yahoo.com> [Sat, 03 Apr 2021 21:15:45 -0400] rev 46869
setup: copy python3.dll next to hg.exe when building on Windows for hgext.git
I thought I took care of this already, but it must have been that I just
manually copied the file over locally when debugging why the pygit2 library
wasn't loading. The problem with that is what was copied over was from py38,
and then running a py39 build hard crashed when the extension was loaded.
Differential Revision: https://phab.mercurial-scm.org/D10301
Matt Harbison <matt_harbison@yahoo.com> [Sat, 03 Apr 2021 20:26:45 -0400] rev 46868
tests: stablize test-hook.t on Windows
Apparently, hooks can't run `echo` directly, even from MSYS.
Differential Revision: https://phab.mercurial-scm.org/D10300
Matt Harbison <matt_harbison@yahoo.com> [Sat, 03 Apr 2021 20:25:37 -0400] rev 46867
tests: handle Windows file separator differences in test-config.t
Differential Revision: https://phab.mercurial-scm.org/D10299
Matt Harbison <matt_harbison@yahoo.com> [Sat, 03 Apr 2021 15:02:09 -0400] rev 46866
tests: update the detailed exit codes for icasefs gated tests
The fact that there's already a detailed exit code after the last change here in
test-casecollision-merge.t gives me some pause, but maybe it was found and
changed manually?
Differential Revision: https://phab.mercurial-scm.org/D10298
Matt Harbison <matt_harbison@yahoo.com> [Sat, 03 Apr 2021 15:00:20 -0400] rev 46865
tests: update the detailed exit codes in test-phabricator.t
I'm guessing this has slipped through because most people don't have the
necessary `pytest-vcr` package installed.
Differential Revision: https://phab.mercurial-scm.org/D10297
Martin von Zweigbergk <martinvonz@google.com> [Thu, 25 Mar 2021 15:38:31 -0700] rev 46864
tests: avoid using rebaseskipobsolete=0
I'm about to delete the `rebaseskipobsolete` config. This patch
updates a test to get hidden commits into the rebase state by using
`hg debugobsolete` instead of setting `rebaseskipobsolete=0`.
Differential Revision: https://phab.mercurial-scm.org/D10270
Martin von Zweigbergk <martinvonz@google.com> [Thu, 25 Mar 2021 11:35:16 -0700] rev 46863
rebase: when using --keep, don't care about pruned commits or divergence
`hg rebase --keep` creates duplicate commits (not successors), so I
was surprised that it still skips pruned commits and errors out if it
"would cause divergence" (it wouldn't). I guess this was just an
oversight. We didn't have any tests for it, so I also included that.
Differential Revision: https://phab.mercurial-scm.org/D10269
Martin von Zweigbergk <martinvonz@google.com> [Thu, 25 Mar 2021 08:38:16 -0700] rev 46862
rebase: remove duplicate initialization of a field
`obsolete_with_successor_in_destination` is already initialized in the
constructor.
Differential Revision: https://phab.mercurial-scm.org/D10268
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Apr 2021 12:22:12 +0200] rev 46861
rust: bump rust-cpython version to 0.5.2
we need a newer version to define "property" on Rust defined object.
Differential Revision: https://phab.mercurial-scm.org/D10308
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Apr 2021 12:21:58 +0200] rev 46860
revlog: directly use the Struct object for related operation
The Struct object has all the piece we needs, so no need to duplicate
information on the revlog itself.
Differential Revision: https://phab.mercurial-scm.org/D10307
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Apr 2021 12:21:23 +0200] rev 46859
revlog: move the "index header" struct inside revlog.utils.constants
The struct was previous called "version", but this is actually "version" +
"flags". So header seems like a better name.
The move to the `constants` module has the same motivation as the INDEX_ENTRY_V#
ones.
Differential Revision: https://phab.mercurial-scm.org/D10306
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Apr 2021 12:21:12 +0200] rev 46858
revlog: move the details of revlog "v2" index inside revlog.utils.constants
the revlog module is quite large and this kind of format information would handy
for other module. So let us start to gather this information about the format in
a more appropriate place.
We update various reference to this information to use the new "source of truth"
in the process.
Differential Revision: https://phab.mercurial-scm.org/D10305
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Apr 2021 12:21:01 +0200] rev 46857
revlog: move the details of revlog "v1" index inside revlog.utils.constants
The revlog module is quite large and this kind of format information would handy
for other module. So let us start to gather this information about the format in
a more appropriate place.
We update various reference to this information to use the new "source of truth"
in the process.
Differential Revision: https://phab.mercurial-scm.org/D10304
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Apr 2021 12:20:52 +0200] rev 46856
revlog: move the details of revlog "v0" index inside revlog.utils.constants
the revlog module is quite large and this kind of format information would handy
for other module. So let us start to gather this information about the format in
a more appropriate place.
Differential Revision: https://phab.mercurial-scm.org/D10303
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 05 Apr 2021 12:20:40 +0200] rev 46855
revlog: add some comment in the header sections
We are about to add more content so let us organise the existing content first.
Differential Revision: https://phab.mercurial-scm.org/D10302
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Apr 2021 10:37:55 +0200] rev 46854
store: drop the `filefilter` argument to `_walk`
No code use it anywhere. Dropping it will help replacing the function with
something with a more precise semantic.
Differential Revision: https://phab.mercurial-scm.org/D10314
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Apr 2021 10:37:47 +0200] rev 46853
store: document the `walk` method
Differential Revision: https://phab.mercurial-scm.org/D10313
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Tue, 06 Apr 2021 13:49:19 -0400] rev 46852
revlog: fix error about unknown compression format in py3
In py2, the error is something like:
abort: unknown compression type 'x'!
In py3, we get the following unhelpful message:
abort: unknown compression type <memory at 0x
7f4650b5cdc8>!
Switch to something like:
abort: unknown compression type 78!
Differential Revision: https://phab.mercurial-scm.org/D10318
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 07 Apr 2021 12:15:28 +0200] rev 46851
revlog-compression: fix computation of engine availability
We don't just need the engine to be define, we need it to be available and able
to do be used for revlog compression. Without this change, `zstd` could be
selected as a viable option for repository creation on platform where it is not
available.
Differential Revision: https://phab.mercurial-scm.org/D10325
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 07 Apr 2021 00:12:07 +0200] rev 46850
test: explicitly use zlib compression in tests/test-share-safe.t
We need the implicit value to be explicit until we can change the default in
some case.
Differential Revision: https://phab.mercurial-scm.org/D10323
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Apr 2021 18:50:32 +0200] rev 46849
test: explicitly use zlib compression in tests/test-upgrade-repo.t
We need the implicit value to be explicit until we can change the default in
some case.
Differential Revision: https://phab.mercurial-scm.org/D10322
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 06 Apr 2021 18:49:01 +0200] rev 46848
test: explicitly use zlib compression in tests/test-repo-compengines.t
We need the implicit value to be explicit until we can change the default in
some case.
Differential Revision: https://phab.mercurial-scm.org/D10321
Kyle Lippincott <spectral@google.com> [Tue, 06 Apr 2021 14:21:03 -0700] rev 46847
remotefilelog: include file contents in bundles produced during strip
`hg strip` and other things that use repair.strip (such as the narrow
extension's `hg tracked --removeinclude`) will "save" some commits that have a
higher revision number than the oldest commit we're stripping, but aren't
actually descended from any of the commits that we're stripping. It saves them
in a bundle, and then reapplies them to the repo.
Remotefilelog doesn't generally participate in strip, it doesn't contribute
files to either the backup bundle or the "saved" bundle, and doesn't adjust
linknodes when commits are stripped. This can break things like push, which
rely on the linknodes.
This change makes it so that remotefilelog includes files in these bundles
during strip operations. During reapplication, the files are reapplied from the
bundle, and the linknode is properly updated.
Differential Revision: https://phab.mercurial-scm.org/D10320
Kyle Lippincott <spectral@google.com> [Tue, 06 Apr 2021 15:38:33 -0700] rev 46846
tests: add test-remotefilelog-strip.t to demonstrate an issue with linknodes
### Background
Every time a commit is modified, remotefilelog updates the metadata for the file
object to point to the new commit (I believe that this is different from
non-remotefilelog hg, which leaves the linkrevs pointing to the obsolete
commits; doing otherwise would involve changing data in the middle of revlogs).
With `hg strip` (or other things that use repair.strip()), when you strip a
commit that's not the tip of the revlog, there may be commits after it in revnum
order that aren't descended from it and don't need to be (and shouldn't be)
stripped. These are "saved" by strip in a bundle, and that bundle is reapplied
after truncating the relevant revlogs.
### The problem
Remotefilelog generally avoids being involved at all in strip. Currently, that
includes even providing file contents to this backup bundle. This can cause the
linknode to point to a changeset that is no longer in the repository.
Example:
```
@ 3
df91f74b871e
|
| x 2
70494d7ec5ef
|/
| x 1
1e423846dde0
|/
o 0
b292c1e3311f
```
Commits 1, 2, and 3 are related via obsolescence, and are description-only
changes. The linknode for the file in these commits changed each time we updated
the description, so it's currently df91f7. If I strip commits 1 and 3, however,
the linknode *remains* df91f7, which no longer exists in the repository. Commit
70494d was "saved", stripped, and then reapplied, so it is in the repository (as
revision 1 instead of 2 now), and was unobsoleted since the obsmarker was
stripped as well. The linknode for the file should point to 70494d, the most
recent commit that is in the repository that modified the file.
Remotefilelog has some logic to handle broken linknodes, but it can be slow. We
have actually disabled it internally because it's too slow for our purposes.
Differential Revision: https://phab.mercurial-scm.org/D10319
Joerg Sonnenberger <joerg@bec.de> [Wed, 31 Mar 2021 00:19:52 +0200] rev 46845
mergestate: remove unused import
Differential Revision: https://phab.mercurial-scm.org/D10291