Matt Harbison <matt_harbison@yahoo.com> [Wed, 04 Mar 2020 10:25:07 -0500] rev 44448
phabricator: make `hg phabread` work outside of a repository
This is similar to 16312ea45a8b and 2513f0f70a26- we don't need a repo, but will
load .hg/hgrc if inside one.
Differential Revision: https://phab.mercurial-scm.org/D8214
Matt Harbison <matt_harbison@yahoo.com> [Sun, 16 Feb 2020 15:06:20 -0500] rev 44447
phabricator: refactor `phabread` to write all patches at once
This will be necessary to create a first class `phabimport` command. That
command requires a transaction, and will import all named patches within a
single transaction. But if Phabricator queries also happen within the
transaction, that leaves open the chance that an exception is raised, the
transaction is abandoned, and the next command that is run will complain about
needing to run `hg recover`.
Differential Revision: https://phab.mercurial-scm.org/D8135
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Feb 2020 13:14:44 -0500] rev 44446
phabricator: make `hg phabupdate` work outside of a repository
This is similar to 16312ea45a8b- we don't need a repo, but will load .hg/hgrc if
inside one.
Differential Revision: https://phab.mercurial-scm.org/D8208
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Feb 2020 13:28:40 -0500] rev 44445
phabricator: pass ui instead of repo to `userphids()`
Also not a repository operation.
Differential Revision: https://phab.mercurial-scm.org/D8207
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Feb 2020 13:08:01 -0500] rev 44444
phabricator: pass ui instead of repo to `querydrev()`
Also not a repository operation.
Differential Revision: https://phab.mercurial-scm.org/D8206
Matt Harbison <matt_harbison@yahoo.com> [Mon, 17 Feb 2020 13:01:16 -0500] rev 44443
phabricator: pass ui instead of repo to `readpatch()`
This makes it a little clearer that it isn't a repository operation.
Differential Revision: https://phab.mercurial-scm.org/D8205
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Mon, 02 Mar 2020 14:36:20 -0500] rev 44442
logtoprocess: avoid traceback when running long commands
$ hg log -r "present($(yes | tr -d '\n' | head -c 130000))" "$(yes | tr -d '\n' | head -c 5000)" --config extensions.logtoprocess= --config logtoprocess.commandfinish=whatever
Traceback (most recent call last):
File "/usr/bin/hg", line 67, in <module>
dispatch.run()
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 111, in run
status = dispatch(req)
File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 290, in dispatch
canonical_command=req.canonical_command,
File "/usr/lib64/python2.7/site-packages/mercurial/ui.py", line 1991, in log
logger.log(self, event, msg, opts)
File "/usr/lib64/python2.7/site-packages/hgext/logtoprocess.py", line 72, in log
procutil.runbgcommand(script, fullenv, shell=True)
File "/usr/lib64/python2.7/site-packages/mercurial/utils/procutil.py", line 597, in runbgcommand
b'error running %r: %s' % (cmd, os.strerror(returncode)),
OSError: [Errno 7] error running 'whatever': Argument list too long
This can happen if you pass a bunch of filenames to hg commit, for instance.
This is due to a size limit on individual env vars (on linux, but I
imagine there are similar limits in other OSes):
$ FOO=$(yes | head -c 131000) /usr/bin/true
$ FOO=$(yes | head -c 132000) /usr/bin/true
-bash: /usr/bin/true: Argument list too long
I propose to avoid this by truncating the message. I didn't make the
limit configurable as it doesn't seem particularly convenient to
customize this. I'm not sure if various OSes would want radically
different limits here?
Differential Revision: https://phab.mercurial-scm.org/D8203
Raphaël Gomès <rgomes@octobus.net> [Mon, 24 Feb 2020 17:57:57 +0100] rev 44441
rust-cpython: make `NonNormalEntires` iterable to fix `fsmonitor` (issue6276)
This fixes a bug when using `fsmonitor` that tries to iterate on the non normal
set, by adding a shared iterator interface.
Differential Revision: https://phab.mercurial-scm.org/D8143
Joerg Sonnenberger <joerg@bec.de> [Sat, 07 Sep 2019 14:50:39 +0200] rev 44440
hgext: start building a library for simple hooks
Many workflows depend on hooks to enforce certain policies, e.g. to
prevent forced pushes. The Mercurial Guide includes some cases and
Google can help finding others, but it can save users a lot of time
if hg itself has a couple of examples for further customization.
Differential Revision: https://phab.mercurial-scm.org/D6825
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Tue, 25 Feb 2020 20:27:39 -0500] rev 44439
exchange: turn on option that makes concurrent pushes work better
The motivation is simply to make hg work better out of the box.
This is a slight backwards compatibility break, because client
extensions could have assumed that the list of heads the client sees
during discovery will be the list of heads during the entirety of the
push. It seems unlikely to matter, and not worth mentioning.
There's a fair amount of diff in tests, but this is just due to
sending a few more bytes on the wire, except for test-acl.t.
The extra "invalid branch cache" lines in test-acl.t don't seem to
indicate a problem: the branchcache now get computed during the bundle
application (because of the check:updated-heads bundle part), but
doesn't get rolled back when transactions rollback, thus causing a
message in the next operation computing the branch cache. Before this
change, I assume the branchcache was only computed on transaction
commit, so not computed at all when the transactions roll back, thus
no messages.
Differential Revision: https://phab.mercurial-scm.org/D8202
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Mon, 02 Mar 2020 15:34:51 -0500] rev 44438
update: simplify slightly
Differential Revision: https://phab.mercurial-scm.org/D8204
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sun, 01 Mar 2020 21:16:45 -0500] rev 44437
help: clarify behavior of server.concurrent-push-mode
So it doesn't seemingly say that old clients cannot talk to server
configured with concurrent-push-mode=check-related. They can, they
just don't get the benefit.
Differential Revision: https://phab.mercurial-scm.org/D8201
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Feb 2020 10:30:18 -0800] rev 44436
commit: error out on unresolved files even if commit would be empty
Differential Revision: https://phab.mercurial-scm.org/D8195
Martin von Zweigbergk <martinvonz@google.com> [Fri, 28 Feb 2020 10:49:13 -0800] rev 44435
tests: add test of committing with conflicts but no changes in wdir
I'm about to change the behavior slightly here, so let's have a test
that shows that.
Differential Revision: https://phab.mercurial-scm.org/D8194
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 28 Feb 2020 00:17:26 +0100] rev 44434
transaction: clarify the logic around pre-finalize/post-finalize
I am taking a bit more verbose route, but I find it easier to follow for people
who (re)discover the code.
(This is a gratuitous cleanup I did while looking at something else.)
Differential Revision: https://phab.mercurial-scm.org/D8176
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 28 Feb 2020 00:02:03 +0100] rev 44433
transaction: move constant to upper case
These constant are internal to the module and can be safely renamed. Having them
upper case help to clarify their "constant" status.
(This is a gratuitous cleanup I did while looking at something else.)
Differential Revision: https://phab.mercurial-scm.org/D8175
Joerg Sonnenberger <joerg@bec.de> [Wed, 26 Feb 2020 22:26:28 +0100] rev 44432
tests: handle In-Reply-To headers for line wrapping
Python 3 tends to insert a newline for both Message-ID and In-Reply-To
headers, so unwrap both. Just check the wrapped line format explicitly
without regular expression.
Differential Revision: https://phab.mercurial-scm.org/D8171
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Feb 2020 10:48:56 -0800] rev 44431
tests: use new, use-case-specific methods from merge module
Differential Revision: https://phab.mercurial-scm.org/D8169
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Feb 2020 10:40:31 -0800] rev 44430
merge: introduce a merge() for that use-case
In the same vein as some earlier patches like f546d2170b0f (merge:
introduce a clean_update() for that use-case, 2020-01-15).
Differential Revision: https://phab.mercurial-scm.org/D8168
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Feb 2020 11:00:50 -0800] rev 44429
merge: drop redundant mergeforce argument from hg.merge()
The only caller that passed a value for either `force` or `mergeforce`
passed the same value for both, so let's simplify the interface by
accepting only `force`.
Differential Revision: https://phab.mercurial-scm.org/D8167
Martin von Zweigbergk <martinvonz@google.com> [Wed, 26 Feb 2020 10:54:17 -0800] rev 44428
merge: change default of hg.merge()'s "force" argument from None to False
The argument is only passed to `mergemode.update()`, and that function
treats `None` just like `False`, so `False` seems clearer.
Differential Revision: https://phab.mercurial-scm.org/D8166
Martin von Zweigbergk <martinvonz@google.com> [Thu, 13 Feb 2020 21:14:20 -0800] rev 44427
debugmergestate: make templated
Our IntelliJ team wants to be able to read the merge state in order to
help the user resolve merge conflicts. They had so far been reading
file contents from p1() and p2() and their merge base. That is not
ideal for several reasons (merge base is not necessarily the "graft
base", renames are not handled, commands like `hg update -m` is not
handled). It will get especially bad as of my D7827. This patch makes
the output s a templated. I haven't bothered to make it complete
(e.g. merge driver states are not handled), but it's probably good
enough as a start.
I've done a web search for "debugmergestate" and I can't find any
indication that any tools currently rely on its output. If it turns
out that we get bug reports for it once this is released, I won't
object to backing this patch out on the stable branch (and then
perhaps replace it by a separate command, or put it behind a new
flag).
The changes in test-backout.t are interesting, in particular this:
```
- other path: foo (node not stored in v1 format)
+ other path: (node foo)
```
I wonder if that means that we actually read v1 format
incorrectly. That seems to be an old format that was switched away
from in 2014, so it doesn't matter now anyway.
Differential Revision: https://phab.mercurial-scm.org/D8120
Martin von Zweigbergk <martinvonz@google.com> [Thu, 13 Feb 2020 21:55:38 -0800] rev 44426
tests: add tests of debugmergestate with unresolved/resolved path conflicts
I'm about to change `hg debugmergestate` and it broke on these "pu"
and "pr" records on my first attempt (D8113), so let's add test
coverage.
Differential Revision: https://phab.mercurial-scm.org/D8119
Martin von Zweigbergk <martinvonz@google.com> [Thu, 13 Feb 2020 17:15:08 -0800] rev 44425
mergestate: determine if active without looking for state files on disk
I couldn't think of a reason that we need to check state files on disk
to determine if a merge is active. I could imagine them being for
there for detecting broken state files that would then be cleaned up
by some later command, but we always delete the entire `.hg/merge/`
tree, so that doesn't seem to be it.
The checks were added in 4e932dc5c113 (resolve: abort when not
applicable (BC), 2014-04-18). Perhaps there were needed for that and
then made obsolete by 6062593d8b06 (resolve: don't abort resolve -l
even when no merge is in progress, 2014-05-23).
The reason I want to delete the checks is that I think `ms =
mergestate.read(repo); ms.active() and ms.local` should be a valid
pattern, but it crashes when the merge state file is an empty file if
we consider mere presence of the file as "active".
Differential Revision: https://phab.mercurial-scm.org/D8118
Matt Harbison <matt_harbison@yahoo.com> [Wed, 26 Feb 2020 14:43:02 -0500] rev 44424
phabricator: update the protocol documentation
The `branch` property wasn't added to the `hg:meta` example when it was added to
the metadata in d49ab47be8a6. Additionally, `properties` in the Differential
Revision dict is a dinctionary, not a list. While here, also alphabetize the
responses from Phabricator because that's how it is being printed with
`hg debugcallconduit`, and this makes it easier to compare.
Differential Revision: https://phab.mercurial-scm.org/D8170
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 26 Feb 2020 10:56:27 -0500] rev 44423
relnotes: move entry to the right spot
It appears a conflict resolution went wrong.
Differential Revision: https://phab.mercurial-scm.org/D8151
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 26 Feb 2020 17:16:25 +0100] rev 44422
revlog-compression: release note entry for update the config to be a list
I updated the changeset, but forgot to phabsend apparently.
Differential Revision: https://phab.mercurial-scm.org/D8165
Georges Racinet <georges.racinet@octobus.net> [Tue, 18 Feb 2020 19:11:18 +0100] rev 44421
rust-nodemap: a method for full invalidation
This will be used for exceptional operations,
such as a `__delitem__` on the `MixedIndex` with
Rust nodemap.
In principle, `NodeTree` should also be able to forget
an entry in an efficient way, by accepting to insert
`Element::None` instead of only `Element::Rev(r)`,
but that seems really overkill at this point. We need
to support exceptional operations such as `__delitem__`,
only for completeness of the revlog index as seen from
Python. The Python callers don't seem to even really
need it, deciding to drop the nodemap unconditionally at
at higher level when calling `hg strip`. Also, `hg strip`
is very costly for reasons that are unrelated to nodemap
aspects.
Differential Revision: https://phab.mercurial-scm.org/D8098
Georges Racinet <georges.racinet@octobus.net> [Tue, 18 Feb 2020 19:11:17 +0100] rev 44420
rust-nodemap: accounting for dead blocks
By the very append-only nature of the `NodeTree`, inserting
new blocks has the effect of making some of the older ones
useless as they become unreachable.
Therefore some automatic housekeeping will need to be provided.
This is standard procedure in the word of databases, under names
such as "repack" or "vacuum".
The new `masked_readonly_blocks()` will provide callers with
useful information to decide if the nodetree is ripe for
repacking, but all the `NodeTree` can provide is how many
blocks have been masked in the currently mutable part. Analysing
the readonly part would be way too long to do it for each
transaction and defeat the whole purpose of nodemap persistence.
Serializing callers (from the Python layer) will get this figure
before each extraction and maintain an aggregate counter of
unreachable blocks separately.
Note: at this point, the most efficient repacking is just to restart
afresh with a full rescan.
Differential Revision: https://phab.mercurial-scm.org/D8097
Georges Racinet <georges.racinet@octobus.net> [Tue, 18 Feb 2020 19:11:17 +0100] rev 44419
rust-nodemap: core implementation for shortest
In this implementation, we just make `lookup()` return also the number
of steps that have been needed to come to a conclusion from the
nodetree data, and `validate_candidate()` takes care of the special
cases related to `NULL_NODE`.
This way of doing minimizes code duplication, but it means that
the comparatively slower finding of first non zero nybble will run
for all calls to `find()` where it is not needed.
Still running on the file generated for the mozilla-central repository,
it seems indeed that we now get more ofter 320 ns than 310. The odds that
this could have a significant impact on real life Mercurial performance
are still looking low. Let's wait for actual benchmark runs to see if
an optimization is needed here.
Differential Revision: https://phab.mercurial-scm.org/D7819
Georges Racinet <georges.racinet@octobus.net> [Tue, 18 Feb 2020 19:11:16 +0100] rev 44418
rust-nodemap: special case for prefixes of NULL_NODE
We have to behave as though NULL_NODE was stored in the node tree,
although we don't store it.
Differential Revision: https://phab.mercurial-scm.org/D7798
Georges Racinet <georges.racinet@octobus.net> [Tue, 18 Feb 2020 19:11:15 +0100] rev 44417
rust-nodemap: pure Rust example
To run, use `cargo run --release --example nodemap`
This demonstrates that simple scenarios entirely written
in Rust can content themselves with `NodeTree<T>`.
The example mmaps both the nodemap file and the changelog index.
We had of course to include an implementation of `RevlogIndex`
directly, which isn't much at this stage. It felt a bit
prematurate to include it in the lib.
Here are some first performance measurements, obtained with
this example, on a clone of mozilla-central with 440000
changesets:
(create) Nodemap constructed in RAM in 153.638305ms
(query CAE63161B68962) found in 22.362us: Ok(Some(269489))
(bench) Did 3 queries in 36.418µs (mean 12.139µs)
(bench) Did 50 queries in 184.318µs (mean 3.686µs)
(bench) Did 100000 queries in 31.053461ms (mean 310ns)
To be fair, even between bench runs, results tend to depend whether
the file is still in kernel caches, and it's not so easy to
get back to a real cold start. The worst we've seen was in the
50us ballpark.
In any busy server setting, the pages would always be in RAM.
We hope it's good enough not to be significantly slower on any
concrete Mercurial operation than the C nodetree when fully in RAM,
and of course this implementation has the serious headstart advantage
of persistence.
Differential Revision: https://phab.mercurial-scm.org/D7797
Georges Racinet <georges.racinet@octobus.net> [Tue, 18 Feb 2020 19:11:15 +0100] rev 44416
rust-nodemap: input/output primitives
These allow to initiate a `NodeTree` from an immutable opaque
sequence of bytes, which could be passed over from Python
(extracted from a `PyBuffer`) or directly mmapped from a file.
Conversely, we can consume
a `NodeTree`, extracting the bytes that express what
has been added to the immutable part, together with the
original immutable part.
This gives callers the choice to start a new Nodetree.
After writing to disk, some would prefer to reread for
best guarantees (very cheap if mmapping), some others will
find it more convenient to grow the memory that was considered
immutable in the `NodeTree` and continue from there.
This is enough to build examples running on real data and
start gathering performance hints.
Differential Revision: https://phab.mercurial-scm.org/D7796
Martin von Zweigbergk <martinvonz@google.com> [Thu, 13 Feb 2020 15:33:36 -0800] rev 44415
pyoxidizer: allow extensions to be loaded from the file system
It seems that setting this config is all that's needed to be able to
load extensions from the file system (which we clearly want). Thanks
for making this work, Gregory Szorc!.
Differential Revision: https://phab.mercurial-scm.org/D8122
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Mon, 17 Feb 2020 20:30:03 -0500] rev 44414
graft: always allow hg graft --base . (issue6248)
`hg graft --base . -r abc` is rejected before this change with a
"nothing to merge" error, if `abc` does not descend from `.`.
This looks like an artifact of the implementation rather than intended
behavior. It makes perfect sense to apply the diff between `.` and
`abc` to the working copy (i.e. degenerate into `hg revert`),
regardless of what `abc` is.
Differential Revision: https://phab.mercurial-scm.org/D8127
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 19 Feb 2020 17:30:04 +0100] rev 44413
revlog-compression: update the config to be a list
format.revlog-compression is now a list of engine, the first supported one is to
be used. Doing this have several benefits:
1) this is fully backward compatible, config using a single entry will be read
as a single item list, not changing any behavior.
2) This open the way to use zstd by default without impacting platform were it
is not available. This will be done in a later changesets.
Using zstd provide a significant performance boost explained in : bb271ec2fbfb.
However zstd is not available in some cases, A notable example is the `--pure`
version of Mercurial which doesn't come with zstd support.
Differential Revision: https://phab.mercurial-scm.org/D8148
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 19 Feb 2020 13:39:00 +0530] rev 44412
remotefilelog: add 'changelog' arg to shallowcg1packer.generate (issue6269)
This cause traceback on widening using narrow extension when remotefilelog
is enabled.
Differential Revision: https://phab.mercurial-scm.org/D8134
Martin von Zweigbergk <martinvonz@google.com> [Tue, 25 Feb 2020 12:41:35 -0800] rev 44411
drawdag: abide by new createmarkers() API
The `obsolete.createmarkers()` API was changed in 6335c0de80fa
(obsolete: allow multiple predecessors in createmarkers, 2018-09-22)
to prefer its precursors input to be a tuple instead of a single
precursor. Let's fix `drawdag.py` to comply.
Differential Revision: https://phab.mercurial-scm.org/D8149
Matt Harbison <matt_harbison@yahoo.com> [Mon, 24 Feb 2020 14:52:46 -0500] rev 44410
lfutil: provide a hint if the largefiles/lfs cache path cannot be determined
A coworker hit this error using an LFS repo in a stripped down environment, and
didn't know how to resolve it.
The final conditional is a bit fast and loose, but there is currently no 'posix'
test in hghave, and it doesn't seem like it's worth adding for this since I
think Windows is the only non-POSIX platform we run tests on.
Differential Revision: https://phab.mercurial-scm.org/D8145
Matt Harbison <matt_harbison@yahoo.com> [Mon, 24 Feb 2020 17:43:34 -0500] rev 44409
tests: replace truncate(1) with inline python
MSYS doesn't have truncate(1) installed by default, and FreeBSD looked unhappy
with the arguments provided.
Differential Revision: https://phab.mercurial-scm.org/D8147
Matt Harbison <matt_harbison@yahoo.com> [Mon, 24 Feb 2020 16:59:35 -0500] rev 44408
tests: stabilize test-rename-merge2.t on Windows
I have no idea why, but this shifted in b4057d001760.
Differential Revision: https://phab.mercurial-scm.org/D8146
Augie Fackler <augie@google.com> [Mon, 24 Feb 2020 13:50:55 -0500] rev 44407
merge with stable
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Thu, 13 Feb 2020 22:51:17 -0500] rev 44406
bookmarks: prevent pushes of divergent bookmarks (foo@remote)
Before this change, such bookmarks are write-only: a client can push
them but not pull/read them. And because these bookmark can't be read,
even pushes are limited (for instance trying to delete such a bookmark
fails with a vanilla client because the client thinks the bookmark is
neither on the local nor the remote).
This change makes the server refuses such bookmarks, and for earlier
errors, makes the client refuse to send them.
I think the change of behavior is acceptable because I think this is a
bug in push/pull, and I don't think we change the behavior of `hg
unbundle`, because it doesn't seem that `hg bundle` ever store
bookmarks (and even if it did, it would seem weird anyway to try to
send divergent bookmarks).
Differential Revision: https://phab.mercurial-scm.org/D8117
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Thu, 13 Feb 2020 22:06:57 -0500] rev 44405
bookmarks: refactor in preparation for next commit
Differential Revision: https://phab.mercurial-scm.org/D8116
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 15 Feb 2020 14:51:33 -0500] rev 44404
bookmarks: avoid traceback when two pushes race to delete the same bookmark
`hg push -f -B remote-only-bookmark` can raise server-side in
`bookmarks._del` (specifically in `self._refmap.pop(mark)`), if the
remote-only bookmark got deleted concurrently.
Fix this by simply not deleting the non-existent bookmark in that
case.
For avoidance of doubt, refusing to delete a bookmark that doesn't
exist when the push starts is taking care of elsewhere; no change of
behavior there.
Differential Revision: https://phab.mercurial-scm.org/D8124
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 15 Feb 2020 15:06:41 -0500] rev 44403
relnotes: add entry about previous `hg recover` change
Differential Revision: https://phab.mercurial-scm.org/D8123
Kyle Lippincott <spectral@google.com> [Thu, 20 Feb 2020 15:15:23 -0800] rev 44402
darwin: add another preemptive gui() call when using chg
Changeset a89381e04c58 added this gui() call before background forks, and
Google's extensions do background forks on essentially every invocation for
logging purposes. The crash is reliably (though not 100%) reproducible without
this change when running `HGPLAIN=1 chg status` in one of our repos. With this
fix, I haven't been able to trigger the crash anymore.
Differential Revision: https://phab.mercurial-scm.org/D8141
Martin von Zweigbergk <martinvonz@google.com> [Fri, 20 Dec 2019 13:24:46 -0800] rev 44401
copy: add experimental support for marking committed copies
The simplest way I'm aware of to mark a file as copied/moved after
committing is this:
hg uncommit --keep <src> <dest> # <src> needed for move, but not copy
hg mv --after <src> <dest>
hg amend
This patch teaches `hg copy` a `--at-rev` argument to simplify that
into:
hg copy --after --at-rev . <src> <dest>
In addition to being simpler, it doesn't touch the working copy, so it
can easily be used even if the destination file has been modified in
the working copy.
Differential Revision: https://phab.mercurial-scm.org/D8035
Martin von Zweigbergk <martinvonz@google.com> [Thu, 26 Dec 2019 14:02:50 -0800] rev 44400
copy: move argument validation a little earlier
Argument validation is usually done early and I will want it done
before some code that I'm about to add.
Differential Revision: https://phab.mercurial-scm.org/D8033
Martin von Zweigbergk <martinvonz@google.com> [Tue, 28 Jan 2020 14:07:57 -0800] rev 44399
copy: add experimetal support for unmarking committed copies
The simplest way I'm aware of to unmark a file as copied after
committing is this:
hg uncommit --keep <dest>
hg forget <dest>
hg add <dest>
hg amend
This patch teaches `hg copy --forget` a `-r` argument to simplify that into:
hg copy --forget --at-rev . <dest>
In addition to being simpler, it doesn't touch the working copy, so it
can easily be used even if the destination file has been modified in
the working copy.
I'll teach `hg copy` without `--forget` to work with `--at-rev` next.
Differential Revision: https://phab.mercurial-scm.org/D8030
Martin von Zweigbergk <martinvonz@google.com> [Fri, 20 Dec 2019 15:50:13 -0800] rev 44398
copy: add option to unmark file as copied
To unmark a file as copied, the user currently has to do this:
hg forget <dest>
hg add <dest>
The new command simplifies that to:
hg copy --forget <dest>
That's not a very big improvement, but I'm planning to also teach `hg
copy [--forget]` a `--at-rev` argument for marking/unmarking copies
after commit (usually with `--at-rev .`).
Differential Revision: https://phab.mercurial-scm.org/D8029
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 11 Feb 2020 11:18:52 +0100] rev 44397
nodemap: introduce an option to use mmap to read the nodemap mapping
The performance and memory benefit is much greater if we don't have to copy all
the data in memory for each information. So we introduce an option (on by
default) to read the data using mmap.
This changeset is the last one definition the API for index support nodemap
data. (they have to be able to use the mmaping).
Below are some benchmark comparing the best we currently have in 5.3 with the
final step of this series (using the persistent nodemap implementation in
Rust). The benchmark run `hg perfindex` with various revset and the following
variants:
Before:
* do not use the persistent nodemap
* use the CPython implementation of the index for nodemap
* use mmapping of the changelog index
After:
* use the MixedIndex Rust code, with the NodeTree object for nodemap access
(still in review)
* use the persistent nodemap data from disk
* access the persistent nodemap data through mmap
* use mmapping of the changelog index
The persistent nodemap greatly speed up most operation on very large
repositories. Some of the previously very fast lookup end up a bit slower because
the persistent nodemap has to be setup. However the absolute slowdown is very
small and won't matters in the big picture.
Here are some numbers (in seconds) for the reference copy of mozilla-try:
Revset Before After abs-change speedup
-10000: 0.004622 0.005532 0.000910 × 0.83
-10: 0.000050 0.000132 0.000082 × 0.37
tip 0.000052 0.000085 0.000033 × 0.61
0 + (-10000:) 0.028222 0.005337 -0.022885 × 5.29
0 0.023521 0.000084 -0.023437 × 280.01
(-10000:) + 0 0.235539 0.005308 -0.230231 × 44.37
(-10:) + :9 0.232883 0.000180 -0.232703 ×1293.79
(-10000:) + (:99) 0.238735 0.005358 -0.233377 × 44.55
:99 + (-10000:) 0.317942 0.005593 -0.312349 × 56.84
:9 + (-10:) 0.313372 0.000179 -0.313193 ×1750.68
:9 0.316450 0.000143 -0.316307 ×2212.93
On smaller repositories, the cost of nodemap related operation is not as big, so
the win is much more modest. Yet it helps shaving a handful of millisecond here
and there.
Here are some numbers (in seconds) for the reference copy of mercurial:
Revset Before After abs-change speedup
-10: 0.000065 0.000097 0.000032 × 0.67
tip 0.000063 0.000078 0.000015 × 0.80
0 0.000561 0.000079 -0.000482 × 7.10
-10000: 0.004609 0.003648 -0.000961 × 1.26
0 + (-10000:) 0.005023 0.003715 -0.001307 × 1.35
(-10:) + :9 0.002187 0.000108 -0.002079 ×20.25
(-10000:) + 0 0.006252 0.003716 -0.002536 × 1.68
(-10000:) + (:99) 0.006367 0.003707 -0.002660 × 1.71
:9 + (-10:) 0.003846 0.000110 -0.003736 ×34.96
:9 0.003854 0.000099 -0.003755 ×38.92
:99 + (-10000:) 0.007644 0.003778 -0.003866 × 2.02
Differential Revision: https://phab.mercurial-scm.org/D7894
Raphaël Gomès <rgomes@octobus.net> [Fri, 14 Feb 2020 15:03:26 +0100] rev 44396
rust-dirstatemap: directly return `non_normal` and `other_entries`
This cleans up the interface which I previously thought needed to be uglier
than in reality. No performance difference, simple refactoring.
Differential Revision: https://phab.mercurial-scm.org/D8121
Martin von Zweigbergk <martinvonz@google.com> [Thu, 26 Dec 2019 14:12:45 -0800] rev 44395
copy: rename `wctx` to `ctx` since it will not necessarily be working copy
Differential Revision: https://phab.mercurial-scm.org/D8032
Martin von Zweigbergk <martinvonz@google.com> [Fri, 20 Dec 2019 14:03:12 -0800] rev 44394
copy: rewrite walkpat() to depend less on dirstate
I want to add a `hg cp/mv -r <rev>` option to mark files as
copied/moved in an existing commit (amending that commit). The code
needs to not depend on the dirstate for that.
Differential Revision: https://phab.mercurial-scm.org/D8031
Martin von Zweigbergk <martinvonz@google.com> [Thu, 13 Feb 2020 10:12:12 -0800] rev 44393
merge with stable
Yuya Nishihara <yuya@tcha.org> [Sat, 01 Feb 2020 12:57:32 +0900] rev 44392
pathutil: resurrect comment about path auditing order
It was removed at 51c86c6167c1, but expensive symlink traversal isn't the
only reason we should walk path components from the root.
Raphaël Gomès <rgomes@octobus.net> [Wed, 16 Oct 2019 14:12:48 +0200] rev 44391
rust-dirstatemap: remove additional lookup in dirstate.matches
We use the same trick as the Python implementation
Differential Revision: https://phab.mercurial-scm.org/D7119
Georges Racinet <georges.racinet@octobus.net> [Tue, 31 Dec 2019 12:43:57 +0100] rev 44390
rust-nodemap: insert method
In this implementation, we are in direct competition
with the C version: this Rust version will have a clear
startup advantage because it will read the data from disk,
but the insertion happens all in memory for both.
Differential Revision: https://phab.mercurial-scm.org/D7795
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 22 Jan 2020 14:21:34 -0500] rev 44389
recover: don't verify by default
The reason is:
- it's not that hard to trigger interrupted transactions: just run out
of disk space
- it takes forever to verify on large repos. Before --no-verify, I
told people to C-c hg recover when the progress bar showed up. Now I
tell them to pass --no-verify.
- I don't remember a single case where the verification step was
useful
This is technically a change of behavior. Perhaps this would be better
suited for tweakdefaults?
Differential Revision: https://phab.mercurial-scm.org/D7972
Augie Fackler <augie@google.com> [Tue, 11 Feb 2020 00:08:28 -0500] rev 44388
context: use manifest.find() instead of two separate calls
I noticed this while debugging an extension that's implementing the manifest
interface. Always nice to save a function call.
Differential Revision: https://phab.mercurial-scm.org/D8109
Raphaël Gomès <rgomes@octobus.net> [Thu, 16 Jan 2020 23:06:01 +0100] rev 44387
rust-matchers: implement `visit_children_set` for `FileMatcher`
As per the removed inline comment, this will become useful in a future patch
in this series as the `IncludeMatcher` is introduced.
Differential Revision: https://phab.mercurial-scm.org/D7914
Augie Fackler <augie@google.com> [Wed, 05 Feb 2020 17:13:51 -0500] rev 44386
manifest: move matches method to be outside the interface
In order to adequately smoke out any legacy consumers of the method, we rename
it to _matches so it's clear that it's class-private. To my amazement, all
consumers of this method really only wanted matching filenames, not a full
filtered manifest.
Differential Revision: https://phab.mercurial-scm.org/D8085
Augie Fackler <augie@google.com> [Mon, 10 Feb 2020 21:02:22 -0500] rev 44385
tags: use modern // operator for division
Fixes a test on Python 3.
# skip-blame only correcting a division operator, not a substantive change
Differential Revision: https://phab.mercurial-scm.org/D8108
Augie Fackler <augie@google.com> [Mon, 10 Feb 2020 20:47:19 -0500] rev 44384
tags: fix some type confusion exposed in python 3
# skip-blame just b-prefix and %-format cleanup, no meaningful change
Differential Revision: https://phab.mercurial-scm.org/D8107
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 Jan 2020 17:20:12 -0800] rev 44383
rebase: remove some now-unused parent arguments
Differential Revision: https://phab.mercurial-scm.org/D7829
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 Jan 2020 21:40:01 -0800] rev 44382
rebase: remove some redundant setting of dirstate parents
Since we're now setting the dirstate parents to its correct values
from the beginning (right after `merge.update()`), we usually don't
need to set them again before committing. The only case we need to
care about is when committing collapsed commits. So we can remove the
`setparents()` calls just before committing and add one only for the
collapse case.
Differential Revision: https://phab.mercurial-scm.org/D7828
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 Jan 2020 14:22:20 -0800] rev 44381
rebase: don't use rebased node as dirstate p2 (BC)
When rebasing a node, we currently use the rebased node as p2 in the
dirstate until just before we commit it (we then change to the desired
parents). This p2 is visible to the user when the rebase gets
interrupted because of merge conflicts. That can be useful to the user
as a reminder of which commit is currently being rebased, but I
believe it's incorrect for a few reasons:
* I think the dirstate parents should be the ones that will be set
when the commit is created.
* I think having two parents means that you're merging those two
commits, but when rebasing, you're generally grafting, not merging.
* When rebasing a merge commit, we should use the two desired parents
as dirstate parents (and we clearly can't have the rebased node as
a third dirstate parent).
* `hg graft` (and `hg update --merge`) sets only one parent and `hg
rebase` should be consistent with that.
I realize that this is a somewhat large user-visible change, but I
think it's worth it because it will simplify things quite a bit.
Differential Revision: https://phab.mercurial-scm.org/D7827
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 Jan 2020 14:17:56 -0800] rev 44380
rebase: stop relying on having two parents to resume rebase
I'm about to make it so we don't have two parents when a rebase is
interrupted (unless we're just rebasing on a merge commit). The code
for detecting if we're resuming a rebase relied on having two parents,
so this patch rewrites that to instead set a boolean when we resume.
Note that `self.resume` in the new condition implies `not
self.inmemory` (rebase cannot be resumed in memory), so that's why
that part can be omitted.
Differential Revision: https://phab.mercurial-scm.org/D7826
Martin von Zweigbergk <martinvonz@google.com> [Tue, 28 Jan 2020 21:49:50 -0800] rev 44379
graphlog: use '%' for other context in merge conflict
This lets the user more easily find the commit that is involved in the
conflict, such as the source of `hg update -m` or the commit being
grafted by `hg graft`.
Differential Revision: https://phab.mercurial-scm.org/D8043
Martin von Zweigbergk <martinvonz@google.com> [Wed, 29 Jan 2020 14:42:54 -0800] rev 44378
tests: add `hg log -G` output when there are merge conflicts
The next commit will change the behavior for these. I've used slightly
different commands in the different tests to match the surrounding
style.
Differential Revision: https://phab.mercurial-scm.org/D8042
Martin von Zweigbergk <martinvonz@google.com> [Wed, 29 Jan 2020 11:30:35 -0800] rev 44377
revset: add a revset for parents in merge state
This may be particularly useful soon, when I'm going to change how `hg
rebase` sets its parents during conflict resolution.
Differential Revision: https://phab.mercurial-scm.org/D8041
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 Jan 2020 17:46:10 -0800] rev 44376
tests: add test of rebase with conflict in merge commit
It doesn't seem like we had any tests of this. I think it's pretty
weird that the two parents we're merging are not the working copy
parents during the conflict resolution.
Differential Revision: https://phab.mercurial-scm.org/D7824
Martin von Zweigbergk <martinvonz@google.com> [Thu, 16 Jan 2020 00:03:19 -0800] rev 44375
rebase: always be graft-like, not merge-like, also for merges
Rebase works by updating to a commit and then grafting changes on
top. However, before this patch, it would actually merge in changes
instead of grafting them in in some cases. That is, it would use the
common ancestor as base instead of using one of the parents. That
seems wrong to me, so I'm changing it so `defineparents()` always
returns a value for `base`.
This fixes the bad behavior in test-rebase-newancestor.t, which was
introduced in 65f215ea3e8e (tests: add test for rebasing merges with
ancestors of the rebase destination, 2014-11-30).
The difference in test-rebase-dest.t is because the files in the tip
revision were A, D, E, F before this patch and A, D, F, G after it. I
think both files should ideally be there.
Differential Revision: https://phab.mercurial-scm.org/D7907
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:51:01 +0100] rev 44374
nodemap: update the index with the newly written data (when appropriate)
If we are to use mmap to read the nodemap data, and if the python code is
responsible for the IO, we need to refresh the mmap after each write and provide
it back to the index.
We start this dance without the mmap first.
Differential Revision: https://phab.mercurial-scm.org/D7893
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:50:52 +0100] rev 44373
nodemap: never read more than the expected data amount
Since we are tracking this number we can use it to detect corrupted rawdata file
and to only read the correct amount of data when possible.
Differential Revision: https://phab.mercurial-scm.org/D7892
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:50:43 +0100] rev 44372
nodemap: write new data from the expected current data length
If the amount of data in the file exceed the expect amount, we will overwrite
the extra data. This is a simple way to be safer.
Differential Revision: https://phab.mercurial-scm.org/D7891
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:50:33 +0100] rev 44371
nodemap: double check the source docket when doing incremental update
In theory, the index will have the information we expect it to have. However by
security, it seems safer to double check that the incremental data are generated
from the data currently on disk.
Differential Revision: https://phab.mercurial-scm.org/D7890
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:50:24 +0100] rev 44370
nodemap: track the total and unused amount of data in the rawdata file
We need to keep that information around:
* total data will allow transaction to start appending new information without
confusing other reader.
* unused data will allow to detect when we should regenerate new rawdata file.
Differential Revision: https://phab.mercurial-scm.org/D7889
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:50:14 +0100] rev 44369
nodemap: track the maximum revision tracked in the nodemap
We need a simple way to detect when the on disk data contains less revision
than the index we read from disk. The docket file is meant for this, we just had
to start tracking that data.
We should also try to detect strip operation, but we will deal with this in
later changesets. Right now we are focusing on defining the API for index
supporting persistent nodemap.
Differential Revision: https://phab.mercurial-scm.org/D7888
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:50:04 +0100] rev 44368
nodemap: add a flag to dump the details of the docket
We are about to add more information to the docket. We first introduce a way to
debug its content.
Differential Revision: https://phab.mercurial-scm.org/D7887
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:49:54 +0100] rev 44367
nodemap: introduce append-only incremental update of the persistent data
Rewriting the full nodemap for each transaction has a cost we would like to
avoid. We introduce a new way to write persistent nodemap data by adding new
information at the end for file. Any new and updated block as added at the end
of the file. The last block is the new root node.
With this method, some of the block already on disk get "dereferenced" and
become dead data. In later changesets, We'll start tracking the amount of dead
data to eventually re-generate a full nodemap.
Differential Revision: https://phab.mercurial-scm.org/D7886
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:49:45 +0100] rev 44366
nodemap: keep track of the docket for loaded data
To perform incremental update of the on disk data, we need to keep tracks of
some aspect of that data.
Differential Revision: https://phab.mercurial-scm.org/D7885
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:49:35 +0100] rev 44365
nodemap: introduce an explicit class/object for the docket
We are about to add more information to this docket, having a clear location to
stock them in memory will help.
Differential Revision: https://phab.mercurial-scm.org/D7884
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:49:26 +0100] rev 44364
nodemap: keep track of the ondisk id of nodemap blocks
If we are to incrementally update the files, we need to keep some details about
the data we read.
Differential Revision: https://phab.mercurial-scm.org/D7883
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:49:16 +0100] rev 44363
nodemap: provide the on disk data to indexes who support it
Time to start defining the API and prepare the rust index support. We provide
a method to do so. We use a distinct method instead of passing them in the
constructor because we will need this method anyway later (to refresh the mmap
once we update the data on disk).
Differential Revision: https://phab.mercurial-scm.org/D7847
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:49:06 +0100] rev 44362
nodemap: all check that revision and nodes match in the nodemap
More check is always useful.
Differential Revision: https://phab.mercurial-scm.org/D7846
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:48:57 +0100] rev 44361
nodemap: add basic checking of the on disk nodemap content
The simplest check it so verify we have all the revision we needs, and nothing
more.
Differential Revision: https://phab.mercurial-scm.org/D7845
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:48:47 +0100] rev 44360
nodemap: code to parse the persistent binary nodemap data
We now have code to read back what we persisted. This will be put to use in
later changesets.
Differential Revision: https://phab.mercurial-scm.org/D7844
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:48:38 +0100] rev 44359
nodemap: move the iteratio inside the Block object
Having the iteration inside the serialization function does not help
readability. Now that we have a `Block` object, let us move that code there.
Differential Revision: https://phab.mercurial-scm.org/D7843
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:48:28 +0100] rev 44358
nodemap: use an explicit "Block" object in the reference implementation
This will help us to introduce some test around the data currently written on
disk.
Differential Revision: https://phab.mercurial-scm.org/D7842
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:48:19 +0100] rev 44357
nodemap: add a optional `nodemap_add_full` method on indexes
This method can be used to obtains persistent data for a full nodemap. The end
goal is for some index implementation to managed the nodemap serialization them
selves (eg: the rust implementation)
Differential Revision: https://phab.mercurial-scm.org/D7841
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:48:09 +0100] rev 44356
nodemap: add a (python) index class for persistent nodemap testing
Using the persistent nodemap require a compeling performance boost and an
existing implementation. The benefit of the persistent nodemap for pure python
code is unclear and we don't have a C implementation for it. Yet we would like
to actually start testing it in more details and define an API for using that
persistent nodemap.
We introduce a new `devel` config option to use an index class dedicated to
Nodemap Testing. This feature is "pure" only because having using a pure-python
index with the `cext` policy proved more difficult than I would like.
There is nothing going on in that class for now, but the coming changeset will
change that.
Differential Revision: https://phab.mercurial-scm.org/D7840
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:47:59 +0100] rev 44355
nodemap: delete older raw data file when creating a new ones
When we write new full files, it replace an older one with a different name. We
add the associated cleanup for the older file to be removed after the
transaction.
We delete all file matching the expected pattern to give use extra chance to
delete orphan files we might have failed to delete earlier.
Note: eventually we won't rewrite all data for each transaction. This is coming
in later changesets.
Differential Revision: https://phab.mercurial-scm.org/D7839
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:47:50 +0100] rev 44354
nodemap: use an intermediate "docket" file to carry small metadata
This intermediate file will make mmapping, transaction and content validation
easier. (Most of this usefulness will arrive gradually in later changeset). In
particular it will become very useful to append new data are the end of raw file
instead of rewriting on the file on each transaction.
See in code comments for details.
Differential Revision: https://phab.mercurial-scm.org/D7838
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:47:40 +0100] rev 44353
nodemap: only use persistent nodemap for non-inlined revlog
Revlog are inlined while they are small (to avoid having too many file to deal
with). The persistent nodemap will only provides a significant boost for large
enough revlog index. So it does not make sens to add an extra file to store
nodemap for small revlog.
We could consider inclining the nodemap data inside the revlog itself, but the
benefit is unclear so let it be an adventure for another time.
Differential Revision: https://phab.mercurial-scm.org/D7837
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:47:31 +0100] rev 44352
nodemap: add a function to read the data from disk
This changeset is small and mostly an excuse to introduce an API function
reading the data from disk.
Differential Revision: https://phab.mercurial-scm.org/D7836
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:47:21 +0100] rev 44351
nodemap: write nodemap data on disk
Let us start writing data on disk (so that we can read it from there later).
This series of changeset is going to focus first on having data on disk and
updating it.
Right now the data is written right next to the revlog data, in the store. We
might move it to cache (with proper cache validation mechanism) later, but for
now revlog have a storevfs instance and it is simpler to us it. The right
location for this data is not the focus of this series.
Differential Revision: https://phab.mercurial-scm.org/D7835
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:47:12 +0100] rev 44350
nodemap: have some python code writing a nodemap in persistent binary form
This python code aims to be as "simple" as possible. It is a reference
implementation of the data we are going to write on disk (and possibly,
later a way for pure python install to make sure the on disk data are up to
date).
It is not optimized for performance and rebuild the full data structure from
the index every time.
This is a stepping stone toward a persistent nodemap on disk.
Differential Revision: https://phab.mercurial-scm.org/D7834
Augie Fackler <augie@google.com> [Mon, 10 Feb 2020 17:31:05 -0500] rev 44349
cleanup: re-run black on the codebase
Looks like a few patches have landed without having been blackened. I
strongly suspect I should write a patch for baymax that blackens
things on the way in...
# skip-blame automatic formatting
Differential Revision: https://phab.mercurial-scm.org/D8104
Raphaël Gomès <rgomes@octobus.net> [Thu, 16 Jan 2020 13:34:04 +0100] rev 44348
rust-re2: add wrapper for calling Re2 from Rust
This assumes that Re2 is installed following Google's guide. I am not sure
how we want to integrate it in the project, but I think a follow-up patch would
be more appropriate for such work.
As it stands, *not* having Re2 installed results in a compilation error, which
is a problem as it breaks install compatibility. Hence, this is gated behind
a non-default `with-re2` compilation feature.
Differential Revision: https://phab.mercurial-scm.org/D7910
Raphaël Gomès <rgomes@octobus.net> [Thu, 16 Jan 2020 11:27:12 +0100] rev 44347
rust-filepatterns: add support for `include` and `subinclude` patterns
This prepares a future patch for `IncludeMatcher` on the road to bare
`hg status` support.
Differential Revision: https://phab.mercurial-scm.org/D7909
Raphaël Gomès <rgomes@octobus.net> [Thu, 16 Jan 2020 10:28:40 +0100] rev 44346
rust-filepatterns: improve API and robustness for pattern files parsing
Within the next few patches we will be using this new API.
Differential Revision: https://phab.mercurial-scm.org/D7908
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 17:10:20 +0100] rev 44345
rust-utils: add util for canonical path
Differential Revision: https://phab.mercurial-scm.org/D7871
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 10:24:32 -0500] rev 44344
httpconnection: allow `httpsendfile` subclasses to suppress the progressbar
This will be neccessary for LFS, which manages the progressbar outside of the
file.
Differential Revision: https://phab.mercurial-scm.org/D7960
Raphaël Gomès <rgomes@octobus.net> [Mon, 10 Feb 2020 21:54:12 +0100] rev 44343
rust-dirstatemap: add `NonNormalEntries` class
This fix introduces the same encapsulation as the `copymap`. There is no easy
way of doing this any better for now.
`hg up -r null && time HGRCPATH= HGMODULEPOLICY=rust+c hg up tip` on Mozilla
Central, (not super recent, but it doesn't matter):
Before: 7:44,08 total
After: 1:03,23 total
Pretty brutal regression!
Differential Revision: https://phab.mercurial-scm.org/D8049
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sun, 09 Feb 2020 16:18:26 -0500] rev 44342
help: when possible, indicate flags implied by tweakdefaults
Differential Revision: https://phab.mercurial-scm.org/D8101
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sun, 09 Feb 2020 15:50:36 -0500] rev 44341
help: add a mechanism to change flags' help depending on config
It seems reasonable to have a similar mechanism for the rest of the
help, but no such thing is implemented.
The goal is to make the help of commands clearer in the presence of
significant default changes, like tweakdefaults or with company-wide
hgrcs. In these cases, a user looking at the help of a command doesn't
exactly know what his hgrc is doing.
Apply to this to the --git option of commands that display diffs, as
this option in particular causes confusion for some reason.
Differential Revision: https://phab.mercurial-scm.org/D8100
Matt Harbison <matt_harbison@yahoo.com> [Sat, 08 Feb 2020 23:39:55 -0500] rev 44340
lfs: use str for the open() mode when opening a blob for py3
The other fix for this was to leave the mode as bytes, and import
`pycompat.open()` like a bunch of other modules do. But I think it's confusing
to still use bytes at the python boundary, and obviously error prone. Grepping
for ` open\(.+, ['"][a-z]+['"]\)` and ` open\(.+, b['"][a-z]+['"]\)` outside of
`tests`, there are 51 and 87 uses respectively, so it's not like this is a rare
direct usage.
Differential Revision: https://phab.mercurial-scm.org/D8099
Raphaël Gomès <rgomes@octobus.net> [Thu, 30 Jan 2020 14:57:02 +0100] rev 44339
rust-dirstatemap: cache non normal and other parent set
Performance of `hg update` was significantly worse since the introduction of
the Rust `dirstatemap`. This regression was noticed by Valentin Gatien-Baron
when working on a large repository, as it goes unnoticed for smaller
repositories like Mercurial itself.
This fix introduces the same getter/setter mechanism at `hg-core` level as
for `set/get_dirs`.
While this technique is, as previously discussed, quite suboptimal, it fixes an
important enough problem. Refactoring `hg-core` to use the typestate
pattern could be a good approach to improving code quality in a future patch.
Differential Revision: https://phab.mercurial-scm.org/D8048
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Fri, 07 Feb 2020 16:01:32 -0500] rev 44338
tags: behave better if a tags cache entry is partially written
This is done by discarding any partial cache entry, instead of
filling the partial cache entry with 0xff before.
Differential Revision: https://phab.mercurial-scm.org/D8095
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Fri, 07 Feb 2020 15:55:26 -0500] rev 44337
tags: show how hg behaves if a tags cache entry is truncated
I'm seeing an error of this form in production on the order of once a
month. I'm not sure how it happens, but I suspect interrupting a pull
might result in half written cache entries.
Differential Revision: https://phab.mercurial-scm.org/D8094
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Fri, 07 Feb 2020 13:54:09 -0500] rev 44336
tags: add a debug command to display .hg/cache/hgtagsfnodes1
Differential Revision: https://phab.mercurial-scm.org/D8093
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Sat, 08 Feb 2020 10:22:47 -0500] rev 44335
purge: add -i flag to delete ignored files instead of untracked files
It's convenient for deleting build artifacts. Using --all instead
would delete other things too.
Differential Revision: https://phab.mercurial-scm.org/D8096
Matt Harbison <matt_harbison@yahoo.com> [Thu, 30 Jan 2020 19:50:43 -0500] rev 44334
pyoxidizer: use `legacy_windows_stdio` on Windows
The C executable sets this too, otherwise no output shows up (when paging?).
There is also `legacy_windows_fs_encoding`, but I'm not setting that for now
because the C executable doesn't either.
Differential Revision: https://phab.mercurial-scm.org/D8053
Augie Fackler <augie@google.com> [Wed, 05 Feb 2020 17:12:39 -0500] rev 44333
merge: use manifestdict.walk() instead of manifestdict.matches()
As with other patches in this series, this avoids making a
potentially-expensive copy of a manifest.
Differential Revision: https://phab.mercurial-scm.org/D8084
Augie Fackler <augie@google.com> [Wed, 05 Feb 2020 16:58:50 -0500] rev 44332
manifest: rewrite filesnotin to not make superfluous manifest copies
This also skips using diff() when all we care about is the filenames. I'm
expecting the built in set logic to be plenty fast. For really large manifests
with a matcher in play this should copy substantially less data around.
Differential Revision: https://phab.mercurial-scm.org/D8082
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 08 Feb 2020 03:13:45 +0530] rev 44331
merge with stable
Augie Fackler <augie@google.com> [Thu, 06 Feb 2020 16:55:39 -0500] rev 44330
archival: use walk() instead of matches() on manifest
All we care about is the filepaths, so this avoids a pointless copy of the
manifest that we only used to extract matching filenames.
Differential Revision: https://phab.mercurial-scm.org/D8090
Raphaël Gomès <rgomes@octobus.net> [Fri, 24 Jan 2020 11:10:07 +0100] rev 44329
rust-dirs-multiset: improve temporary error message
While we wait on a future patch that could verify that the paths passed to
`DirsMultiset` have been audited, we still need to handle this error.
This patch makes it easier to bubble up and makes the error clearer.
Also, this patch introduces the `subslice_index` function that could be useful
for other - albeit niche - purposes.
Differential Revision: https://phab.mercurial-scm.org/D7921
Matt Harbison <matt_harbison@yahoo.com> [Wed, 22 Jan 2020 12:11:35 -0500] rev 44328
exchange: check the `ui.clonebundleprefers` form while processing (issue6257)
Otherwise the clone command will emit a long stacktrace if there is no `=`
character.
Differential Revision: https://phab.mercurial-scm.org/D7969
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 13 Dec 2019 16:49:05 +0100] rev 44327
copies: add a new test dedicated to testing chain of changeset with merge
The copies test we currently have usually focus on simple case that do not dive
too much into longer chains involving merges. This new test file focus on
extensive testing of these case to validate their behavior and make sure the
various copies algorithm have the same behavior.
And… actually these test are currently broken for the changeset centric
algorithm since 99ebde4fec99, but it went undetected because these case were not
tested.
Differential Revision: https://phab.mercurial-scm.org/D8078
Joerg Sonnenberger <joerg@bec.de> [Wed, 18 Sep 2019 06:07:09 +0200] rev 44326
hgext: initial version of fastexport extension
Differential Revision: https://phab.mercurial-scm.org/D7733
Julien Cristau <jcristau@mozilla.com> [Fri, 07 Feb 2020 15:55:21 +0100] rev 44325
hghave: cache the result of gethgversion
hghave --test-features calls it 90 times, each one calling hg --version
which takes a tenth of a second on my workstation, adding up to about
10s win on test-hghave.t.
Fixes https://bugs.debian.org/939756
Differential Revision: https://phab.mercurial-scm.org/D8092
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 14:11:43 -0800] rev 44324
clean: delete obsolete unlinking of .hg/graftstate
The responsibility for clearing it is now in
`cmdutil.clearunfinished()`, so we shouldn't have to unlink it in
`hg.clean()`.
Differential Revision: https://phab.mercurial-scm.org/D7992
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 Feb 2020 10:16:30 -0800] rev 44323
copies: avoid filtering by short-circuit dirstate-only copies earlier
The call to `y.ancestor(x)` triggered repo filtering, which we'd like
to avoid in the simple `hg status --copies` case.
Differential Revision: https://phab.mercurial-scm.org/D8071
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 Feb 2020 10:14:44 -0800] rev 44322
tests: add test showing that repo filter is calculated for `hg st --copies`
Differential Revision: https://phab.mercurial-scm.org/D8070
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 11:40:15 -0500] rev 44321
lfs: enable workers by default
With the stall issue seemingly fixed, there's no reason not to use workers. The
setting is left for now to keep the test output deterministic, and in case other
issues come up. If none do, this can be converted to a developer setting for
usage with the tests.
Differential Revision: https://phab.mercurial-scm.org/D7963
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 11:32:33 -0500] rev 44320
lfs: fix the stall and corruption issue when concurrently uploading blobs
We've avoided the issue up to this point by gating worker usage with an
experimental config. See 10e62d5efa73, and the thread linked there for some of
the initial diagnosis, but essentially some data was being read from the blob
before an error occurred and `keepalive` retried, but didn't rewind the file
pointer. So the leading data was lost from the blob on the server, and the
connection stalled, trying to send more data than available.
In trying to recreate this, I was unable to do so uploading from Windows to
CentOS 7. But it reproduced every time going from CentOS 7 to another CentOS 7
over https.
I found recent fixes in the FaceBook repo to address this[1][2]. The commit
message for the first is:
The KeepAlive HTTP implementation is bugged in it's retry logic, it supports
reading from a file pointer, but doesn't support rewinding of the seek cursor
when it performs a retry. So it can happen that an upload fails for whatever
reason and will then 'hang' on the retry event.
The sequence of events that get triggered are:
- Upload file A, goes OK. Keep-Alive caches connection.
- Upload file B, fails due to (for example) failing Keep-Alive, but LFS file
pointer has been consumed for the upload and fd has been closed.
- Retry for file B starts, sets the Content-Length properly to the expected
file size, but since file pointer has been consumed no data will be uploaded,
causing the server to wait for the uploaded data until either client or
server reaches a timeout, making it seem as our mercurial process hangs.
This is just a stop-gap measure to prevent this behavior from blocking Mercurial
(LFS has retry logic). A proper solutions need to be build on top of this
stop-gap measure: for upload from file pointers, we should support fseek() on
the interface. Since we expect to consume the whole file always anyways, this
should be safe. This way we can seek back to the beginning on a retry.
I ported those two patches, and it works. But I see that `url._sendfile()` does
a rewind on `httpsendfile` objects[3], so maybe it's better to keep this all in
one place and avoid a second seek. We may still want the first FaceBook patch
as extra protection for this problem in general. The other two uses of
`httpsendfile` are in the wire protocol to upload bundles, and to upload
largefiles. Neither of these appear to use a worker, and I'm not sure why
workers seem to trigger this, or if this could have happened without a worker.
Since `httpsendfile` already has a `close()` method, that is dropped. That
class also explicitly says there's no `__len__` attribute, so that is removed
too. The override for `read()` is necessary to avoid the progressbar usage per
file.
[1] https://github.com/facebookexperimental/eden/commit/c350d6536d90c044c837abdd3675185644481469
[2] https://github.com/facebookexperimental/eden/commit/77f0d3fd0415e81b63e317e457af9c55c46103ee
[3] https://www.mercurial-scm.org/repo/hg/file/5.2.2/mercurial/url.py#l176
Differential Revision: https://phab.mercurial-scm.org/D7962
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 10:34:15 -0500] rev 44319
lfs: add a method to the local blobstore to convert OIDs to file paths
This is less ugly than passing an open callback to the `httpsendfile`
constuctor.
Differential Revision: https://phab.mercurial-scm.org/D7961
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 14:47:38 -0800] rev 44318
merge: introduce a revert_to() for that use-case
In the same vein as the previous patch.
Differential Revision: https://phab.mercurial-scm.org/D7901
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 15:30:25 -0800] rev 44317
merge: introduce a clean_update() for that use-case
I find it hard to understand what value to pass for all the arguments
to `merge.update()`. I would like to introduce functions that are more
specific to each use-case. We already have `graft()`. This patch
introduces a `clean_update()` and uses it in some places to show that
it works.
Differential Revision: https://phab.mercurial-scm.org/D7902
Augie Fackler <augie@google.com> [Wed, 05 Feb 2020 16:16:15 -0500] rev 44316
manifest: fix _very_ subtle bug with exact matchers passed to walk()
Prior to this fix, manifestdict.walk() with an exact matcher would blindly
list the files in the matcher, even if they weren't in the manifest. This was
exposed by my next patch where I rewrite filesnotin() to use walk() instead of
matches().
Differential Revision: https://phab.mercurial-scm.org/D8081
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 17:08:45 +0100] rev 44315
rust-utils: add `Escaped` trait
This will be used as a general interface for displaying things to the user.
The upcoming `IncludeMatcher` will use it to store its patterns in a
user-displayable string.
Differential Revision: https://phab.mercurial-scm.org/D7870
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 17:04:32 +0100] rev 44314
rust-dirs-multiset: add `DirsChildrenMultiset`
In a future patch, this structure will be needed to store information needed by
the (also upcoming) `IgnoreMatcher`.
Differential Revision: https://phab.mercurial-scm.org/D7869
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 16:50:35 +0100] rev 44313
rust-hg-path: add useful methods to `HgPath`
This changeset introduces the use of the `pretty_assertions` crate for easier
to read test output.
Differential Revision: https://phab.mercurial-scm.org/D7867
Raphaël Gomès <rgomes@octobus.net> [Wed, 05 Feb 2020 17:05:37 +0100] rev 44312
rust-pathauditor: add Rust implementation of the `pathauditor`
It does not offer the same flexibility as the Python implementation, but
should check incoming paths just as well.
Differential Revision: https://phab.mercurial-scm.org/D7866
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 22 Jan 2020 03:17:06 +0530] rev 44311
py3: catch AttributeError too with ImportError
Looks like py3 raises AttributeError instead of ImportError. This is caught on
windows.
Differential Revision: https://phab.mercurial-scm.org/D7965
Augie Fackler <augie@google.com> [Wed, 05 Feb 2020 15:15:18 -0500] rev 44310
context: use manifest.walk() instead of manifest.match() to get file list
The former doesn't create a whole extra manifest in order to produce the
matching file list, which is all we actually cared about here. Sigh.
Differential Revision: https://phab.mercurial-scm.org/D8080
Augie Fackler <augie@google.com> [Wed, 05 Feb 2020 15:01:22 -0500] rev 44309
manifest: remove `.new()` from the interface
Nothing used it.
Differential Revision: https://phab.mercurial-scm.org/D8079
Kyle Lippincott <spectral@google.com> [Wed, 29 Jan 2020 13:39:50 -0800] rev 44308
chg: force-set LC_CTYPE on server start to actual value from the environment
Python 3.7+ will "coerce" the LC_CTYPE variable in many instances, and this can
cause issues with chg being able to start up. D7550 attempted to fix this, but a
combination of a misreading of the way that python3.7 does the coercion and an
untested state (LC_CTYPE being set to an invalid value) meant that this was
still not quite working.
This change will cause differences between chg and hg: hg will have the LC_CTYPE
environment variable coerced, while chg will not. This is unlikely to cause any
detectable behavior differences in what Mercurial itself outputs, but it does
have two known effects:
- When using hg, the coerced LC_CTYPE will be passed to subprocesses, even
non-python ones. Using chg will remove the coercion, and this will not
happen. This is arguably more correct behavior on chg's part.
- On macOS, if you set your region to Brazil but your language to English,
this isn't representable in locale strings, so macOS sets LC_CTYPE=UTF-8. If
this value is passed along when ssh'ing to a non-macOS machine, some
functions (such as locale.setlocale()) may raise an exception due to an
unsupported locale setting. This is most easily encountered when doing an
interactive commit/split/etc. when using ui.interface=curses.
Differential Revision: https://phab.mercurial-scm.org/D8039
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 Feb 2020 09:00:05 +0100] rev 44307
perf: fix list formatting in perfindex documentation
Differential Revision: https://phab.mercurial-scm.org/D8067
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 01 Feb 2020 09:14:36 +0100] rev 44306
test: simplify test-amend.t to avoid race condition
Insted on relying on sleep, we could simply have the editor do the file change.
This remove the reliance on "sleep" and avoid test failing on heavy load
machine.
To test this, I reverted the code change in 5558e3437872 and the test started
failing again.
Differential Revision: https://phab.mercurial-scm.org/D8065
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 13 Dec 2019 11:32:36 +0100] rev 44305
test: document test-copy-move-merge.t
Differential Revision: https://phab.mercurial-scm.org/D8077
Augie Fackler <augie@google.com> [Mon, 03 Feb 2020 22:16:36 -0500] rev 44304
manifest: remove optional default= argument on flags(path)
It had only one caller inside manifest.py, and treemanifest was
actually incorrectly implemented. treemanifest is still missing the
fastdelta() method from the interface (and so doesn't yet conform),
but this is at least progress.
Differential Revision: https://phab.mercurial-scm.org/D8069
Augie Fackler <augie@google.com> [Mon, 03 Feb 2020 11:56:02 -0500] rev 44303
resourceutil: blacken
Augie Fackler <augie@google.com> [Mon, 03 Feb 2020 11:51:52 -0500] rev 44302
merge with stable
Martin von Zweigbergk <martinvonz@google.com> [Fri, 31 Jan 2020 10:53:50 -0800] rev 44301
rebase: abort if the user tries to rebase the working copy
I think it's more correct to treat `hg rebase -r 'wdir()' -d foo`
as `hg co -m foo`, but I'm instead making it error out. That's partly
because it's probably what the user wanted (in the case I heard from a
user, they had done `hg rebase -s f` where `f` resolved to `wdir()`)
and partly because I don't want to think about more complicated cases
where the user specifies the working copy together with other commits.
Differential Revision: https://phab.mercurial-scm.org/D8057
Martin von Zweigbergk <martinvonz@google.com> [Fri, 31 Jan 2020 10:41:50 -0800] rev 44300
tests: add tests for rebasing wdir() revision
Differential Revision: https://phab.mercurial-scm.org/D8056
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Jan 2020 13:29:26 -0800] rev 44299
merge: when rename was made on both sides, use ancestor as merge base
When both sides of a merge have renamed a file to the same place, we
would treat that as a "both created" action in merge.py. That means
that we'd use an empty diffbase. It seems better to use the copy
source as diffbase. That can be done by simply dropping code that
prevented us from doing that. I think I did it that way in
57203e0210f8 (copies: calculate mergecopies() based on pathcopies(),
2019-04-11) only to preserve the existing behavior. I also suspect it
was just an accident that it behaved that way before that commit.
Note that until fa9ad1da2e77 (merge: start using the per-side copy
dicts, 2020-01-23), it was non-deterministic (depending on iteration
order of the `allsources` set in `copies._fullcopytracing()`) which
source was used in the affected test case in test-rename-merge1.t. We
could easily have fixed that by sorting them, but now we can instead
detect the case (the TODO added in the previous patch).
Differential Revision: https://phab.mercurial-scm.org/D7974
Martin von Zweigbergk <martinvonz@google.com> [Fri, 31 Jan 2020 08:47:32 -0800] rev 44298
absorb: graduate -i flag from experimental
The interactive mode seems to work well. I have previously thought
that `-i` should be what `-e` does, but the current behavior matches
what other `-i` flags do (select a subset of the hunks), so I think
that is what we want.
Differential Revision: https://phab.mercurial-scm.org/D8055
Yuya Nishihara <yuya@tcha.org> [Sat, 25 Jan 2020 17:30:24 +0900] rev 44297
rust-cpython: remove PySharedRefCell and its companion structs
Also updates py_shared_iterator!() documentation accordingly.
Yuya Nishihara <yuya@tcha.org> [Sat, 25 Jan 2020 17:26:23 +0900] rev 44296
rust-cpython: switch to upstreamed version of PySharedRefCell
Our PyLeaked is identical to cpython::UnsafePyLeaked. I've renamed it because
it provides mostly unsafe functions.
Yuya Nishihara <yuya@tcha.org> [Sat, 25 Jan 2020 17:21:06 +0900] rev 44295
rust-cpython: rename inner_shared() to inner()
The "shared" accessor will be automatically generated, and will have the
same name as the data itself.
Yuya Nishihara <yuya@tcha.org> [Fri, 31 Jan 2020 00:08:30 +0900] rev 44294
rust-cpython: use PyList.insert() instead of .insert_item()
Silences the deprecated warning.
https://github.com/dgrunwald/rust-cpython/commit/e8cbe864841714c5555db8c90e057bd11e360c7f
Yuya Nishihara <yuya@tcha.org> [Fri, 31 Jan 2020 00:01:29 +0900] rev 44293
rust-cpython: bump cpython to 0.4 to switch to upstreamed PySharedRef
Yuya Nishihara <yuya@tcha.org> [Thu, 30 Jan 2020 23:57:19 +0900] rev 44292
rust: update dependencies
For no particular reason, but just because I'll bump the rust-cpython version.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 24 Jan 2020 12:50:27 +0100] rev 44291
contrib: a small script to nudge lingering diff
After a discussion on IRC with various reviewers. It seems like a good idea to
have some automatic cleanup of old, inactive diffs.
Here is a small script able to do so. I am preparing to unleash it on our
phabricator instance.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 26 Jan 2020 16:23:57 -0800] rev 44290
packaging: add support for PyOxidizer
I've successfully built Mercurial on the development tip of
PyOxidizer on Linux and Windows. It mostly "just works" on Linux.
Windows is a bit more finicky.
In-memory resource files are probably not all working correctly
due to bugs in PyOxidizer's naming of modules. PyOxidizer now
now supports installing files next to the produced binary. (We
do this for templates in the added file.) So a workaround
should be available.
Also, since the last time I submitted support for PyOxidizer,
PyOxidizer gained the ability to auto-generate Rust projects
to build executables. So we don't need to worry about vendoring
any Rust code to initially support PyOxidizer. However, at some
point we will likely want to write our own command line driver
that embeds a Python interpreter via PyOxidizer so we can run
Rust code outside the confines of a Python interpreter. But that
will be a follow-up.
I would also like to add packaging.py CLI commands to build
PyOxidizer distributions. This can come later, if ever.
PyOxidizer's new "targets" feature makes it really easy to define
packaging tasks in its Starlark configuration file. While not
much is implemented yet, eventually we should be able to produce
MSIs, etc using a `pyoxidizer build` one-liner. We'll get there...
Differential Revision: https://phab.mercurial-scm.org/D7450
Martin von Zweigbergk <martinvonz@google.com> [Wed, 29 Jan 2020 11:30:16 -0800] rev 44289
mergestate: add accessors for local and other nodeid, not just contexts
The mergestate can contain invalid nodeids. In that case,
`mergestate.localctx` or `mergestate.otherctx` will fail. This patch
provides a way of accessing the nodeid without failing in such cases.
Differential Revision: https://phab.mercurial-scm.org/D8040
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 22:24:16 -0800] rev 44288
rebase: define base in only place in defineparents()
Just a little refactoring to prepare for the next patch.
Differential Revision: https://phab.mercurial-scm.org/D7906
Martin von Zweigbergk <martinvonz@google.com> [Fri, 20 Dec 2019 16:16:57 -0800] rev 44287
tests: use full `uncommit` command name in tests
I'm about to add a `hg uncopy`, so the `hg unc` we used for `hg
uncommit` would become ambiguous.
Differential Revision: https://phab.mercurial-scm.org/D8028
Martin von Zweigbergk <martinvonz@google.com> [Tue, 28 Jan 2020 14:53:23 -0800] rev 44286
graft: default `base` argument to common case of `ctx.p1()`
I also updated the callers that wanted that, partly to simplify and
partly to show that it works.
Differential Revision: https://phab.mercurial-scm.org/D8027
Martin von Zweigbergk <martinvonz@google.com> [Fri, 10 Jan 2020 13:12:24 -0800] rev 44285
graft: let caller pass in overlayworkingctx to merge.graft()
Passing in a different `wctx` than `repo[None]` is useful because it
allows the caller to decide to not touch the working directory.
Differential Revision: https://phab.mercurial-scm.org/D8026
Martin von Zweigbergk <martinvonz@google.com> [Wed, 29 Jan 2020 23:14:31 -0800] rev 44284
copies: fix crash when copy source is not in graft base
Differential Revision: https://phab.mercurial-scm.org/D8046
Martin von Zweigbergk <martinvonz@google.com> [Wed, 29 Jan 2020 23:05:02 -0800] rev 44283
tests: add test showing crash when shelving ghosted rename target
When you `hg rename` a file and then delete the rename target, `hg
shelve` will give you a traceback.
Note that the shelve succeeds and the shelve is correct, it's just the
update to the parent that fails (i.e. to the parent of the commit that
was created for the shelve).
This can be squashed into the next commit if the reviewer prefers.
Differential Revision: https://phab.mercurial-scm.org/D8045
Yuya Nishihara <yuya@tcha.org> [Tue, 22 Oct 2019 16:04:34 +0900] rev 44282
rust-cpython: mark all PyLeaked methods as unsafe
Unfortunately, these methods can be abused to obtain the inner 'static
reference. The simplest (pseudo-code) example is:
let leaked: PyLeaked<&'static _> = shared.leak_immutable();
let static_ref: &'static _ = &*leaked.try_borrow(py)?;
// PyLeakedRef::deref() tries to bound the lifetime to itself, but
// the underlying data is a &'static reference, so the returned
// reference can be &'static.
This problem can be easily fixed by coercing the lifetime, but there are
many other ways to achieve that, and there wouldn't be a generic solution:
let leaked: PyLeaked<&'static [_]> = shared.leak_immutable();
let leaked_iter: PyLeaked<slice::Iter<'static, _>>
= unsafe { leaked.map(|v| v.iter()) };
let static_slice: &'static [_] = leaked_iter.try_borrow(py)?.as_slice();
So basically I failed to design the safe borrowing interface. Maybe we'll
instead have to add much more restricted interface on top of the unsafe
PyLeaked methods? For instance, Iterator::next() could be implemented if
its Item type is not &'a (where 'a may be cheated.)
Anyway, this seems not an easy issue, so it's probably better to leave the
current interface as unsafe, and get broader comments while upstreaming this
feature.
Yuya Nishihara <yuya@tcha.org> [Sat, 19 Oct 2019 17:01:28 +0900] rev 44281
rust-cpython: make PySharedRef::try_borrow_mut() return BorrowMutError
As I said, it shouldn't be an error of Python layer, but is something like
a coding error. Returning BorrowMutError makes more sense.
There's a weird hack to propagate the borrow-by-leaked state to RefCell
to obtain BorrowMutError. If we don't like it, maybe we can add our own
BorrowMutError.
Yuya Nishihara <yuya@tcha.org> [Sat, 19 Oct 2019 16:48:34 +0900] rev 44280
rust-cpython: inline PySharedState::leak_immutable() and PyLeaked::new()
For the same reason as the previous patch. The unsafe stuff can be better
documented if these functions are inlined.
Yuya Nishihara <yuya@tcha.org> [Sat, 19 Oct 2019 16:34:02 +0900] rev 44279
rust-cpython: inline PySharedState::try_borrow_mut()
Since the core borrowing/leaking logic has been moved to PySharedRef* and
PyLeaked*, it doesn't make sense that PySharedState had a function named
"try_borrow_mut". Let's turn it into a pure data struct.
Yuya Nishihara <yuya@tcha.org> [Sat, 12 Oct 2019 23:34:05 +0900] rev 44278
rust-cpython: add panicking version of borrow_mut() and use it
The original borrow_mut() is renamed to try_borrow_mut().
Since leak_immutable() no longer incref the borrow count, the caller should
know if the underlying value is borrowed or not. No Python world is involved.
That's why we can simply use the panicking borrow_mut().
Matt Harbison <matt_harbison@yahoo.com> [Tue, 28 Jan 2020 22:27:30 -0500] rev 44277
setup: don't skip the search for global hg.exe if there is no local instance
The point of trying not to blindly execute `hg` on Windows is that the local
hg.exe would be given precedence, and if py3 isn't on PATH, it errors out with a
modal dialog. But that's not a problem if there is no local executable that
could be run.
The problem that I recently ran into was I upgraded the repo format to use zstd.
But doing a `make clean` deletes all of the supporting libraries, causing the
next run to abort with a message about not understanding the
`revlog-compression-zstd` requirement. By getting rid of the local executable
in the previous commit when cleaning, we avoid leaving a broken executable
around, and avoid the py3 PATH problem too. There is still a small hole in that
`hg.exe` needs to be deleted before switching between py2/py3/PyOxidizer builds,
because the zstd module won't load. But that seems like good hygiene anyway.
Differential Revision: https://phab.mercurial-scm.org/D8038
Matt Harbison <matt_harbison@yahoo.com> [Tue, 28 Jan 2020 22:35:08 -0500] rev 44276
make: also delete hg.exe when cleaning
This will be needed for the next patch, which has more details. It has to come
before the call into setup.py because even `python setup.py clean` calls hg to
generate the version file.
Differential Revision: https://phab.mercurial-scm.org/D8037
Martin von Zweigbergk <martinvonz@google.com> [Thu, 23 Jan 2020 15:44:30 -0800] rev 44275
merge: start using the per-side copy dicts
The point of this patch is mostly to clarify `manifestmerge()`. I find
it much easier to reason about now.
Differential Revision: https://phab.mercurial-scm.org/D7990
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Jan 2020 14:35:30 -0800] rev 44274
copies: define a type to return from mergecopies()
We'll soon return two instances of many of the dicts from
`copies.mergecopies()`. That will mean that we need to return 9
different dicts, which is clearly not manageable. This patch instead
encapsulates the 4 dicts we'll duplicate in a new type. For now, we
still just return one instance of it (plus the separate `diverge`
dict).
Differential Revision: https://phab.mercurial-scm.org/D7989
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Jan 2020 16:45:56 -0800] rev 44273
merge: move initialization of copy dicts to one place
Differential Revision: https://phab.mercurial-scm.org/D7988
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 10:39:55 -0800] rev 44272
copies: print debug information about copies per side/branch
Differential Revision: https://phab.mercurial-scm.org/D7987
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Jan 2020 15:31:17 -0800] rev 44271
copies: make mergecopies() distinguish between copies on each side
I find it confusing that most of the dicts returned from
`mergecopies()` have entries specific to one branch of the merge, but
they're still combined into dict. For example, you can't tell if `copy
= {"bar": "foo"}` means that "foo" was copied to "bar" on the first
branch or the second.
It also feels like there are bugs lurking here because we may mistake
which side the copy happened on. However, for most of the dicts, it's
not possible that there is disagreement. For example, `renamedelete`
keeps track of renames that happened on one side of the merge where
the other side deleted the file. There can't be a disagreement there
(because we record that in the `diverge` dict instead). For regular
copies/renames, there can be a disagreement. Let's say file "foo" was
copied to "bar" on one branch and file "baz" was copied to "bar" on
the other. Beacause we only return one `copy` dict, we end up
replacing the `{"bar": "foo"}` entry by `{"bar": "baz"}`. The merge
code (`manifestmerge()`) will then decide that that means "both
renamed from 'baz'". We should probably treat it as a conflict
instead.
The next few patches will make `mergecopies()` return two instances of
most of the returned copies. That will lead to a bit more code (~40
lines), but I think it makes both `copies.mergecopies()` and
`merge.manifestmerge()` clearer.
Differential Revision: https://phab.mercurial-scm.org/D7986
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 17:25:40 -0800] rev 44270
pathutil: mark parent directories as audited as we go
Before 0b7ce0b16d8a (pathauditor: change parts verification order to
be root first, 2016-02-11), we used to validate child directories
before parents. It was then important to only mark the child audited
only after we had audited its parent (ancestors). I'm pretty sure we
don't need to do that any more, now that we audit parents before
children.
Differential Revision: https://phab.mercurial-scm.org/D8002
Martin von Zweigbergk <martinvonz@google.com> [Mon, 27 Jan 2020 09:14:19 -0800] rev 44269
cmdutil: change check_incompatible_arguments() *arg to single iterable
This makes it clearer on the call-sites that the first argument is
special. Thanks to Yuya for the suggestion.
Differential Revision: https://phab.mercurial-scm.org/D8018
Martin von Zweigbergk <martinvonz@google.com> [Mon, 27 Jan 2020 12:38:59 -0800] rev 44268
rust: remove an unnecessary set of parentheses
My build complained about this. I guess it started after I upgraded
rustc.
Differential Revision: https://phab.mercurial-scm.org/D8020
Kyle Lippincott <spectral@google.com> [Mon, 27 Jan 2020 18:16:05 -0800] rev 44267
profiling: flush stdout before writing profile to stderr
On py3, stdout and stderr appear to be buffered and this causes my command's
output to be intermixed with the profiling output.
Differential Revision: https://phab.mercurial-scm.org/D8024
Martin von Zweigbergk <martinvonz@google.com> [Tue, 28 Jan 2020 10:40:19 -0800] rev 44266
rust: re-format with nightly rustfmt
This fixes test-check-rust-format.t.
Differential Revision: https://phab.mercurial-scm.org/D8025
Matt Harbison <matt_harbison@yahoo.com> [Tue, 28 Jan 2020 22:03:00 -0500] rev 44265
tests: stablize test-rename-merge1.t on Windows
This goes with d7622fdec3b5.
Differential Revision: https://phab.mercurial-scm.org/D8036
Yuya Nishihara <yuya@tcha.org> [Sat, 21 Sep 2019 17:27:14 +0900] rev 44264
rust-cpython: make sure PySharedRef::borrow_mut() never panics
Since it returns a Result, it shouldn't panic depending on where the
borrowing fails.
PySharedRef::borrow_mut() will be renamed to try_borrow_mut() by the next
patch.
Yuya Nishihara <yuya@tcha.org> [Tue, 22 Oct 2019 11:38:43 +0900] rev 44263
rust-cpython: remove useless wrappers from PyLeaked, just move by map()
This series prepares for migrating to the upstreamed version of PySharedRef.
I found this last batch wasn't queued while rewriting the callers.
While Option<T> was historically needed, it shouldn't be required anymore.
I wasn't aware that each filed can be just moved.
Georges Racinet <georges.racinet@octobus.net> [Mon, 27 Jan 2020 20:28:47 +0100] rev 44262
rust-node: avoid meaningless read at the end of odd prefix
This should be heavily factored out by the CPU branch predictor
anyway.
Differential Revision: https://phab.mercurial-scm.org/D8019
Georges Racinet <georges.racinet@octobus.net> [Fri, 27 Dec 2019 16:06:54 +0100] rev 44261
rust-nodemap: generic NodeTreeVisitor
This iterator will help avoid code duplication when we'll
implement `insert()`, in which we will need to
traverse the node tree, and to remember the visited blocks.
The structured iterator item will allow different usages from
`lookup()` and the upcoming `insert()`.
Differential Revision: https://phab.mercurial-scm.org/D7794
Georges Racinet <georges.racinet@octobus.net> [Fri, 27 Dec 2019 15:11:43 +0100] rev 44260
rust-nodemap: mutable NodeTree data structure
Thanks to the previously indexing abstraction,
the only difference in the lookup algorithm is that we
don't need the special case for an empty NodeTree any more.
We've considered making the mutable root an `Option<Block>`,
but that leads to unpleasant checks and `unwrap()` unless we
abstract it as typestate patterns (`NodeTree<Immutable>` and
`NodeTree<Mutated>`) which seem exaggerated in that
case.
The initial copy of the root block is a very minor
performance penalty, given that it typically occurs just once
per transaction.
Differential Revision: https://phab.mercurial-scm.org/D7793
Georges Racinet <georges.racinet@octobus.net> [Thu, 26 Dec 2019 15:47:14 +0100] rev 44259
rust-nodemap: abstracting the indexing
In the forthcoming mutable implementation, we'll have to visit
node trees that are more complex than a single slice, although
the algorithm will still be expressed in simple indexing terms.
We still refrain using `#[inline]` indications as being
premature optimizations, but we strongly hope the compiler will
indeed inline most of the glue.
Differential Revision: https://phab.mercurial-scm.org/D7792
Georges Racinet <georges.racinet@octobus.net> [Thu, 23 Jan 2020 17:18:13 +0100] rev 44258
rust-nodemap: NodeMap trait with simplest implementation
We're defining here only a small part of the immutable
methods it will have at the end. This is so we can
focus in the following changesets on the needed abstractions
for a mutable append-only serializable version.
The first implementor exposes the actual lookup
algorithm in its simplest form. It will have to be expanded
to account for the missing methods, and the special cases
related to NULL_NODE.
Differential Revision: https://phab.mercurial-scm.org/D7791
Georges Racinet <georges.racinet@octobus.net> [Fri, 27 Dec 2019 23:04:18 +0100] rev 44257
rust-node: handling binary Node prefix
Parallel to the inner signatures of the nodetree functions in
revlog.c, we'll have to handle prefixes of `Node` in binary
form.
Another motivation is that it allows to convert from full Node
references to `NodePrefixRef` without copy. This is expected to
be by far the most common case in practice.
There's a slight complication due to the fact that we'll be sometimes
interested in prefixes with an odd number of hexadecimal digits,
which translates in binary form by a last byte in which only the
highest weight 4 bits are considered. This is totally transparent for
callers and could be revised once we have proper means to measure
performance.
The C implementation does the same, passing the length in nybbles as
function arguments. Because Rust byte slices already have a length, we carry
the even/odd informaton as a boolean, to avoid introducing logical
redundancies and the related potential inconsistency bugs.
There are a few candidates for inlining here, but we refrain from
such premature optimizations, letting the compiler decide.
Differential Revision: https://phab.mercurial-scm.org/D7790
Georges Racinet <georges.racinet@octobus.net> [Wed, 22 Jan 2020 16:35:56 +0100] rev 44256
rust-revlog: a trait for the revlog index
As explained in the doc comment, this is the minimum needed
for our immediate concern, which is to implement a nodemap
in Rust.
The trait will be later implemented in `hg-cpython` by the
index Python object implemented in C, thanks to exposition
of the corresponding functions as a capsule.
The `None` return cases in `node()` match what the `index_node()`
C function does.
Differential Revision: https://phab.mercurial-scm.org/D7789
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 17:10:45 -0800] rev 44255
pathauditor: drop a redundant call to bytes.lower()
`_lowerclean(s)` calls `s.lower()`, so we don't need to do that before
calling it.
Differential Revision: https://phab.mercurial-scm.org/D8001
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 15:18:19 -0800] rev 44254
merge: replace a repo.lookup('.') by more typical repo['.'].node()
The `repo.lookup('.')` form comes from b3311e26f94f (merge: fix
--preview to show all nodes that will be merged (issue2043).,
2010-02-15). I don't know why that commit changed from `repo['.']`,
but I don't think there's any reason to do that. Note that performance
should not be a reason (anymore?), because repo.lookup() is
implemented by first creating a context object.
Differential Revision: https://phab.mercurial-scm.org/D7998
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 16:07:42 -0800] rev 44253
merge: drop now-unused "abort" argument from hg.merge()
Differential Revision: https://phab.mercurial-scm.org/D7997
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 17:49:21 -0800] rev 44252
merge: don't auto-pick destination with `hg merge 'wdir()'`
If the user doesn't specify a commit to merge with, we'll have
`node==None` in `commands.merge()`. We'll then try to find a good
commit to merge with. However, if the user, for some strange reason,
runs `hg merge 'wdir()'`, we'll also have `node==None` and we'll do
that same. That's clearly not the intent, so let's not do that.
It turns out we'd instead crash on that command after this patch, so I
added special handling of it too.
Differential Revision: https://phab.mercurial-scm.org/D7996
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 16:05:11 -0800] rev 44251
merge: call hg.abortmerge() directly and return early
It's seem really weird to go through a lot of unrelated code before we
call `hg.merge(..., abort=True)` when we can just call
`hg.abortmerge()` and return early.
Differential Revision: https://phab.mercurial-scm.org/D7995
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 16:00:54 -0800] rev 44250
merge: check that there are no conflicts after --abort
Same idea as in abcc82bf0717 (clean: check that there are no conflicts
after, 2020-01-24). We should reuse more code here, but that will come
later.
Differential Revision: https://phab.mercurial-scm.org/D7994
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 15:07:44 -0800] rev 44249
merge: use check_incompatible_arguments() for --abort
Differential Revision: https://phab.mercurial-scm.org/D7993
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 17:15:45 -0800] rev 44248
rebase: move some variables after an error cases where they're not needed
Differential Revision: https://phab.mercurial-scm.org/D7905
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 10:44:23 -0800] rev 44247
rebase: clarify a little by calculating a set in Python instead of in revset
By calculating the set in Python, we can give it a name, which helps
readability.
Differential Revision: https://phab.mercurial-scm.org/D7904
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 15:12:50 -0800] rev 44246
merge: avoid a negation in the definition of updatedirstate
We only use `partial` in one place: the definition of
`updatedirstate`. Let's simplify that a little.
Differential Revision: https://phab.mercurial-scm.org/D7900
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 08:32:35 -0800] rev 44245
merge: move definition of `partial` closer to where it's used
Differential Revision: https://phab.mercurial-scm.org/D7983
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Jan 2020 13:06:56 -0800] rev 44244
copies: extract function for finding directory renames
The directory rename code is logically quite isolated, so it makes
sense to make it physically isolated as well.
Differential Revision: https://phab.mercurial-scm.org/D7977
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Jan 2020 15:23:30 -0800] rev 44243
copies: avoid calculating debug-only stuff without --debug
`renamedeleteset` and `divergeset` is only used with
`repo.ui.debugflag`, so let's avoid calculating them otherwise. While
at it, I also added a `del renamedeleteset` for consistency.
Differential Revision: https://phab.mercurial-scm.org/D7976
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Jan 2020 15:20:12 -0800] rev 44242
copies: move early return in mergecopies() earlier
It wasn't obvious that the early return happened only when there are
no copies. That is the case, however, because if `fullcopy` is empty,
then so is `copies1` and `copies2`, and then so is `inversecopies1`
and `inversecopies2`, and then so is `allsources`, and then so is
`copy`, `diverge` and `renamedelete`.
By moving the early return earlier, we also avoid calculating the set
of added files from the base to each side.
Differential Revision: https://phab.mercurial-scm.org/D7975
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 07:00:45 -0800] rev 44241
tests: test merge of renames of different sources to same target
This is a really obscure scenario, but let's still have it tested so
we know when it changes.
Differential Revision: https://phab.mercurial-scm.org/D7985
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 09:33:02 -0800] rev 44240
clean: check that there are no conflicts after
As noted by Pulkit, there should never be any conflicts after doing a
clean update, so `hg.clean()` should never return `True`. Let's check
that assertion instead to clarify the code. The callers will now get a
`None` instead of a `False` returned, but that should be fine (both
result in a 0 exit status).
Differential Revision: https://phab.mercurial-scm.org/D7984
Martin von Zweigbergk <martinvonz@google.com> [Fri, 24 Jan 2020 14:32:53 -0800] rev 44239
progress: delete deprecated ui.progress()
Differential Revision: https://phab.mercurial-scm.org/D7991
Raphaël Gomès <rgomes@octobus.net> [Fri, 17 Jan 2020 15:34:11 +0100] rev 44238
rust-dependencies: update rayon
This is just to make sure we use the latest version and also makes it easier
to peruse the docs.
Differential Revision: https://phab.mercurial-scm.org/D7926
Augie Fackler <augie@google.com> [Fri, 24 Jan 2020 11:05:42 -0500] rev 44237
merge with stable
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 15:08:42 -0800] rev 44236
merge: define updatedirstate a little earlier and reuse it
Differential Revision: https://phab.mercurial-scm.org/D7899
Martin von Zweigbergk <martinvonz@google.com> [Wed, 15 Jan 2020 15:07:43 -0800] rev 44235
merge: don't call update hook when using in-memory context
I'm pretty sure many hook implementors will assume that they can
inspect the working copy and/or dirstate parents when the hook is
called, so I don't think we should call the hook when using an
in-memory context. The new behavior matches that of the preupdate
hook.
Differential Revision: https://phab.mercurial-scm.org/D7898
Martin von Zweigbergk <martinvonz@google.com> [Thu, 23 Jan 2020 13:10:48 -0800] rev 44234
merge with stable
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 22 Jan 2020 20:01:38 -0800] rev 44233
packaging: add configparser to inno requirements file
This dependency is missing and pip complains about it in
strict hashing mode. How this was missed, I have no clue.
Differential Revision: https://phab.mercurial-scm.org/D7973
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 22 Jan 2020 22:23:04 -0800] rev 44232
python-zstandard: blacken at 80 characters
I made this change upstream and it will make it into the next
release of python-zstandard. I figured I'd send it Mercurial's
way because it will allow us to drop this directory from the black
exclusion list.
# skip-blame blackening
Differential Revision: https://phab.mercurial-scm.org/D7937
Martin von Zweigbergk <martinvonz@google.com> [Tue, 21 Jan 2020 15:45:06 -0800] rev 44231
tests: move non-collapse test out of test-rebase-collapse
The test case was added in 76630fbbf4fa (test-rebase-collapse: Add
test for rebase regression introduced in 12309c09d19a, 2012-01-23). I
think `hg rebase --collapse` was involved in either the regression or
in the fix that caused the regression, but the test that was added
doesn't use `--collapse`, so it doesn't seem to belong in
test-rebase-collapse.t. The test case is about copies, so I moved it
to test-rebase-rename.t.
Differential Revision: https://phab.mercurial-scm.org/D7968
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 22 Nov 2019 20:27:09 -0800] rev 44230
debugcommands: add Python implementation to debuginstall
This seems like a useful detail to print.
Differential Revision: https://phab.mercurial-scm.org/D7979
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 22 Nov 2019 20:12:10 -0800] rev 44229
run-tests: remove --py3-warnings
This Python 2 only mode was to help Python 2 alert when doing
things not supported on Python 3. Now that we have test coverage with
Python 3, I don't think we need it.
Differential Revision: https://phab.mercurial-scm.org/D7978
Georges Racinet <georges.racinet@octobus.net> [Wed, 22 Jan 2020 16:37:05 +0100] rev 44228
rust-node: binary Node ID and conversion utilities
The choice of type makes sure that a `Node` has the exact
wanted size. We'll use a different type for prefixes.
Added dependency: hexadecimal conversion relies on the
`hex` crate.
The fact that sooner or later Mercurial is going to need
to change its hash sizes has been taken strongly in
consideration:
- the hash length is a constant, but that is not directly
exposed to callers. Changing the value of that constant
is the only thing to do to change the hash length (even
in unit tests)
- the code could be adapted to support several sizes of hashes,
if that turned out to be useful. To that effect, only the
size of a given `Node` is exposed in the public API.
- callers not involved in initial computation, I/O and FFI
are able to operate without a priori assumptions on the hash
size. The traits `FromHex` and `ToHex` have not been directly
implemented, so that the doc-comments explaining these
restrictions would stay really visible in `cargo doc`
Differential Revision: https://phab.mercurial-scm.org/D7788
Georges Racinet <georges.racinet@octobus.net> [Wed, 22 Jan 2020 16:23:29 +0100] rev 44227
rust-nodemap: building blocks for nodetree structures
This is similar to `nodetreenode` in `revlog.c`. We give it
a higher level feeling for ease of handling in Rust context
and provide tools for tests and debugging.
The encoding choice is dictated by our ultimate goal in this
series, that is to make an append-only persistent version of
`nodetree`: the 0th Block must be adressed from other Blocks.
Differential Revision: https://phab.mercurial-scm.org/D7787
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 10:13:08 -0500] rev 44226
lfs: move the initialization of the upload request into the try block
This (almost) guarantees that the file is closed in the case of an exception.
The one hole is if the `seek(SEEK_END)`/`tell()`/`seek(0)` sequence fails. But
that's going to go away when subclassing `httpconnection.httpsendfile` to fix
the worker problem, so I'm not going to worry too much. (And that class appears
to have the same problem.)
Differential Revision: https://phab.mercurial-scm.org/D7959
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 09:55:35 -0500] rev 44225
lfs: drop an unnecessary r'' prefix
No longer necessary since the source transformer was removed.
# skip-blame for changing string prefixes
Differential Revision: https://phab.mercurial-scm.org/D7958
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 09:51:39 -0500] rev 44224
lfs: explicitly close the file handle for the blob being uploaded
The previous code relied on reading the blob fully to close it. The obvious
problem is if an error occurs before that point. But there is also a problem
when using workers where the data may need to be re-read, which can't happen
once it is closed. This eliminates the surprising behavior before attempting to
fix the worker problem.
Differential Revision: https://phab.mercurial-scm.org/D7957
Matt Harbison <matt_harbison@yahoo.com> [Tue, 21 Jan 2020 09:40:40 -0500] rev 44223
lfs: drop the unused progressbar code in the `filewithprogress` class
This has been unused since f98fac24b757, which added worker based transfers for
concurrency, shifting the progressbar maintenance to the single thread waiting
on the worker to complete. Since the name no longer fits, rename the class.
Differential Revision: https://phab.mercurial-scm.org/D7956
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 16:58:07 +0100] rev 44222
rust-filepatterns: remove bridge code for filepatterns-related functions
These functions will be used internally by `hg-core` without needed to be
exposed to Python.
Differential Revision: https://phab.mercurial-scm.org/D7868
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 18:03:28 +0100] rev 44221
rust-utils: add Rust implementation of Python's "os.path.splitdrive"
I also wrote the NT version although I didn't mean to at first, so I thought
I would keep it, so that any further effort to get the Rust code working on
Windows is a little easier.
Differential Revision: https://phab.mercurial-scm.org/D7864
Alexander Pyhalov <apyhalov@gmail.com> [Mon, 13 Apr 2020 16:30:13 +0300] rev 44220
setup: link osutil.so to libsocket on Solaris/illumos (issue6299)
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 06 Apr 2020 00:24:57 +0200] rev 44219
discovery: avoid wrongly saying there are nothing to pull
We can get in a situation where a revision passed through `hg pull --rev REV`
are available on the server, but not a descendant of the advertised server
heads.
For example the server could lying be during heads advertisement, to hide some
pull request. Or obsolete/hidden content could be explicitly pulled.
So in this case the lookup associated to `REV` returned successfully, but the
normal discovery will find all advertised heads already known locally. This flip
a special boolean `anyinc` that will prevent any fetch attempt, preventing `REV`
to be pulled over.
We add three line of code to detect this case and make sure a pull actually
happens.
My main target is to make some third party extensions happy (I expect the
associated test to move upstream with the extension). However this fix already
make some of the `infinitepush` test happier.
Augie Fackler <raf@durin42.com> [Thu, 02 Apr 2020 12:05:41 -0400] rev 44218
Added signature for changeset 8fca7e8449a8
Augie Fackler <raf@durin42.com> [Thu, 02 Apr 2020 12:05:40 -0400] rev 44217
Added tag 5.3.2 for changeset 8fca7e8449a8
Kyle Lippincott <spectral@google.com> [Wed, 01 Apr 2020 14:14:55 -0700] rev 44216
histedit: add missing b prefix to a string
If i18n is disabled (such as via HGPLAIN=1), `_()` doesn't convert from str to
bytes, so this raises a TypeError on py3.
Differential Revision: https://phab.mercurial-scm.org/D8354
Martin von Zweigbergk <martinvonz@google.com> [Wed, 25 Mar 2020 18:50:40 -0700] rev 44215
py3: require values in changelog extras to be bytes
I don't know what happened here because b436059c1cca (py3: use
pycompat.bytestr() on extra values because it can be int, 2019-02-05)
came about b44a47214122 (py3: use string for "close" value in commit
extras, 2018-02-11). Whatever happened, we shouldn't need to convert
the values to bytes now. It's better to not convert because that might
cover up bugs where someone sets a unicode value in the extras and
that works until the unicode value happens to contain non-ascii (at
which point it will fail because `bytestr()` expects its argument to
be ascii if it's unicode).
Differential Revision: https://phab.mercurial-scm.org/D8332
Martin von Zweigbergk <martinvonz@google.com> [Wed, 25 Mar 2020 18:25:58 -0700] rev 44214
py3: make setup.py's hgcommand() consistently return bytes
Before this patch, it returned unicode when the command failed. That
made e.g. `make local PYTHON=python3` fail on an obsolete commit.
Differential Revision: https://phab.mercurial-scm.org/D8331
Kyle Lippincott <spectral@google.com> [Mon, 23 Mar 2020 14:38:00 -0700] rev 44213
darwin: use vim, not vi, to avoid data-loss inducing posix behavior
Apple's version of vim, available at
opensource.apple.com/release/macos-1015.html (for Catalina, but this behavior
has been there for a while) has several tweaks from the version of vim from
vim.org. Most of these tweaks appear to be for "Unix2003" compatibility.
One of the tweaks is that if any ex command raises an error, the entire
process will (when you exit, possibly minutes/hours later) also exit non-zero.
Ex commands are things like `:foo`.
Luckily, they only enabled this if vim was executed (via a symlink or copying
the binary) as `vi` or `ex`. If you start it as `vim`, it doesn't have this
behavior, so let's do that.
To see this in action, run the following two commands on macOS:
```
$ vi -c ':unknown' -c ':qa' ; echo $?
1
$ vim -c ':unknown' -c ':qa' ; echo $?
0
```
We don't want to start ignoring non-zero return types from the editor because
that will mean you can't use `:cquit` to intentionally exit 1 (which,
shows up as 2 if you combine an ex command error and a cquit, but only a 1 if
you just use cquit, so we can't differentiate between the two statuses). Since
we can't differentiate, we have to assume that all non-zero exit codes are
intentional and an indication of the user's desire to not continue with whatever
we're doing. If this was a complicated `hg split` or `hg histedit`, this is
especially disastrous :(
Differential Revision: https://phab.mercurial-scm.org/D8321
Matt Harbison <matt_harbison@yahoo.com> [Fri, 20 Mar 2020 10:04:13 -0400] rev 44212
cext: move variable declaration to the top of the block for C89 support
Not sure if we still care about C89 in general, but MSVC requires this style
too.
Differential Revision: https://phab.mercurial-scm.org/D8304
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 18 Mar 2020 21:27:45 +0100] rev 44211
byteify-string: resolve symlink before byteifying
Otherwise the script turns symlinks into regular files.
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 12 Mar 2020 15:23:47 +0100] rev 44210
cext-index: propagate inline_scan error in `index_deref`
Before this change, revlog index corruption could be silently ignored in some
situation.
Differential Revision: https://phab.mercurial-scm.org/D8276
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 12 Mar 2020 18:53:37 +0100] rev 44209
heptapod-ci: fix test paths in the listing file
Now what we run the test from the root, we need to list test name from the root.
Differential Revision: https://phab.mercurial-scm.org/D8275
Raphaël Gomès <rgomes@octobus.net> [Fri, 06 Mar 2020 10:38:37 +0100] rev 44208
hg-core: add a compilation error if trying to compile outside of Linux
For now, we can only provide support for Linux in `hg-core`, so we have to be
explicit about it in case anyone wonders why their Dirstate is suddenly broken,
or why the crate does not compile (on Windows for example).
Differential Revision: https://phab.mercurial-scm.org/D8246
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 10 Mar 2020 18:54:44 +0100] rev 44207
gzip: use the stdlib version with python 3 (issue6284)
It turned out that the stdlib gained the feature we missed in python 3.1. We can
now use it directly.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 10 Mar 2020 18:53:19 +0100] rev 44206
gzip: indent the custom Gzip code
We need a new conditional in the next changesets (Adding 3.8 support). We do the
large code move beforehand for clarity.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 10 Mar 2020 18:57:49 +0100] rev 44205
gzip: rename the argument to `mtime` to match upstream python
Python gained the feature we missed in 3.1. The argument name is `mtime`. We
align our version for consistency.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 10 Mar 2020 16:49:38 +0100] rev 44204
heptapod-ci: run the test from outside the test directory
This will help detecting case where this is broken.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 10 Mar 2020 16:47:02 +0100] rev 44203
run-tests: fix conditional when tests are run outside of `tests`
The logic to find `hghave` was using `__file__` but… non absolute path in
`__file__` gets wrong as soon as we change directory (thanks python… you are
being helpful). The rest of the `run-tests.py` code already deal with this
fine. We simply reuse the `RUNTESTDIR` variable to fix it.
Manuel Jacob <me@manueljacob.de> [Mon, 09 Mar 2020 01:11:59 +0100] rev 44202
tests: fix isinstance test of wrong variable
3086a8627b2970cd7fbf49bc69413c08c68d5514 changed self._case to be a list, but
it was forgotten to adjust this line.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 06 Mar 2020 23:27:28 +0100] rev 44201
discovery: avoid wrong detection of multiple branch heads (issue6256)
This fix the code using obsolescence markers to remove "to be obsoleted" heads
during the detection of new head creation from push. The code turned out to not
use the branch information at all. This lead changeset from different branch to
be detected as new head on unrelated branch.
The code fix is actually quite simple. New tests have been added to covers
these cases.
Differential Revision: https://phab.mercurial-scm.org/D8259
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 06 Mar 2020 00:28:09 +0100] rev 44200
test: cleanly skip test-remotefilelog-datapack.py on policy that breaks it
That tests requires the pure module to be available.
Differential Revision: https://phab.mercurial-scm.org/D8245
Raphaël Gomès <rgomes@octobus.net> [Thu, 05 Mar 2020 09:26:45 +0100] rev 44199
rust-format: cleanup ancestors.rs to make rustfmt happy
Differential Revision: https://phab.mercurial-scm.org/D8224
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 05 Mar 2020 15:47:22 +0100] rev 44198
heptapod-ci: use strict module policy
Without this, test can silently fallback to other compatible policy in some
cases.
Differential Revision: https://phab.mercurial-scm.org/D8234
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Mar 2020 08:30:11 -0800] rev 44197
histedit: fix formatting after D8150
Differential Revision: https://phab.mercurial-scm.org/D8230
Raphaël Gomès <rgomes@octobus.net> [Thu, 05 Mar 2020 17:17:02 +0100] rev 44196
rust-format: make rustfmt happy
Differential Revision: https://phab.mercurial-scm.org/D8228
Augie Fackler <raf@durin42.com> [Wed, 04 Mar 2020 11:51:13 -0500] rev 44195
Added signature for changeset 6d121acbb82e
Augie Fackler <raf@durin42.com> [Wed, 04 Mar 2020 11:51:12 -0500] rev 44194
Added tag 5.3.1 for changeset 6d121acbb82e
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 19 Feb 2020 13:39:00 +0530] rev 44193
remotefilelog: add 'changelog' arg to shallowcg1packer.generate (issue6269)
This cause traceback on widening using narrow extension when remotefilelog
is enabled.
Differential Revision: https://phab.mercurial-scm.org/D8134
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 28 Feb 2020 11:22:47 +0100] rev 44192
remotefilelog-test: glob some flaky output line
This is similar to ee0959e7d435. The affected line is flaky underload, yet the
final result is correct. The command involves background pre-check of output,
these are not stable probably because they run in parallel in multiple process.
If it become useful to start testing precise internal details of the, they will
have to be tested in a more appropriate framework than `.t` tests.
Differential Revision: https://phab.mercurial-scm.org/D8179
Steve Fink <sfink@mozilla.com> [Tue, 25 Feb 2020 13:23:37 -0800] rev 44191
histedit: py3 fixes for curses mode
Differential Revision: https://phab.mercurial-scm.org/D8150
Manuel Jacob <me@manueljacob.de> [Sun, 01 Mar 2020 19:39:23 +0100] rev 44190
branch: make --force work even when specifying revs
The `hg branch` command accepts a `--force` parameter that allows to
"set branch name even if it shadows an existing branch". However, before this
patch, that didn’t work when specifying revs with `-r`.
Matt Harbison <matt_harbison@yahoo.com> [Mon, 24 Feb 2020 00:20:47 -0500] rev 44189
setup: exclude the __index__ module from itself when generating
This module is generated on Windows to hold all of the extension names and the
help summaries, so that they are discoverable inside the py2exe zipfile. The
problem is this file is generated by dumping the disabled list, and that list
comes from walking the filesystem. So once an install from source into a
virtualenv created this module, then next build from source from that virtualenv
would also see __index__.py in the filesystem, and include it. Clearly that's
wrong because this isn't a real extension, so just filter it from the list when
generating it.
The Mercurial installer was unaffected by this, but the TortoiseHg package was.
In the final package, `hg help -v extensions` and the panel of extensions both
showed it.
Differential Revision: https://phab.mercurial-scm.org/D8142
Matt Harbison <matt_harbison@yahoo.com> [Mon, 24 Feb 2020 16:33:10 -0500] rev 44188
tests: stabilize test-amend.t on Windows
If $TESTTMP isn't quoted in this context, it ends up like
`C:Temphgtests.pikkoxchild1test-amend.t-obsstore-off`.
Differential Revision: https://phab.mercurial-scm.org/D8144
Yuya Nishihara <yuya@tcha.org> [Mon, 24 Feb 2020 13:28:49 +0900] rev 44187
py3: fix EOL detection in commandserver.channeledinput
This breaks TortoiseHg's email preview which sends b'\n' while readline
request is issued and the loop never ends. Spotted by Matt Harbison.
Kyle Lippincott <spectral@google.com> [Thu, 20 Feb 2020 16:21:00 -0800] rev 44186
shelve: fix ordering of merge labels
Differential Revision: https://phab.mercurial-scm.org/D8140
Kyle Lippincott <spectral@google.com> [Thu, 20 Feb 2020 17:06:01 -0800] rev 44185
shelve: add test clearly demonstrating that the conflict labels are backwards
Differential Revision: https://phab.mercurial-scm.org/D8139
Matt Harbison <matt_harbison@yahoo.com> [Sun, 16 Feb 2020 17:05:18 -0500] rev 44184
import: don't ignore `--secret` when `--bypass` is specified
Differential Revision: https://phab.mercurial-scm.org/D8126
Matt Harbison <matt_harbison@yahoo.com> [Tue, 18 Feb 2020 13:46:10 -0500] rev 44183
phabricator: fix a phabsend crash when processing a renamed binary
This was a trivial fix, and some more tests are added to cover binary files.
Since the old filecontext is passed in, the old name is still available. But I
noticed some weirdness around what it marked as binary and not, and what is
viewable in Phabricator. Those things have been flagged, and will probably take
some digging.
Differential Revision: https://phab.mercurial-scm.org/D8133
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 13 Dec 2019 10:37:45 +0100] rev 44182
test: pin the number of CPU for issue4074 tests
On machine with an hundreds of CPUs, the "user" CPU time reported can be
inflated by the status steps. Since the test especially focus on the diff
computation, we restrict the number of CPU to avoid potential issues.
Differential Revision: https://phab.mercurial-scm.org/D8112
Raphaël Gomès <rgomes@octobus.net> [Wed, 12 Feb 2020 23:23:59 +0100] rev 44181
rust-dirstatemap: add `NonNormalEntries` class
This fix introduces the same encapsulation as the `copymap`. There is no easy
way of doing this any better for now.
`hg up -r null && time HGRCPATH= HGMODULEPOLICY=rust+c hg up tip` on Mozilla
Central, (not super recent, but it doesn't matter):
Before: 7:44,08 total
After: 1:03,23 total
Pretty brutal regression!
This is a graft on stable of cf1f8660e568
Differential Revision: https://phab.mercurial-scm.org/D8111
Raphaël Gomès <rgomes@octobus.net> [Thu, 30 Jan 2020 14:57:02 +0100] rev 44180
rust-dirstatemap: cache non normal and other parent set
Performance of `hg update` was significantly worse since the introduction of
the Rust `dirstatemap`. This regression was noticed by Valentin Gatien-Baron
when working on a large repository, as it goes unnoticed for smaller
repositories like Mercurial itself.
This fix introduces the same getter/setter mechanism at `hg-core` level as
for `set/get_dirs`.
While this technique is, as previously discussed, quite suboptimal, it fixes an
important enough problem. Refactoring `hg-core` to use the typestate
pattern could be a good approach to improving code quality in a future patch.
This is a graft of stable of 83b2b829c94e
Differential Revision: https://phab.mercurial-scm.org/D8110
Yuya Nishihara <yuya@tcha.org> [Tue, 11 Feb 2020 19:53:56 +0900] rev 44179
chgserver: spawn new process if schemes change
The schemes extension updates hg.schemes table. It's technically possible
for hg.repository() to look for e.g. ui.schemes instead of depending on
module-local table, but I don't think the change would make much sense
since [schemes] is usually specified in ~/.hgrc and thus it can be considered
static data.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 10 Feb 2020 15:52:52 -0800] rev 44178
tests: accept new bzr message about switching branches
The new version apparently prints "Switched to branch at " instead of
"Switched to branch: ".
Differential Revision: https://phab.mercurial-scm.org/D8106
Martin von Zweigbergk <martinvonz@google.com> [Mon, 10 Feb 2020 15:50:26 -0800] rev 44177
tests: add workaround for bzr bug
This started failing for me today. I guess my bzr was upgraded.
Differential Revision: https://phab.mercurial-scm.org/D8105
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 01 Feb 2020 09:14:36 +0100] rev 44176
test: simplify test-amend.t to avoid race condition
Insted on relying on sleep, we could simply have the editor do the file change.
This remove the reliance on "sleep" and avoid test failing on heavy load
machine.
To test this, I reverted the code change in 5558e3437872 and the test started
failing again.
This is a graft on stable of 141ceec06b55 which should have targeted for stable.
Differential Revision: https://phab.mercurial-scm.org/D8103
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 09 Feb 2020 01:34:37 +0100] rev 44175
remotefilelog-test: glob some flaky output line (issue6083)
The two following lines are flaky underload, yet the final result is correct.
The command involves background pre-check of output, these are not stable
probably because they run in parallel in multiple process.
I spent a couple of hours trying to understand the pattern and gave up. The
documented intend of these tests is safely guaranteed by checking the cache
content after the command.
If it become useful to start testing precise internal details of the, they will
have to be tested in a more appropriate framework than `.t` tests.
Differential Revision: https://phab.mercurial-scm.org/D8102
Augie Fackler <augie@google.com> [Mon, 03 Feb 2020 11:56:02 -0500] rev 44174
resourceutil: blacken
Kyle Lippincott <spectral@google.com> [Thu, 06 Feb 2020 15:46:55 -0800] rev 44173
py3: fully fix bundlepart.__repr__ to return str not bytes
My previous fix did not fully fix the issue: it would attempt to use
%-formatting to combine two strs into a bytes, which won't work. Let's just
switch the entire function to operating in strs. This can cause a small output
difference that will likely not be noticed since no one noticed that the method
wasn't working at all before: if `id` or `type` are not-None, they'll be shown
as `b'val'` instead of `val`. Since this is a debugging aid and these strings
shouldn't be shown to the user, slightly rough output is most likely fine and
it's likely not worthwhile to add the necessary conditionals to marginally
improve it.
Differential Revision: https://phab.mercurial-scm.org/D8091
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 17 Nov 2019 01:18:14 +0100] rev 44172
heptapod-ci: add a job to test the rust version of Mercurial
The rust version of Mercurial is not currently tested by anything else. So it
get quite important that developer runs it.
Differential Revision: https://phab.mercurial-scm.org/D8017
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 16 Nov 2019 12:26:54 +0100] rev 44171
heptapod-ci: run the --pure test too
These are usually rarely run by individual developper because they are slow.
However it is important that they stay happy.
Differential Revision: https://phab.mercurial-scm.org/D8016
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 25 Jan 2020 14:56:36 +0100] rev 44170
heptapod-ci: run the normal test suite
The usual tests should be run too. We skip the "tests-check*.t" one because
their are already covered by another Ci step.
Differential Revision: https://phab.mercurial-scm.org/D8015
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 18 Nov 2019 09:38:40 +0100] rev 44169
heptapod-ci: also run the dedicated rust test for the rust code
The Rust code has various standard rust test that are fast to run. So let's run them.
Differential Revision: https://phab.mercurial-scm.org/D8014
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 16 Nov 2019 12:25:53 +0100] rev 44168
heptapod-ci: run test with python3 too
Python3 is the future^W present, it is important to run tests with it too.
Differential Revision: https://phab.mercurial-scm.org/D8013
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 24 Jan 2020 23:22:29 +0100] rev 44167
heptapod-ci: colorize output
The run result are nicer to read with color.
Differential Revision: https://phab.mercurial-scm.org/D8012
Pierre-Yves David <pierre-yves.david@octobus.net> [Sat, 25 Jan 2020 17:57:40 +0100] rev 44166
heptapod-ci: add a basic file to be able to run tests with heptapod
Having this yaml file somewhere in the main mercurial repository makes it
trivial for contributors using heptapod to run CI on their in-progress work.
There are alot of different combination (python2/python3 pure/cext/rust/pypy)
to be tested and making sure all of them are covered manually is cumbersome.
Automatic CI runnig on draft really helps in that matters. We start small bu
later changesets will add more step testing more of the variants.
The series is targetted on stable to make it available to the widest amount of contribution possible.
The definition of the docker files used for this are available here:
https://dev.heptapod.net/octobus/ci-dockerfiles
Differential Revision: https://phab.mercurial-scm.org/D8011
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com> [Tue, 04 Feb 2020 22:07:36 +0100] rev 44165
worker: manually buffer reads from pickle stream
My previous fix (D8051, cb52e619c99e, which added Python's built-in buffering
to the pickle stream) has the problem that the selector will ignore the buffer.
When multiple pickled objects are read from the pipe into the buffer at once,
only one object will be loaded.
This can repeat until the buffer is full and delays the processing of completed
items until the worker exits, at which point the pipe is always considered
readable and all remaining items are processed.
This changeset reverts D8051, removing the buffer again. Instead, on Python 3
only, we use a wrapper to modify the "read" provided to the Unpickler to behave
more like a buffered read. We never read more bytes from the pipe than the
Unpickler requests, so the selector behaves as expected.
Also add a test case for "pickle data was truncated" issue.
https://phab.mercurial-scm.org/D8051#119193
Differential Revision: https://phab.mercurial-scm.org/D8076
Kyle Lippincott <spectral@google.com> [Thu, 02 Jan 2020 11:04:18 -0800] rev 44164
py3: __repr__ needs to return str, not bytes
Differential Revision: https://phab.mercurial-scm.org/D8089
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 04 Feb 2020 12:07:37 +0100] rev 44163
config: also respect HGRCSKIPREPO in the zeroconf extension
Differential Revision: https://phab.mercurial-scm.org/D8075
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 04 Feb 2020 12:07:42 +0100] rev 44162
config: also respect HGRCSKIPREPO in hgwebdir_mod
Differential Revision: https://phab.mercurial-scm.org/D8074
Pierre-Yves David <pierre-yves.david@octobus.net> [Mon, 03 Feb 2020 20:41:11 +0100] rev 44161
config: also respect HGRCSKIPREPO in `dispatch._getlocal`
For some reason, we are also reading the local config in that function.
Differential Revision: https://phab.mercurial-scm.org/D8073
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 04 Feb 2020 12:31:19 +0100] rev 44160
config: add a function in `rcutil` to abstract HGRCSKIPREPO
We wil need to respect this environment variable in more place.
Differential Revision: https://phab.mercurial-scm.org/D8072
Matt Harbison <matt_harbison@yahoo.com> [Mon, 03 Feb 2020 20:12:47 -0500] rev 44159
packaging: make the path to Win32 requirements absolute when building WiX
Otherwise this broke automation when not launched from `contrib/packaging`.
Differential Revision: https://phab.mercurial-scm.org/D8068
Augie Fackler <raf@durin42.com> [Mon, 03 Feb 2020 11:07:34 -0500] rev 44158
Added signature for changeset 7f5410dfc8a6
Augie Fackler <raf@durin42.com> [Mon, 03 Feb 2020 11:07:33 -0500] rev 44157
Added tag 5.3 for changeset 7f5410dfc8a6
Raphaël Gomès <rgomes@octobus.net> [Wed, 29 Jan 2020 11:11:18 +0100] rev 44156
rust-dirstatemap: add missing @propertycache
While investigating a regression on `hg update` performance introduced by the
Rust `dirstatemap`, two missing `@propertycache` were identified when comparing
against the Python implementation. This adds back the first one, that has
no observable impact on behavior. The second one (`nonnormalset`) is going to
be more involved, as the caching has to be done from the Rust side of things.
Differential Revision: https://phab.mercurial-scm.org/D8047
Jan Alexander Steffens (heftig) <jan.steffens@gmail.com> [Thu, 30 Jan 2020 19:16:12 +0100] rev 44155
worker: Use buffered input from the pickle stream
On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError:
pickle data was truncated") when it gets a short read, i.e. it receives fewer
bytes than it requested.
On our build machine, Mercurial seems to frequently hit this problem while
updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy
to trigger with:
#wipe the workdir
rm -rf *
hg update null
chrt -b 0 hg update default
I've also written the following program, which demonstrates the core problem:
from __future__ import print_function
import io
import os
import pickle
import time
obj = {"a": 1, "b": 2}
obj_data = pickle.dumps(obj)
assert len(obj_data) > 10
rfd, wfd = os.pipe()
pid = os.fork()
if pid == 0:
os.close(rfd)
for _ in range(4):
time.sleep(0.5)
print("First write")
os.write(wfd, obj_data[:10])
time.sleep(0.5)
print("Second write")
os.write(wfd, obj_data[10:])
os._exit(0)
try:
os.close(wfd)
rfile = os.fdopen(rfd, "rb", 0)
print("Reading")
while True:
try:
obj_copy = pickle.load(rfile)
assert obj == obj_copy
except EOFError:
break
print("Success")
finally:
os.kill(pid, 15)
The program reliably fails with Python 3.8 and succeeds with Python 2.7.
Providing the unpickler with a buffered reader fixes the issue, so let
"os.fdopen" create one.
https://bugzilla.mozilla.org/show_bug.cgi?id=1604486
Differential Revision: https://phab.mercurial-scm.org/D8051
Matt Harbison <matt_harbison@yahoo.com> [Sat, 01 Feb 2020 01:32:28 -0500] rev 44154
packaging: lowercase the `contrib` and `templates` directories with Inno
I have no idea why these (and `contrib/vim`) were leading with uppercase with
Inno, but not WiX. It probably doesn't matter too much, but might be a problem
with `templates` if the user enabled case sensitivity on NTFS.
Differential Revision: https://phab.mercurial-scm.org/D8063
Matt Harbison <matt_harbison@yahoo.com> [Sun, 02 Feb 2020 00:56:40 -0500] rev 44153
packaging: merge the requirements.txt files for WiX and Inno
Now that the content is common, there's no need to have separate files. The
content still differs from the non-Windows platforms though.
Differential Revision: https://phab.mercurial-scm.org/D8066
Matt Harbison <matt_harbison@yahoo.com> [Sat, 01 Feb 2020 00:58:34 -0500] rev 44152
packaging: bundle dulwich, keyring, and pywin32-ctypes with WiX too
TortoiseHg installs these, which is possibly where they originated (though I
would have thought it more likely to be in the WiX installer, given its
heritage). When I was working on the TortoiseHg app for Mac (which uses the
similar `py2app`), it wasn't possible to use the keyring extension (even
externally) without bundling this keyring package into the app. Assuming the
same principle applies here, these would enable some common extensions. One of
the things that the TortoiseHg packager on macOS does now is it adds the user's
local `site-packages` directory to `sys.path`. That would allow the user to
install these critical modules in cases like this. But that can probably wait
for py3 packaging.
The only difference in the installed packages that I see now is WiX also bundles
distutils for some reason. I suppose that's not harming anything, so I'm not
touching it.
The only orphans in the install directories when comparing WiX and Inno now is
the Copying.txt vs COPYING.rtf, the two uninstaller files for Inno, and a
`Mercurial.url` file in Inno. I have no idea what that is, and it has *.ini
syntax with a single field pointing to the Mercurial homepage.
Differential Revision: https://phab.mercurial-scm.org/D8062
Matt Harbison <matt_harbison@yahoo.com> [Sat, 01 Feb 2020 00:48:08 -0500] rev 44151
packaging: bundle the default mercurial.ini template with Inno also
This is a step towards converging on the same installer content on Windows.
Differential Revision: https://phab.mercurial-scm.org/D8061
Matt Harbison <matt_harbison@yahoo.com> [Sat, 01 Feb 2020 00:41:37 -0500] rev 44150
packaging: set the FileVersion field in the Inno installer executable
Previously, Properties > Details > File version showed "0.0.0.0". This appears
to be a longstanding issue, and not part of the refactoring this cycle.
Differential Revision: https://phab.mercurial-scm.org/D8060
Matt Harbison <matt_harbison@yahoo.com> [Sat, 01 Feb 2020 00:32:46 -0500] rev 44149
packaging: move the version normalization function to the util module
This will be used with Inno as well. Since this module isn't platform specific,
rename to include that this is meant for Windows. (Mac has a different format.)
Differential Revision: https://phab.mercurial-scm.org/D8059
Matt Harbison <matt_harbison@yahoo.com> [Fri, 31 Jan 2020 22:20:39 -0500] rev 44148
resourceutil: account for the non-resource-like file hierarchy under py2exe
After 9e367157a990, config files for py2exe were expected to be in
C:\Program Files\Mercurial\mercurial\defaultrc because of the implied resource
structure of 'mercurial.defaultrc.*.rc', relative to the executable.
Accomodating this would require changes to the WIX and Inno scripts (and perhaps
the script that generates the WIX script), as well as 3rd party bundlers like
TortoiseHg. But these files aren't read as resources anyway- they fall back to
the filesystem APIs. (If we really wanted to carry on the charade, the
installer would have to also sprinkle various empty __init__.py files around.)
Instead, this simply prunes the 'mercurial.' portion of the resource name when
run with py2exe. (PyOxidizer uses the resources API, not the filesystem
fallback, so it is unaffected.) Since this hack only affects the py2 Windows
installers and is less risky, I think it's reasonable. We haven't needed to
load any 3rd party resource up to this point, and would have to make packaging
changes anyway to handle that.
Differential Revision: https://phab.mercurial-scm.org/D8058
Matt Harbison <matt_harbison@yahoo.com> [Thu, 30 Jan 2020 19:37:06 -0500] rev 44147
wix: restore COPYING.rtf
This got truncated to 0 bytes in 0ab651b5f77c when the Phabricator extension
crashed because it's a binary file. That caused the license page in the WIX
installer to be empty. I don't remember if I needed to resubmit after the bug
was fixed, so let's try this again with the current stable. If this fails, I'll
retry with 5.1 to see if this is a regression in the API changeover last cycle.
Differential Revision: https://phab.mercurial-scm.org/D8052
Matt Harbison <matt_harbison@yahoo.com> [Thu, 30 Jan 2020 23:48:45 -0500] rev 44146
resourceutil: correct the root path for file based lookup under py2exe
This silly copy/paste error caused "Mercurial" to be truncated from
"C:\Program Files". The fact that "helptext" and "defaultrc" are now in a
subpackage of "mercurial" added it back on, and everything seemed to work. But
that broke if not installed to the default directory, and also caused TortoiseHg
to look at Mercurial's config files instead of its own.
Differential Revision: https://phab.mercurial-scm.org/D8054
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 24 Jan 2020 20:27:59 -0800] rev 44145
wix: use original version string for MSI filename
Version string normalization is mostly to placate MSI requirements.
I think it makes sense to use the original version string in
filenames.
Since we can have distinct versions normalizing to the same MSI
version string, this will allow us to distinguish between different
actual version strings based on the filename.
Differential Revision: https://phab.mercurial-scm.org/D8005
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 24 Jan 2020 20:24:29 -0800] rev 44144
wix: always normalize version string
Before, it was possible to pass in a custom version string
which would not be valid in MSI. So we always normalize the
version string.
While we're here, also print when we normalize the version string,
for better visibility.
Differential Revision: https://phab.mercurial-scm.org/D8004
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 24 Jan 2020 20:21:53 -0800] rev 44143
wix: more robust normalization of RC version components
MSI has strict version requirements where the format is
`A.B.C[.D]` and all fields must be numeric
(https://docs.microsoft.com/en-us/windows/win32/msi/productversion?redirectedfrom=MSDN).
Only the first 3 are used by the installer itself.
Mercurial's version strings can have `rcN` and an optional
`+<commit>-<date>` fragment at the end.
This commit teaches the MSI version normalization to handle
both of these more robustly.
Before, we would throw away the `.rcN` component completely.
e.g. `5.3rc1` would get normalized to `5.3.0`. And worse,
`5.3rc0+5-abcdef` would get normalized to `5.3.5`.
After this commit, presence of an `.rcN` provides the
value for a 4th field. e.g. `5.3rc1` -> `5.3.0.1`. In
addition, the commit count from the `+` suffix gets
normalized into the 4th version component, but only if
the original version string didn't have a 4th version
component or if no `rcN` is present. e.g. `5.3+5-abcdef`
is `5.3.0.5`.
Differential Revision: https://phab.mercurial-scm.org/D8003
Matt Harbison <matt_harbison@yahoo.com> [Sat, 25 Jan 2020 00:16:04 -0500] rev 44142
copyright: update to 2020
Differential Revision: https://phab.mercurial-scm.org/D8006
Matt Harbison <matt_harbison@yahoo.com> [Sat, 25 Jan 2020 01:06:46 -0500] rev 44141
phabricator: fix a crash when submitting binaries (issue6260)
I think this assumed that `p1()` returned the changectx instead of the previous
filelog entry.
Differential Revision: https://phab.mercurial-scm.org/D8010
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 22 Jan 2020 20:01:38 -0800] rev 44140
packaging: add configparser to inno requirements file
This dependency is missing and pip complains about it in
strict hashing mode. How this was missed, I have no clue.
Differential Revision: https://phab.mercurial-scm.org/D7973
Augie Fackler <raf@durin42.com> [Fri, 24 Jan 2020 11:02:59 -0500] rev 44139
Added signature for changeset e4344e463c0c
Augie Fackler <raf@durin42.com> [Fri, 24 Jan 2020 11:02:58 -0500] rev 44138
Added tag 5.3rc1 for changeset e4344e463c0c
Matt Harbison <matt_harbison@yahoo.com> [Fri, 24 Jan 2020 01:37:18 -0500] rev 44137
packaging: rename hgrc.d to defaultrc for Windows config files next to the exe
The code and the help still says that it will read hgrc.d next to the
executable. But this directory needs to exist to read the resource based config
files. Otherwise even `hg version` errors out:
$ /c/Program\ Files/Mercurial/hg.exe version
Traceback (most recent call last):
File "hg", line 43, in <module>
File "mercurial\dispatch.pyc", line 110, in run
File "mercurial\dispatch.pyc", line 226, in dispatch
File "mercurial\ui.pyc", line 308, in load
File "mercurial\rcutil.pyc", line 99, in rccomponents
File "mercurial\rcutil.pyc", line 69, in default_rc_resources
File "mercurial\utils\resourceutil.pyc", line 84, in contents
WindowsError: [Error 3] The system cannot find the path specified: 'c:\\Program Files\\mercurial\\defaultrc\\*.*'
Differential Revision: https://phab.mercurial-scm.org/D7981
Matt Harbison <matt_harbison@yahoo.com> [Fri, 24 Jan 2020 01:11:19 -0500] rev 44136
resourceutil: ensure `_rootpath` is defined under py2exe
Can't even run `hg version` without this.
Differential Revision: https://phab.mercurial-scm.org/D7980
Kyle Lippincott <spectral@google.com> [Tue, 21 Jan 2020 17:15:34 -0800] rev 44135
crecord: fix a concatenation of bytes and str on py3
Differential Revision: https://phab.mercurial-scm.org/D7970
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 22 Jan 2020 14:11:11 -0500] rev 44134
recover: fix typos
Differential Revision: https://phab.mercurial-scm.org/D7971
Martin von Zweigbergk <martinvonz@google.com> [Tue, 21 Jan 2020 10:27:39 -0800] rev 44133
relnotes: copy "next" to "5.3" and clear "next"
This is the same thing as we've done for the last two releases.
Differential Revision: https://phab.mercurial-scm.org/D7955
Kyle Lippincott <spectral@google.com> [Tue, 21 Jan 2020 12:10:35 -0800] rev 44132
cext: change two more vars to Py_ssize_t in manifest.c
D7913 fixed a compiler warning with a signedness conflict in a ternary operator
by changing the types of some variables to be Py_ssize_t instead of size_t or
int. That commit missed these two cases since they aren't warned about (at least
on my compiler).
Both of these variables are produced by operations on variables that are
themselves Py_ssize_t now/already, so they should keep the same type.
Differential Revision: https://phab.mercurial-scm.org/D7964
Augie Fackler <raf@durin42.com> [Tue, 21 Jan 2020 13:16:30 -0500] rev 44131
Added signature for changeset 84a0102c05c7
Augie Fackler <raf@durin42.com> [Tue, 21 Jan 2020 13:16:29 -0500] rev 44130
Added tag 5.3rc0 for changeset 84a0102c05c7
Augie Fackler <augie@google.com> [Tue, 21 Jan 2020 13:14:51 -0500] rev 44129
merge to stable for 5.3 release freeze
Matt Harbison <matt_harbison@yahoo.com> [Fri, 17 Jan 2020 16:56:49 -0500] rev 44128
phabricator: use .arcconfig for `phabricator.url` if not set locally
This setting is also per repo; see the previous commit for details.
The existing `conduit_uri` setting is the previous name of `phabricator.uri`[1]
and while it could easily be queried before the latter for compatibility, the
config in this repo has '/api' appended. That's already done in `callconduit()`,
which would clearly end up giving the wrong result. It looks like the path of
the URL is now ignored in user configs[2], so add the modern setting without it
to this repo's .arcconfig.
Sadly, we still need to have contributors configure `auth.hg.phabtoken` (and
therefore `auth.hg.prefix` to link it to `phabricator.url`) in order to submit
patches, but at least now it's localized to a single section.
[1] https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/
[2] https://github.com/phacility/arcanist/blob/cc850163f30c4697e925df0d6212469679600a2c/scripts/arcanist.php#L271
Differential Revision: https://phab.mercurial-scm.org/D7935
Matt Harbison <matt_harbison@yahoo.com> [Fri, 17 Jan 2020 14:21:40 -0500] rev 44127
phabricator: use .arcconfig for the callsign if not set locally (issue6243)
This makes things easier for people working with more than one repository
because this file can be committed to each repository. The bug report asks to
read <repo>/.arcrc, but AFAICT, that file lives in ~/ and holds the credentials.
And we already track an .arcconfig file. Any callsign set globally is still
used if that is all that is present, but .arcconfig will override it if
available. The idea behind letting the local hgrc override .arcconfig is that
the developer may need to do testing against another server, and not dirty the
working directory.
Originally I was going to just try to read the callsign in `getrepophid()` if it
wasn't present in the hg config. That works fine, but I think it also makes
sense to read the URL from this file too. That would have worked less well
because `readurltoken()` doesn't have access to the repo object to know where to
find the file. Supplimenting the config mechanism is less magical because it
reports the source and value of the properties used, and it doesn't need to read
the file twice.
Invalid hgrc files generally cause the program to abort. I only flagged it as a
warning here because it's not our config file, not crucial to the whole program
operating, and really shouldn't be corrupt in the typical case where it is
checked into the repo.
Differential Revision: https://phab.mercurial-scm.org/D7934
Matt Harbison <matt_harbison@yahoo.com> [Fri, 17 Jan 2020 13:29:47 -0500] rev 44126
config: add a function to insert non-file based, but overridable settings
This will be used in the next patch.
Until relatively recently (473510bf0575), there was no official way for
extensions to inject per-repo config data, so it probably makes sense that
`ui.setconfig()` items are sticky, and not affected by loading more config
files. But that makes it cumbersome if the extension wants to allow the data it
might add to be overridden by any data in the local hgrc file. The only thing I
could get to work was to load the local hgrc first, and then check if the source
for the config item that should be overridden was *not* the local hgrc file
name. But that's brittle because in addition to the file name, the source
contains the line number, there are the usual '\' vs '/' platform differences,
etc.
Differential Revision: https://phab.mercurial-scm.org/D7933
Matt Harbison <matt_harbison@yahoo.com> [Thu, 16 Jan 2020 19:48:01 -0500] rev 44125
tests: restore phabricator tests and regenerate the recordings
These contain the new API chatter. Most of the changes are because some new
commits were created, but they're pretty obviously equivalent. I have no idea
why the last recording contains real data, whereas it previously looked fake.
Differential Revision: https://phab.mercurial-scm.org/D7920
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 07 Jan 2020 11:24:05 +0100] rev 44124
hgrc: introduce HGRCSKIPREPO to skip reading the repository's hgrc
We had a way to change the behavior regarding reading the global and user
config, but we had nothing regarding the repository hgrc itself.
This option is useful in situation where scripts need to be able to work around
strange configuration set by the user in his repository. (and were HGPLAIN is
not enough).
Differential Revision: https://phab.mercurial-scm.org/D7807
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 18 Jan 2020 10:37:14 -0800] rev 44123
debugcommands: move away from line buffered output on binary stream
Line buffering on binary file objects is apparently undefined behavior
in Python and emits a RuntimeWarning on Python 3.8. See
https://bugs.python.org/issue32236.
This commit changes the I/O logging file descriptor from line
buffered to unbuffered to work around this. I'm no fan of
unbuffered I/O for performance reasons. But I don't think it
is an issue here given the nature of the code.
With this change, test-ssh-proto.t now passes on Python 3.8.
Differential Revision: https://phab.mercurial-scm.org/D7948
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 18 Jan 2020 10:43:52 -0800] rev 44122
py3: conditionalize test-lfs-serve-access.t for Python 3.8
This is another case where Python 3.8's traceback printing
varies subtly from older Python versions. Again, I'm not sure
why. But this is apparently the new behavior.
With this change, test-lfs-serve-access.t now passes on Python
3.8!
Differential Revision: https://phab.mercurial-scm.org/D7947
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 18 Jan 2020 10:27:03 -0800] rev 44121
py3: add extra traceback line present on Python 3.8
I'm not sure why Python 3.8 is outputting this line. It
appears to be a change in behavior of formatting tracebacks on
Python 3.8. So let's add it to expected output.
With this change, test-hook.t now passes on Python 3.8.
Differential Revision: https://phab.mercurial-scm.org/D7946
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 18 Jan 2020 10:12:41 -0800] rev 44120
py3: conditionalize test-flagprocessor.t on Python 3.8
For reasons I don't understand, Python 3.8 is outputting a different
lint in the traceback than prior Pythons.
The lines in question are:
flagutil.addflagprocessor(
REVIDX_NOOP, (noopdonothingread, noopdonothing, validatehash,)
)
Python <3.8 prints the 2nd line but 3.8 the first line. Perhaps Python
changed its traceback logic to always print the first line of a
multiple line expression?
Whatever the case, with this change, the test now passes on
Python 3.8.
Differential Revision: https://phab.mercurial-scm.org/D7945
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 18 Jan 2020 10:21:45 -0800] rev 44119
tests: conditionalize test-hightlight.t on pygments version
Output changed slightly in pygments 2.5.
I thought it was easier to copy the line and add a version
check than to add a regular expression because the line has some
special characters. I also like tests explicitly calling out where
they vary.
Differential Revision: https://phab.mercurial-scm.org/D7943
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 20 Jan 2020 23:51:25 -0800] rev 44118
hgdemandimport: apply lazy module loading to sys.meta_path finders
Python's `sys.meta_path` finders are the primary objects whose job it
is to find a module at import time. When `import` is called, Python
iterates objects in this list and calls `o.find_spec(...)` to find
a `ModuleSpec` (or None if the module couldn't be found by that
finder). If no meta path finder can find a module, import fails.
One of the default meta path finders is `PathFinder`. Its job is to
import modules from the filesystem and is probably the most important
importer. This finder looks at `sys.path` and `sys.path_hooks` to do
its job.
The `ModuleSpec` returned by `MetaPathImporter.find_spec()` has a
`loader` attribute, which defines the concrete module loader to use.
`sys.path_hooks` is a hook point for teaching `PathFinder` to
instantiate custom loader types.
Previously, we injected a custom `sys.path_hook` that told `PathFinder`
to wrap the default loaders with a loader that creates a module object
that is lazy.
This approach worked. But its main limitation was that it only applied
to the `PathFinder` meta path importer. There are other meta path
importers that are registered. And in the case of PyOxidizer loading
modules from memory, `PathFinder` doesn't come into play since
PyOxidizer's own meta path importer was handling all imports.
This commit changes our approach to lazy module loading by proxying
all meta path importers. Specifically, we overload the `find_spec()`
method to swap in a wrapped loader on the `ModuleSpec` before it
is returned. The end result of this is all meta path importers should
be lazy.
As much as I would have loved to utilize .__class__ manipulation to
achieve this, some meta path importers are implemented in C/Rust
in such a way that they cannot be monkeypatched. This is why we
use __getattribute__ to define a proxy.
Also, this change could theoretically open us up to regressions in
meta path importers whose loader is creating module objects which
can't be monkeypatched. But I'm not aware of any of these in the
wild. So I think we'll be safe.
According to hyperfine, this change yields a decent startup time win of
5-6ms:
```
Benchmark #1: ~/.pyenv/versions/3.6.10/bin/python ./hg version
Time (mean ± σ): 86.8 ms ± 0.5 ms [User: 78.0 ms, System: 8.7 ms]
Range (min … max): 86.0 ms … 89.1 ms 50 runs
Time (mean ± σ): 81.1 ms ± 2.7 ms [User: 74.5 ms, System: 6.5 ms]
Range (min … max): 77.8 ms … 90.5 ms 50 runs
Benchmark #2: ~/.pyenv/versions/3.7.6/bin/python ./hg version
Time (mean ± σ): 78.9 ms ± 0.6 ms [User: 70.2 ms, System: 8.7 ms]
Range (min … max): 78.1 ms … 81.2 ms 50 runs
Time (mean ± σ): 73.4 ms ± 0.6 ms [User: 65.3 ms, System: 8.0 ms]
Range (min … max): 72.4 ms … 75.7 ms 50 runs
Benchmark #3: ~/.pyenv/versions/3.8.1/bin/python ./hg version
Time (mean ± σ): 78.1 ms ± 0.6 ms [User: 70.2 ms, System: 7.9 ms]
Range (min … max): 77.4 ms … 80.9 ms 50 runs
Time (mean ± σ): 72.1 ms ± 0.4 ms [User: 64.4 ms, System: 7.6 ms]
Range (min … max): 71.4 ms … 74.1 ms 50 runs
```
Differential Revision: https://phab.mercurial-scm.org/D7954
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 20 Jan 2020 23:42:19 -0800] rev 44117
hgdemandimport: disable on Python 3.5
The demand importer functionality isn't working at all on Python 3.5.
I'm not sure what's wrong.
Since it isn't working, let's disable it completely.
```
$ HGRCPATH= hyperfine -w 1 -r 50 -- "~/.pyenv/versions/3.5.9/bin/python ./hg version" \
"HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version"
Benchmark #1: ~/.pyenv/versions/3.5.9/bin/python ./hg version
Time (mean ± σ): 163.7 ms ± 2.2 ms [User: 148.5 ms, System: 15.7 ms]
Range (min … max): 161.0 ms … 170.2 ms 50 runs
Benchmark #2: HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version
Time (mean ± σ): 164.3 ms ± 1.4 ms [User: 148.2 ms, System: 16.6 ms]
Range (min … max): 161.4 ms … 169.8 ms 50 runs
```
Differential Revision: https://phab.mercurial-scm.org/D7953
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 18 Jan 2020 11:13:01 -0800] rev 44116
py3: suppress unraisable exceptions in test-worker.t
Python 3.8 calls sys.unraisablehook when an unraisable
exception is encountered. The default behavior is to print a
warning.
test-worker.t was triggering this hook due to a race between
a newly forked process exiting and that process's
_os.register_at_fork handlers running. I was seeing the
stdlib's random module in the stack re-seeding itself. Although
there could be other after-fork handlers in the mix.
This commit defines sys.unraisablehook to effectively no-op.
This suppresses the warning and makes test output on Python 3.8
consistent with prior versions. test-worker.t now passes on
Python 3.8.
Differential Revision: https://phab.mercurial-scm.org/D7949
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Mon, 20 Jan 2020 18:28:46 -0500] rev 44115
rust: add a README
In particular to explain how to build any of the rust. It's neither
obvious, nor easy to find out, nor easy to determine if you did it
right without some documentation.
Differential Revision: https://phab.mercurial-scm.org/D7952
Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> [Mon, 20 Jan 2020 17:44:03 -0500] rev 44114
rust: move hgcli's README out of the way
My understanding is that it's not meant to be used in the current
form.
Differential Revision: https://phab.mercurial-scm.org/D7951
Matt Harbison <matt_harbison@yahoo.com> [Sat, 18 Jan 2020 01:54:17 -0500] rev 44113
verify: avoid spurious integrity warnings in verbose mode (issue6172)
The issue seems to revolve around renames in filtered commits, and only occurred
in verbose mode. The problem occurs in the `# check renames` stage, around line
577. Without using the unfiltered repo, this test would have printed:
$ hg verify -v
repository uses revlog format 1
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
foo@25: checking rename of 71ec0570c325: filtered revision '25'
foobar@26: checking rename of 1b549296015b: filtered revision '26'
checked 28 changesets with 16 changes to 11 files
2 integrity errors encountered!
(first damaged changeset appears to be 25)
[1]
Differential Revision: https://phab.mercurial-scm.org/D7950
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 17 Jan 2020 22:31:47 -0800] rev 44112
py3: glob over exception in test-check-py3-compat.t
Python 3.6+ raise ModuleNotFoundError and older versions raise
ImportError. Glob over the exception differences.
For whatever reason, we were already doing this for one failure.
But not all occurrences of ModuleNotFoundError were changed.
Who knows.
This test should now pass on all Python versions (although I didn't
check Windows).
Differential Revision: https://phab.mercurial-scm.org/D7939
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 17 Jan 2020 22:24:27 -0800] rev 44111
py3: string normalization and I/O tweaks in test-lfs.t
The print was inserting b'' on Python 3. In addition, since we
weren't writing to the ui instance (which isn't readily available
in this function), output order could get mixed up.
We add some pycompat casts and a stdout flush to make the test
happy on all Python versions.
Differential Revision: https://phab.mercurial-scm.org/D7938
Matt Harbison <matt_harbison@yahoo.com> [Fri, 17 Jan 2020 21:27:53 -0500] rev 44110
help: minor copy editing to the `config.format` section
Differential Revision: https://phab.mercurial-scm.org/D7936
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 21 Nov 2019 17:27:44 +0100] rev 44109
changectx: mark parent of changesets as non filtered
If a node is not filtered, its parents cannot be filtered.
Differential Revision: https://phab.mercurial-scm.org/D7502
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 21 Nov 2019 23:46:51 +0100] rev 44108
changectx: use unfiltered changelog to walk ancestors in annotate
Since we are only walking ancestors, it is safe to use an unfiltered repository.
(Because if the original rev is not filtered, none of its ancestors will be).
Differential Revision: https://phab.mercurial-scm.org/D7501
Pierre-Yves David <pierre-yves.david@octobus.net> [Thu, 21 Nov 2019 23:25:08 +0100] rev 44107
localrepo: also fast past the parents of working copies parents
There are descent odds that they will be needed too. So we also cache and
fastpath them.
Differential Revision: https://phab.mercurial-scm.org/D7498
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 17 Nov 2019 14:54:41 +0100] rev 44106
localrepo: recognize trivial request for '.'
Same logic as for `null`, this is a command request and skipping the revset
logic can avoid triggering the changelog filtering logic.
Differential Revision: https://phab.mercurial-scm.org/D7495
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 17 Nov 2019 14:47:29 +0100] rev 44105
localrepo: fastpath access to "."
"." is just an alias for `p1(wdir())`, let us handle it that way.
Differential Revision: https://phab.mercurial-scm.org/D7494
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 17 Nov 2019 14:39:28 +0100] rev 44104
localrepo: also fastpath access to working copy parents when possible
If the filter level guarantee that the working copy parents will be visible, we
allow fast path access to them. With this change multiple commands can now run
without triggering filtering.
After using the quick access mechanism introduced, the whole series results in
pretty good performance gain:
```
All benchmarks:
before after ratio
[8e095512] [36b2f659]
- 711±0.8ms 60.7±0.2ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 712±0.8ms 61.6±0.2ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 690±1ms 93.5±0.3ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 688±1ms 93.8±0.3ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 714±1ms 60.7±0.8ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 713±1ms 60.9±0.3ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 689±1ms 93.7±0.2ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 687±2ms 92.8±0.2ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 799±2ms 98.1±0.6ms 0.12 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 800±0.8ms 100.0±0.4ms 0.12 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 711±0.9ms 111±0.2ms 0.16 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 711±1ms 112±0.3ms 0.16 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 760±1ms 59.8±0.1ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 763±2ms 62.2±0.3ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 689±1ms 93.1±0.3ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 688±1ms 94.3±0.3ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 763±1ms 60.1±0.2ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 763±1ms 62.1±0.4ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 689±0.8ms 93.2±0.2ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 687±0.9ms 94.1±0.3ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
```
Differential Revision: https://phab.mercurial-scm.org/D7492
Martin von Zweigbergk <martinvonz@google.com> [Thu, 16 Jan 2020 08:41:38 -0800] rev 44103
examples: refer to nightly rustfmt in Windows-compatible way
Thanks to Jun Wu for the tip. I found that the new form also gave
better error messages when the nightly rustfmt wasn't installed (it
told me which command to run instead of just saying "error: not a
file: <some path>").
Differential Revision: https://phab.mercurial-scm.org/D7911
Joerg Sonnenberger <joerg@bec.de> [Thu, 26 Dec 2019 19:05:55 +0100] rev 44102
convert: refactor authormap into separate function for outside use
Differential Revision: https://phab.mercurial-scm.org/D7732
Inada Naoki <songofacandy@gmail.com> [Tue, 14 Jan 2020 17:57:15 +0900] rev 44101
remotefilelog: fix opening validatecachelog in text mode
Kyle Lippincott <spectral@google.com> [Thu, 16 Jan 2020 12:27:15 -0800] rev 44100
cext: fix compiler warning about sign changing
line.len is a Py_ssize_t, and we're casing to size_t (unsigned). On my compiler,
this causes a warning to be emitted:
```
mercurial/cext/manifest.c: In function 'pathlen':
mercurial/cext/manifest.c:48:44: warning: operand of ?: changes signedness from 'Py_ssize_t' {aka 'long int'} to 'long unsigned int' due to unsignedness of other operand [-Wsign-compare]
return (end) ? (size_t)(end - l->start) : l->len;
^~~~~~
```
Differential Revision: https://phab.mercurial-scm.org/D7913
Matt Harbison <matt_harbison@yahoo.com> [Wed, 15 Jan 2020 23:34:04 -0500] rev 44099
sha1dc: avoid including the nonexistent stdint.h with Visual Studio 2008
Differential Revision: https://phab.mercurial-scm.org/D7903
Kyle Lippincott <spectral@google.com> [Thu, 16 Jan 2020 12:17:03 -0800] rev 44098
py3: fix curses chunkselector fallback (when diffs are too large) on py3
Previously we showed the message using Exception.message, which is removed in
py3. Since crecordmod.fallbackerror inherits from error.Abort, we can just use
`b'%s' % exception` to print the message. This does not print the hint, but
that's fine - we don't set one. We inherit from error.Abort so that if a
codepath doesn't handle fallback specially, it exits to the terminal with a sane
message instead of an unknown exception error.
Differential Revision: https://phab.mercurial-scm.org/D7912
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 15 Jan 2020 15:47:03 +0100] rev 44097
transaction: allow finalizer to add finalizer
It will make some code (persistent nodemap related) simpler to write, because
higher level code can blindly queue finalization without thinking too hard about
the context.
Differential Revision: https://phab.mercurial-scm.org/D7833
Matt Harbison <matt_harbison@yahoo.com> [Sat, 28 Dec 2019 12:25:16 -0500] rev 44096
tests: stabilize test-subrepo-svn.t on Windows
This partially reverts 991e4404e910, because the URL form of `C:\...` gets
escaped to `C%3A/...`, which breaks the substitution of $TESTTMP.
The forget command on 'notafile*' errored out with:
notafile*: The filename, directory name, or volume label syntax is incorrect
which I think is because '*' isn't a legal character in a file name (though I
can't trigger this directly from MSYS or cmd.exe for some reason).
Differential Revision: https://phab.mercurial-scm.org/D7816
Martin von Zweigbergk <martinvonz@google.com> [Mon, 13 Jan 2020 11:18:29 -0800] rev 44095
rebase: fix bug where `--collapse` would apply diff on missing file
Even though the file was missing, the rebase would succeed.
Differential Revision: https://phab.mercurial-scm.org/D7897
Martin von Zweigbergk <martinvonz@google.com> [Mon, 13 Jan 2020 11:11:20 -0800] rev 44094
rebase: extract a variable for a repeated `repo[p1]`
I'll add another use site in the next patch.
Differential Revision: https://phab.mercurial-scm.org/D7896
Martin von Zweigbergk <martinvonz@google.com> [Sun, 29 Dec 2019 17:53:48 -0800] rev 44093
graftcopies: document why the function is useful at all
Despite having spent a significant amount on time on the copy-tracing
code, I thought `graftcopies()` (formerly known as
`duplicatecopies()`) was needed to duplicate copies after calling
`merge.update()` to do a merge (as `merge.graft()` does), but it's
actually usually not needed; `merge.update()` takes care of most
copies. This patch documents what the function is for.
Differential Revision: https://phab.mercurial-scm.org/D7861
Martin von Zweigbergk <martinvonz@google.com> [Fri, 27 Dec 2019 13:47:59 -0800] rev 44092
graftcopies: remove `skip` and `repo` arguments
The `skip` argument was added in 2ba6c9b4e0eb (rebase: fix bug that
caused transitive copy records to disappear (issue4192), 2014-06-07)
in order to fix https://bz.mercurial-scm.org/show_bug.cgi?id=4192. I
ran tests at that commit without the `skiprev` argument and the only
difference I noticed was that `test-rebase-collapse.t` failed
differently, in the call that is now on line 501. Without the
`skiprev` argument, that call would end up creating another commit
because it tried to record an invalid copy. With the previous patch in
this series, such invalid copies are no longer recorded, so it seems
we don't need the `skip` argument anymore.
I also removed the `repo` argument since that also becomes unused with
the removal of the `skip` argument.
Differential Revision: https://phab.mercurial-scm.org/D7860
Martin von Zweigbergk <martinvonz@google.com> [Fri, 27 Dec 2019 15:14:19 -0800] rev 44091
graftcopies: use _filter() for filtering out invalid copies
`graftcopies()` (formerly called `duplicatecopies()`) checked that the
copy destination existed in the working copy, but it didn't check that
copy source existed in the parent of the working copy. In
`test-graft.t` we can see that as warnings about not finding ancestors
of the copied files, and also empty commits getting created.
This patch uses the existing `_filter()` function for filtering out
invalid copies. In addition to the aforementioned types, that also
includes copies where source and destination is the same.
Differential Revision: https://phab.mercurial-scm.org/D7859
Martin von Zweigbergk <martinvonz@google.com> [Mon, 06 Jan 2020 15:24:36 -0800] rev 44090
copies: replace duplicatecopies() by function that takes contexts
The callers mostly have context objects, so let's avoid looking up the
same context objects inside `duplicatecopies()`.
I also renamed the function to `graftcopies()` since I think that
better matches its purpose. I did it in the same commit so it's easier
for extensions to switch between the functions.
Differential Revision: https://phab.mercurial-scm.org/D7858
Martin von Zweigbergk <martinvonz@google.com> [Fri, 27 Dec 2019 13:03:40 -0800] rev 44089
graft: extract repo[None] to a variable
I plan to allow the caller pass in an overlayworkingctx, so this
prepares for that.
Differential Revision: https://phab.mercurial-scm.org/D7857
Aay Jay Chan <aayjaychan@itopia.com.hk> [Thu, 16 Jan 2020 00:30:08 +0800] rev 44088
rust-core: fix typo in comment
Differential Revision: https://phab.mercurial-scm.org/D7895
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 14 Jan 2020 18:59:49 -0800] rev 44087
sha1dc: use buffer protocol when parsing arguments
Without this, functions won't accept bytearray, memoryview,
or other types that can be exposed as bytes to the C API.
The most resilient way to obtain a bytes-like object from
the C API is using the Py_buffer interface.
This commit converts use of s#/y# to s*/y* and uses
Py_buffer for accessing the underlying bytes array.
I checked how hashlib is implemented in CPython and the
the implementation agrees with its use of the Py_buffer
interface as well as using BufferError in cases of bad
buffer types. Sadly, there's no good way to test for
ndim > 1 without writing our own C-backed Python type.
Differential Revision: https://phab.mercurial-scm.org/D7879
Matt Harbison <matt_harbison@yahoo.com> [Tue, 14 Jan 2020 20:05:37 -0500] rev 44086
lfs: avoid quadratic performance in processing server responses
This is also adapted from the Facebook repo[1]. Unlike there, we were already
reading the download stream in chunks and immediately writing it to disk, so we
basically avoided the problem on download. There shouldn't be a lot of data to
read on upload, but it's better to get rid of this pattern.
[1] https://github.com/facebookexperimental/eden/commit/82df66ffe97e21f3ee73dfec093c87500fc1f6a7
Differential Revision: https://phab.mercurial-scm.org/D7882
Matt Harbison <matt_harbison@yahoo.com> [Tue, 14 Jan 2020 19:42:24 -0500] rev 44085
lfs: check content length after downloading content
Adapted from the Facebook repo[1]. The intent is to distinguish between the
connection dying and getting served a corrupt blob.
The original message:
HTTP makes no provision to tell your client that you failed halfway through
producing your response and won't have the answer they're looking for. So, if a
LFS server fails while producing a response, then we'll report an OID mismatch.
We can do a little better and disambiguate between "the server sent us the
wrong blob" (very scary) and "the server crashed" (merely annoying) by looking
at the content length of the response we got back. If it's not what was
advertised, we can reasonably safely assume the server crashed.
[1] https://github.com/facebookexperimental/eden/commit/2a4a6fab4e882ed89b948bfc1e7d56d7c3c99dd2
Differential Revision: https://phab.mercurial-scm.org/D7881
Matt Harbison <matt_harbison@yahoo.com> [Tue, 14 Jan 2020 18:02:20 -0500] rev 44084
lfs: rename a variable to clarify its use
This is the response object, not a request.
Differential Revision: https://phab.mercurial-scm.org/D7880
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 14 Jan 2020 17:53:43 -0800] rev 44083
sha1dc: use proper string functions on Python 2/3
PyString_FromStringAndSize doesn't exist on Python 3: we need
to use PyUnicode_FromStringAndSize.
The extension now compiles without warnings on Python 2 and 3.
Differential Revision: https://phab.mercurial-scm.org/D7878
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 14 Jan 2020 17:39:12 -0800] rev 44082
sha1dc: declare all variables at begininng of block
This is required to appease ancient C language standards, which
msvc 2008 still requires for Python 2.7 on Windows.
Differential Revision: https://phab.mercurial-scm.org/D7877
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 14 Jan 2020 17:37:04 -0800] rev 44081
sha1dc: manually define integer types on msvc 2008
Python 2.7 on Windows builds with MSVC 2008, which
doesn't include stdint.h. So we need to check for the
compiler version and manually define missing types when it
is ancient.
Differential Revision: https://phab.mercurial-scm.org/D7876
Martin von Zweigbergk <martinvonz@google.com> [Tue, 14 Jan 2020 14:18:11 -0800] rev 44080
packaging: leverage os.path.relpath() in setup.py
`os.path.relpath()` has existed since Python 2.6, so we can safely use
it. This fixes a bug in the current code when the common prefix is "/"
(in which case `uplevel` would be one less than it should).
Differential Revision: https://phab.mercurial-scm.org/D7875
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 18:00:05 +0100] rev 44079
rust-utils: add util to find a slice in another slice
Differential Revision: https://phab.mercurial-scm.org/D7863
Raphaël Gomès <rgomes@octobus.net> [Tue, 14 Jan 2020 16:00:57 +0100] rev 44078
dirstate: move rust fast-path calling code to its own method
This logic is about to get bigger, this will make it easier to read and not
pollute the main Python logic.
Differential Revision: https://phab.mercurial-scm.org/D7862
Matt Harbison <matt_harbison@yahoo.com> [Tue, 14 Jan 2020 00:52:53 -0500] rev 44077
lfs: add "bytes" as the unit to the upload/download progress bar
Facebook also passes `util.bytecount()` as a pretty formatter here, but our
progress bar doesn't support that.
Differential Revision: https://phab.mercurial-scm.org/D7872
Matt Harbison <matt_harbison@yahoo.com> [Tue, 14 Jan 2020 16:37:45 -0500] rev 44076
phabricator: post revisions in ascending topological order (issue6241)
The parent in phabricator ends up being the last revision posted, so sorting the
user input into ascending order should be enough to preserve the proper
relationships.
Differential Revision: https://phab.mercurial-scm.org/D7874
Matt Harbison <matt_harbison@yahoo.com> [Tue, 14 Jan 2020 16:29:03 -0500] rev 44075
doc: fix references to `revset.abstractsmartset`
Differential Revision: https://phab.mercurial-scm.org/D7873
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 13 Jan 2020 20:09:32 -0800] rev 44074
fsmonitor: properly handle str ex.msg
ex.msg is always a str, since pywatchman uses str for exception messages.
This commit removes a b'' from a string compare to avoid types
mismatch and adds a coercion to bytes before stuffing the exception
message on our local exception type, which uses bytes for the message
elsewhere in this file.
Differential Revision: https://phab.mercurial-scm.org/D7855
Matt Harbison <matt_harbison@yahoo.com> [Mon, 23 Dec 2019 01:12:20 -0500] rev 44073
verify: allow the storage to signal when renames can be tested on `skipread`
This applies the new marker in the lfs handler to show it in action, and adds
the test mentioned at the beginning of the series to show that fulltext isn't
necessary in the LFS case.
The existing `skipread` isn't enough, because it is also set if an error occurs
reading the revlog data, or the data is censored. It could probably be cleared,
but then it technically violates the interface contract. That wouldn't matter
for the existing verify algorithm, but it isn't clear how that will change as
alternate storage support is added.
The flag is probably pretty revlog specific, given the comments in verify.py.
But there's already filelog specific stuff in there and I'm not sure what future
storage will bring, so I don't want to over-engineer this. Likewise, I'm not
sure that we want the verify method for each storage type to completely drive
the bus when it comes to detecting renames, so I don't want to go down the
rabbithole of having verifyintegrity() return metadata hints at this point.
Differential Revision: https://phab.mercurial-scm.org/D7713
Matt Harbison <matt_harbison@yahoo.com> [Sun, 22 Dec 2019 23:50:19 -0500] rev 44072
lfs: don't skip locally available blobs when verifying
The `skipflags` config was introduced in a2ab9ebcd85b, which specifically calls
out downloading and storing all blobs as potentially too expensive. But I don't
see any reason to skip blobs that are already available locally. Hashing the
blob is the only way to indirectly verify the rawdata content stored in the
revlog.
(The note in that commit about skipping renamed is still correct, but the reason
given about needing fulltext isn't.)
Differential Revision: https://phab.mercurial-scm.org/D7712
Matt Harbison <matt_harbison@yahoo.com> [Fri, 20 Dec 2019 01:11:35 -0500] rev 44071
lfs: add a switch to `hg verify` to ignore the content of blobs
Trying to validate the fulltext of an external revision causes missing blobs to
be downloaded and cached. Since the downloads aren't batch prefetched[1] and
aren't compressed, this can be expensive both in terms of time and space.
I made this a tri-state instead of a simple bool because there's an existing
(undocumented) config to handle this, and it would be weird if `hg verify` were
to suddenly start ignoring that config but an `hg recover` initiated verify
honors it. Since this uses the same config setting, it too will skip
rename verification (which requires fulltext, but not for LFS).
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-April/116118.html
Differential Revision: https://phab.mercurial-scm.org/D7708
Augie Fackler <augie@google.com> [Wed, 08 Jan 2020 14:37:54 -0500] rev 44070
revlog: run rustfmt nightly
I'm a little nervous about folding this back (might be nightly rustfmt
mismatches?) so I want someone to review this.
Differential Revision: https://phab.mercurial-scm.org/D7813
Augie Fackler <augie@google.com> [Wed, 08 Jan 2020 14:37:01 -0500] rev 44069
examples: specify rustfmt nightly using a $() construct
This is ugly, but it's how we have to configure rustfmt for now as we
require nightly rustfmt.
Differential Revision: https://phab.mercurial-scm.org/D7812
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 07 Dec 2019 13:13:48 -0800] rev 44068
hg-core: rustfmt path.rs
The file as vendored does not conform to our source formatting
conventions. Let's reformat it so it does.
# skip-blame automated code reformatting
Differential Revision: https://phab.mercurial-scm.org/D7580
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 07 Dec 2019 10:26:28 -0800] rev 44067
hg-core: vendor Facebook's path utils module
The added file was imported from
https://github.com/facebookexperimental/eden/blob/d1d8fb939a39aa331ae7f162c39cbcaa511d474b/eden/scm/lib/util/src/path.rs
without modifications. The file is not yet integrated into our project. This
will be done in subsequent commits.
Differential Revision: https://phab.mercurial-scm.org/D7573
Georges Racinet <georges.racinet@octobus.net> [Tue, 14 Jan 2020 12:04:12 +0100] rev 44066
revlog-native: introduced ABI version in capsule
Concerns that an inconsistency could arise between the actual contents
of the capsule in revlog.c and the Rust consumer have been raised after
the switch to the array of data and function pointers in f384d68d8ea8.
It has been suggested that the `version` from parsers.c could be use for
this. In this change, we introduce instead a separate ABI version number,
which should have the following advantages:
- no need to change the consuming Rust code for changes that have nothing
to do with the contents of the capsule
- the version number in parsers.c is not explicitely flagged as ABI. It's
not obvious to me whether an ABI change that would be invisible to Python
would warrant an increment
The drawback is that developers now have to consider two version numbers.
We expect the added cost of the check to be negligible because it occurs
at instantiation of `CIndex` only, which in turn is tied to instantiation
of Python objects such as `LazyAncestors` and `MixedIndex`. Frequent calls
to `Cindex::new` should also probably hit the CPU branch predictor.
Differential Revision: https://phab.mercurial-scm.org/D7856
Rodrigo Damazio Bovendorp <rdamazio@google.com> [Mon, 13 Jan 2020 19:11:44 -0800] rev 44065
phases: make phasecache._phasesets immutable
Previously, some code paths would mutate the cache itself, which
could give weird results if multiple revsets got evaluated through
that path.
Differential Revision: https://phab.mercurial-scm.org/D7854