FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:47 +0900] rev 33280
vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
Now, files (to be truncated) are opened with checkambig=True, only if
localrepository caches it.
Therefore, straightforward "copy if EPERM" is always reasonable to
avoid file stat ambiguity at closing.
This patch makes checkambigatclosing close wrapper copy the target
file, and advance mtime on it after renaming, if EPERM. This can avoid
file stat ambiguity, even if the target file is owned by another (see
issue5418 and
issue5584 for detail).
This patch factors main logic out instead of changing
checkambigatclosing._checkambig() directly, in order to reuse it.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:47 +0900] rev 33279
transaction: apply checkambig=True only on limited files for similarity
Now, transaction can determine whether avoidance of file stat
ambiguity is needed for each files, by blacklist "checkambigfiles".
For similarity to truncation in _playback(), this patch apply
checkambig=True only on limited files in code paths below.
- restoring files by util.copyfile(), in _playback()
(checkambigfiles itself is examined at first, because it as a
keyword argument might be None)
- writing files at finalization of transaction, in _generatefiles()
This patch reduces cost of checking stat at writing out and restoring
files, which aren't filecache-ed.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:46 +0900] rev 33278
transaction: avoid file stat ambiguity only for files in blacklist
Advancing mtime by os.utime() fails for EPERM, if the target file is
owned by another.
bff5ccbe5ead and related changes made some code
paths give advancing mtime up in such case, to fix
issue5418.
This causes file stat ambiguity (again), if it is owned by another.
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
To avoid file stat ambiguity in such case, especially for
.hg/dirstate,
ed66ec39933f made vfs.rename() copy the target file, and
advance mtime of renamed one again, if EPERM (see
issue5584 for detail).
But straightforward "copy if EPERM" isn't reasonable for truncation of
append-only files at rollbacking, because rollbacking might cost much
for truncation of many filelogs, even though filelogs aren't
filecache-ed.
Therefore, this patch introduces blacklist "checkambigfiles", and
avoids file stat ambiguity only for files specified in this blacklist.
This patch consists of two parts below, which should be applied at
once in order to avoid regression.
- specify 'checkambig=True' at vfs.open(mode='a') in _playback()
according to checkambigfiles
- invoke _playback() with checkambigfiles
- add transaction.__init__() checkambigfiles argument, for _abort()
- make localrepo instantiate transaction with _cachedfiles
- add rollback() checkambigfiles argument, for "hg rollback/recover"
- make localrepo invoke rollback() with _cachedfiles
After this patch, straightforward "copy if EPERM" will be reasonable
at closing the file opened with checkambig=True, because this policy
is applied only on files, which are listed in blacklist
"checkambigfiles".
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:46 +0900] rev 33277
localrepo: store path and vfs location of cached properties
This information is used to make transaction handle these files
specially, in order to avoid file stat ambiguity of them.
Gathering information about cached files via annotation classes can
avoid overlooking properties newly introduced in the future.
Boris Feld <boris.feld@octobus.net> [Mon, 03 Jul 2017 11:22:00 +0200] rev 33276
template: add successors template
Add a 'successorssets' template that returns the list of all closest known
sucessorssets for a changectx. The elements of the list are changesets.
The "closest successors" are the first locally known revisions encountered
while, walking successors markers. It uses successorsets previously modified
to support the closest argument.
This logic respect repository filtering. So hidden revision will be skipped by
this logic unless --hidden is specified. Since we only display the visible
predecessors, this template will not display anything in most case. It makes a
good candidate for inclusion in the default log output.
I updated the test-obsmarker-template.t test file introduced with the
predecessors template to test successorssets template.
Boris Feld <boris.feld@octobus.net> [Mon, 03 Jul 2017 14:22:28 +0200] rev 33275
template: add tests for more complex cases
We add new tests for improving the coverage of existing obs-markers related
template (predecessors) and the new one we are introducing (successorssets).
Boris Feld <boris.feld@octobus.net> [Fri, 30 Jun 2017 15:27:19 +0200] rev 33274
obsolete: closest divergent support
Add a closest argument to successorssets changing the definition of latest
successors.
With "closest=false" (current behavior), latest successors are "leafs" on the
obsmarker graph. They don't have any successor and are known locally.
With "closest=true", latest successors are the closest locally-known
changesets that are visible in the repository or repoview. Closest successors
can be then obsolete, orphan.
This will be used in a later patch to show the closest successor of
changesets with the successorssets template.
Boris Feld <boris.feld@octobus.net> [Fri, 30 Jun 2017 15:02:19 +0200] rev 33273
obsolete: pass cache argument of successors set explicitly
We plan to add a new argument to successorsets. But first we need to update
all callers to pass cache argument explicitly to avoid arguments confusion.
Boris Feld <boris.feld@octobus.net> [Fri, 30 Jun 2017 13:47:24 +0200] rev 33272
obsolete: small doc update for 'successorssets'
Clarify successorssets documentation before we start updating the main function.
This patch serie will introduce the successorssets template, the opposite of
predecessor templates.
Successors will use successorssets function and requires some improvement so
before doing that, we clean up successorssets a bit.
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 18:52:28 -0700] rev 33271
phabricator: do not read a same revision twice
It's possible to set up non-linear dependencies in Phabricator like:
o D4
|\
| o D3
| |
o | D2
|/
o D1
The old `phabread` code will print D1 twice. This patch adds de-duplication
to prevent that.
Test Plan:
Construct the above dependencies in a Phabricator test instance and make
sure the old code prints D1 twice while the new code won't.
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:41:28 -0700] rev 33270
patch: make parsepatch optionally trim context lines
Previously there is a suspicious `if False and delta > 0` which dates back
to the beginning of hgext/record.py (
b2607267236d).
The "trimming context lines" feature could be useful (and is used by the
next patch). So let's enable it. This patch adds a new `maxcontext`
parameter to `recordhunk` and `parsepatch`, changing the `if False`
condition to respect it.
The old `trimcontext` implementation is also wrong - it does not update
`toline` correctly and it does not do the right thing for `before` context.
A doctest was added to guard us from making a similar mistake again.
Since `maxcontext` is set to `None` (unlimited), there is no behavior
change.
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:36:48 -0700] rev 33269
phabricator: try to fetch differential revisions in batch
Previously, we read Differential Revisions one by one by calling
`differential.query`.
Fetching them one by one is suboptimal. Unfortunately, there is no Conduit
API that allows us to get a stack of diffids using a single API call.
This patch tries to be smarter using a simple heuristic: when fetching D59
as a stack, previous IDs like D51, D52, D53, ..., D58 are likely belonging
to a same stack so just fetch them as well. Since `differential.query` only
returns cheap metadata without expensive diff content, it shouldn't be a big
problem for the server.
Using a test Phabricator instance, this patch reduces `phabread` reading a
10 patch stack from about 13 to 30 seconds to 8 seconds.
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:36:48 -0700] rev 33268
phabricator: avoid calling differential.getcommitmessage
Previously, we call differential.getcommitmessage API to get commit
messages. Now we read that from "Differential Revision" object fetched
via "differential.query" API.
This removes one API call per patch.
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:36:48 -0700] rev 33267
phabricator: rework phabread to reduce memory usage and round-trips
This patch reworked phabread a bit so it fetches the lightweight
"Differential Revision" metadata for a stack first. Then read other data.
This allows the code to:
a) send 1 request to get all `hg:meta` data instead of N requests
b) patches are read in desired order, no need to buffer the output
"b)" reduces the memory usage from O(N^2) to O(N) since we no longer keep
old patch contents in memory.
The above `N` means the number of patches in the stack.
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:36:48 -0700] rev 33266
phabricator: abort if phabsend gets empty revs
Previously we didn't abort. Now we abort if revs is empty. This is
consistent with "hg export" behavior. Maybe "return 1" is also a reasonable
behavior, but that's inconsistent with the existing "hg export".
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:36:48 -0700] rev 33265
phabricator: do not upload new diff if nothing changes
Previously, `phabsend` uploads new diffs as long as the commit hash changes.
That's suboptimal because sometimes the diff is exactly the same as before,
the commit hash change is caused by a parent hash change, or commit message
change which do not affect diff content.
This patch adds a check examining actual diff contents to skip uploading new
diffs in that case.
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:36:48 -0700] rev 33264
phabricator: add node and p1 to hg:meta property
The "hg:meta" property is for extra metadata to reconstruct the patch.
Previously it does not have node or parent information since I think by
reading the patch again, the commit message will be mangled (like, added the
"Differential Revision" line) and we cannot preserve the commit hash.
However, the "parent" information could be useful. It could be helpful to
locate the "base revision" so in case of a conflict applying the patch
directly, we might be able to use 3-way merge to resolve it correctly.
Note: "local:commits" is an existing "property" used by Phabricator that has
the node and parent information. However, it lacks of timezone information
and requires "author" and "authorEmail" to be separated. So we are using a
different "property" - "hg:meta" to be distinguished from "local:commits".
Jun Wu <quark@fb.com> [Tue, 04 Jul 2017 16:16:37 -0700] rev 33263
phabricator: check associated Differential Revision from commit message
Previously, only tags can "associate" a changeset to a Differential
Revision. But the usual pattern (arc patch or hg phabread) is to put the
Differential Revision URL in commit message.
This patch makes the code read commit message to find associated
Differential Revision if associated tags are not found.
This makes some workflows possible. For example, if the author loses their
repo, or switch to another computer, they can continue download their own
patches from Phabricator and update them without needing to manually create
tags.
Augie Fackler <augie@google.com> [Wed, 05 Jul 2017 11:10:11 -0500] rev 33262
tests: replace yet more calls to `python` with $PYTHON
These are some simple cases. More to come in a future change.
Reviewers: krbullock
Reviewed By: krbullock
Differential Revision: https://phab.mercurial-scm.org/D4
Augie Fackler <augie@google.com> [Wed, 05 Jul 2017 11:09:55 -0500] rev 33261
tests: capitalize Python when it's not used as a command name
This avoids some false positives in an upcoming check-code rule.
Reviewers: krbullock
Reviewed By: krbullock
Differential Revision: https://phab.mercurial-scm.org/D3
Augie Fackler <augie@google.com> [Wed, 05 Jul 2017 11:55:26 -0400] rev 33260
merge with stable
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 02 Jul 2017 04:26:42 +0200] rev 33259
vfs: drop the 'mustaudit' API
There are no remaining users of 'mustaudit' so we can safely drop the API.
External user are unlikely from a quick research so no deprecation is added.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 02 Jul 2017 04:26:34 +0200] rev 33258
streamclone: stop using 'vfs.mustaudit = False'
Now that each call disable the auditing on its own, we can safely drop this the
mustaudit usage. No other code is modified.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 02 Jul 2017 02:28:04 +0200] rev 33257
vfs: simplify path audit disabling in stream clone
The whole 'mustaudit' API is quite complex compared to its actual usage by its
unique user in stream clone.
Instead we add a "auditpath" parameter to 'vfs.__call_'. The stream clone code
then explicitly open files with path auditing disabled.
The 'mustaudit' API will be cleaned up in the next changeset.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 02 Jul 2017 02:19:05 +0200] rev 33256
auditor: add simple comment about repo.auditor and al
Every once in a while, I get confused by what these are. Let us add a comment.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 02 Jul 2017 01:41:37 +0200] rev 33255
tag: make sure the repository is locked when tagging
Otherwise, writing localtag can happen without the lock.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 02 Jul 2017 01:38:08 +0200] rev 33254
test: glob a line number in test-devel-warnings.t
This make is simpler to edit the extensions file without side effect.
Pierre-Yves David <pierre-yves.david@octobus.net> [Sun, 02 Jul 2017 01:37:03 +0200] rev 33253
test: add a small comment to explain a section of test-devel-warning
This makes each test boundaries clearer.
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 28 Jun 2017 03:54:19 +0200] rev 33252
obsolete: reports the number of local changeset obsoleted when unbundling
This is a first basic visible usage of the changes tracking in the transaction.
We adds a new function computing the pre-existing changesets obsoleted by a
transaction and a transaction call back displaying this information.
Example output:
added 1 changesets with 1 changes to 1 files (+1 heads)
3 new obsolescence markers
obsoleted 1 changesets
The goal is to evolve the transaction summary into something bigger, gathering
existing output there and adding new useful one. This patch is a good first step
on this road. The new output is basic but give a user to the content of
tr.changes['obsmarkers'] and give an idea of the new options we haves. I expect
to revisit the message soon.
The caller recording the transaction summary should also be moved into a more
generic location but further refactoring is needed before it can happen.
Pierre-Yves David <pierre-yves.david@octobus.net> [Tue, 27 Jun 2017 02:45:09 +0200] rev 33251
transaction: track new obsmarkers in the 'changes' mapping
The obsstore collaborate with transaction to make sure we track all the
obsmarkers added during a transaction. This will be useful for various usages:
hooks, caches, better output, etc.
This is the seconds kind of data added to tr.changes (first one was added revisions)
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:44:04 +0200] rev 33250
configitems: register the 'progress.estimate' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:44:02 +0200] rev 33249
configitems: register the 'progress.clear-complete' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:44:01 +0200] rev 33248
configitems: register the 'progress.assume-tty' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:30 +0200] rev 33247
configitems: register the 'format.usestore' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:29 +0200] rev 33246
configitems: register the 'format.usegeneraldelta' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:28 +0200] rev 33245
configitems: register the 'format.usefncache' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:27 +0200] rev 33244
configitems: register the 'format.obsstore-version' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:15 +0200] rev 33243
configitems: register the 'factotum.service' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:13 +0200] rev 33242
configitems: register the 'factotum.mountpoint' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:12 +0200] rev 33241
configitems: register the 'factotum.executable' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:26 +0200] rev 33240
configitems: register the 'format.maxchainlen' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:24 +0200] rev 33239
configitems: register the 'format.manifestcachesize' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:23 +0200] rev 33238
configitems: register the 'format.generaldelta' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:22 +0200] rev 33237
configitems: register the 'format.dotencode' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:21 +0200] rev 33236
configitems: register the 'format.chunkcachesize' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:42:20 +0200] rev 33235
configitems: register the 'format.aggressivemergedeltas' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Wed, 05 Jul 2017 00:01:30 +0200] rev 33234
configitems: gather comment related to 'worker.backgroundclosemaxqueue'
Thanks to Yuya for pointing this out.
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:46:01 +0200] rev 33233
configitems: register the 'worker.numcpus' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:46:00 +0200] rev 33232
configitems: register the 'worker.backgroundclosethreadcount' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:45:59 +0200] rev 33231
configitems: register the 'worker.backgroundcloseminfilecount' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:45:58 +0200] rev 33230
configitems: register the 'worker.backgroundclosemaxqueue' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:43:35 +0200] rev 33229
configitems: register the 'patch.eol' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:44:16 +0200] rev 33228
configitems: register the 'server.zliblevel' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:44:15 +0200] rev 33227
configitems: register the 'server.validate' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:44:14 +0200] rev 33226
configitems: register the 'server.uncompressedallowsecret' config
Pierre-Yves David <pierre-yves.david@octobus.net> [Fri, 30 Jun 2017 03:44:12 +0200] rev 33225
configitems: register the 'server.preferuncompressed' config