Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 Jul 2017 12:14:12 -0700] rev 33298
sparse: move profile reading into core
One more step towards weaning off methods on repo instances and
moving code to core. While this function is only used once and
is simple, it needs to exist on its own so Facebook can monkeypatch
it to enable simplecache integration.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 Jul 2017 12:14:03 -0700] rev 33297
sparse: move config parsing into core
This patch marks the beginning of moving code from the sparse
extension into core. The goal is to move as much of the
functionality as possible into core, where it will be an
experimental feature. The extension will likely continue to
exist to enable the feature and provide UI elements.
As part of the move, the repo method was converted to a module
function. It doesn't need to exist on repos.
An error message was also updated to reflect that an error isn't
necessarily from the .hg/sparse file. The API should be updated
later to pass in a filename so the error can be more descriptive.
Copyright of the added file was copied from the sparse extension.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 Jul 2017 10:58:45 -0700] rev 33296
sparse: use vfs.tryread()
vfs.exists() followed by a file read is an anti-pattern because it
incurs an extra stat() to test for file presence. vfs.tryread()
returns empty string on missing file and avoids the stat().
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 01 Jul 2017 11:56:39 -0700] rev 33295
sparse: refactor sparsechecksum()
This was relying on garbage collection to close the opened
file, which is a bug. Both callers simply called into self.vfs
to resolve the path. So refactor to use the vfs layer.
While we're here, rename the method to reflect it is internal
and to break anyone relying on the old behavior.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 Jul 2017 10:57:26 -0700] rev 33294
sparse: document config file format
This was previously undocumented. Seems useful to have.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 01 Jul 2017 10:29:27 -0700] rev 33293
sparse: rename command to debugsparse
Sparse checkout is still highly experimental and not protected
by BC guarantees yet. We also haven't had a discussion on the UX.
To discourage use, we rename the sparse command to debugsparse.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 Jul 2017 10:54:23 -0700] rev 33292
sparse: remove reference to simplecache
This is a 3rd party extension authored by Facebook. References in
core are not appropriate.
It will be possible to restore this code/optimization via
monkeypatching. So Facebook won't lose any functionality.
The removed code is important for performance. So add a comment
tracking it.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 01 Jul 2017 10:24:31 -0700] rev 33291
sparse: remove reference to hgwatchman
This is a legacy extension. Now that the extension is in core,
we only need to support what's in core, which is fsmonitor.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 01 Jul 2017 10:36:03 -0700] rev 33290
sparse: expand module docstring
Clarify lack of BC guarantees. And say a bit more about the extension.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 01 Jul 2017 10:43:29 -0700] rev 33289
sparse: vendor Facebook-developed extension
Facebook has developed an extension to enable "sparse" checkouts -
a working directory with a subset of files. This feature is a critical
component in enabling repositories to scale to infinite number of
files while retaining reasonable performance. It's worth noting
that sparse checkout is only one possible solution to this problem:
another is virtual filesystems that realize files on first access.
But given that virtual filesystems may not be accessible to all
users, sparse checkout is necessary as a fallback.
Per mailing list discussion at
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-March/095868.html
we want to add sparse checkout to the Mercurial distribution via
roughly the following mechanism:
1. Vendor extension as-is with minimal modifications (this patch)
2. Refactor extension so it is more clearly experimental and inline
with Mercurial practices
3. Move code from extension into core where possible
4. Drop experimental labeling and/or move feature into core
after sign-off from narrow clone feature owners
This commit essentially copies the sparse extension and tests
from revision 71e0a2aeca92a4078fe1b8c76e32c88ff1929737 of the
https://bitbucket.org/facebook/hg-experimental repository.
A list of modifications made as part of vendoring is as follows:
* "EXPERIMENTAL" added to module docstring
* Imports were changed to match Mercurial style conventions
* "testedwith" value was updated to core Mercurial special value and
comment boilerplate was inserted
* A "clone_sparse" function was renamed to "clonesparse" to appease
the style checker
* Paths to the sparse extension in tests reflect built-in location
* test-sparse-extensions.t was renamed to test-sparse-fsmonitor.t
and references to "simplecache" were removed. The test always skips
because it isn't trivial to run it given the way we currently run
fsmonitor tests
* A double empty line was removed from test-sparse-profiles.t
There are aspects of the added code that are obviously not ideal.
The goal is to make a minimal number of modifications as part of
the vendoring to make it easier to track changes from the original
implementation. Refactoring will occur in subsequent patches.
Augie Fackler <augie@google.com> [Thu, 06 Jul 2017 15:15:02 -0400] rev 33288
contrib: widen "direct use of `python`" net again
I think I've now caught all of them.
Differential Revision: https://phab.mercurial-scm.org/D15
Kevin Bullock <kbullock+mercurial@ringworld.org> [Thu, 06 Jul 2017 14:33:48 -0500] rev 33287
tests: clean up a newly-introduced instance of `python`
Differential Revision: https://phab.mercurial-scm.org/D16
Augie Fackler <augie@google.com> [Tue, 20 Jun 2017 17:31:18 -0400] rev 33286
tests: clean up even more direct `python` calls with $PYTHON
This time ones that are prefixed with =, ", ', or `. This appears to
be the last of them.
Differential Revision: https://phab.mercurial-scm.org/D14
Augie Fackler <augie@google.com> [Tue, 20 Jun 2017 17:25:57 -0400] rev 33285
contrib: widen the "don't use `python`" net a little
I'm still cleaning this up, but it's easier to do in bite-size chunks
like this than all at once. The negative lookahead avoids one false
positive category from some output related to finding Subversion
bindings.
Differential Revision: https://phab.mercurial-scm.org/D13
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 05 Jul 2017 13:54:53 +0200] rev 33284
followlines: join merge parents line ranges in blockdescendants() (issue5595)
In blockdescendants(), we had an assertion when line range of a merge
changeset was not consistent depending on which parent was considered for
computation. For instance, this might occur when file content (in lookup
range) is significantly different between parent branches of the merge as
demonstrated in added tests (where we almost completely rewrite the "baz" file
while also introducing similarities with its content in the other branch we
later merge to).
Now, in such case, we combine line ranges from all parents by storing the
envelope of both line ranges. This is conservative (the line range is
extended, possibly unnecessarily) but at least this should avoid missing
descendants with changes in a range that would fall in that of one parent but
not in another one (the case of "baz: narrow change (2->2+)" changeset in
tests).
Phil Cohen <phillco@fb.com> [Tue, 04 Jul 2017 22:35:52 -0700] rev 33283
workingfilectx: add exists, lexists
Switch the lone call in merge.py to use it.
As with past refactors, the goal is to make wctx hot-swappable with an
in-memory context in the future. This change should be a no-op today.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:47 +0900] rev 33282
vfs: add explanation about cost of checkambig=True in corner case
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:47 +0900] rev 33281
vfs: replace avoiding ambiguity in abstractvfs.rename with _avoidambig
This centralizes common logic to forcibly avoid file stat ambiguity
into _avoidambig(), which was introduced by previous patch.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:47 +0900] rev 33280
vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing
Now, files (to be truncated) are opened with checkambig=True, only if
localrepository caches it.
Therefore, straightforward "copy if EPERM" is always reasonable to
avoid file stat ambiguity at closing.
This patch makes checkambigatclosing close wrapper copy the target
file, and advance mtime on it after renaming, if EPERM. This can avoid
file stat ambiguity, even if the target file is owned by another (see
issue5418 and issue5584 for detail).
This patch factors main logic out instead of changing
checkambigatclosing._checkambig() directly, in order to reuse it.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:47 +0900] rev 33279
transaction: apply checkambig=True only on limited files for similarity
Now, transaction can determine whether avoidance of file stat
ambiguity is needed for each files, by blacklist "checkambigfiles".
For similarity to truncation in _playback(), this patch apply
checkambig=True only on limited files in code paths below.
- restoring files by util.copyfile(), in _playback()
(checkambigfiles itself is examined at first, because it as a
keyword argument might be None)
- writing files at finalization of transaction, in _generatefiles()
This patch reduces cost of checking stat at writing out and restoring
files, which aren't filecache-ed.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:46 +0900] rev 33278
transaction: avoid file stat ambiguity only for files in blacklist
Advancing mtime by os.utime() fails for EPERM, if the target file is
owned by another. bff5ccbe5ead and related changes made some code
paths give advancing mtime up in such case, to fix issue5418.
This causes file stat ambiguity (again), if it is owned by another.
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
To avoid file stat ambiguity in such case, especially for
.hg/dirstate, ed66ec39933f made vfs.rename() copy the target file, and
advance mtime of renamed one again, if EPERM (see issue5584 for detail).
But straightforward "copy if EPERM" isn't reasonable for truncation of
append-only files at rollbacking, because rollbacking might cost much
for truncation of many filelogs, even though filelogs aren't
filecache-ed.
Therefore, this patch introduces blacklist "checkambigfiles", and
avoids file stat ambiguity only for files specified in this blacklist.
This patch consists of two parts below, which should be applied at
once in order to avoid regression.
- specify 'checkambig=True' at vfs.open(mode='a') in _playback()
according to checkambigfiles
- invoke _playback() with checkambigfiles
- add transaction.__init__() checkambigfiles argument, for _abort()
- make localrepo instantiate transaction with _cachedfiles
- add rollback() checkambigfiles argument, for "hg rollback/recover"
- make localrepo invoke rollback() with _cachedfiles
After this patch, straightforward "copy if EPERM" will be reasonable
at closing the file opened with checkambig=True, because this policy
is applied only on files, which are listed in blacklist
"checkambigfiles".
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 04 Jul 2017 23:13:46 +0900] rev 33277
localrepo: store path and vfs location of cached properties
This information is used to make transaction handle these files
specially, in order to avoid file stat ambiguity of them.
Gathering information about cached files via annotation classes can
avoid overlooking properties newly introduced in the future.
Boris Feld <boris.feld@octobus.net> [Mon, 03 Jul 2017 11:22:00 +0200] rev 33276
template: add successors template
Add a 'successorssets' template that returns the list of all closest known
sucessorssets for a changectx. The elements of the list are changesets.
The "closest successors" are the first locally known revisions encountered
while, walking successors markers. It uses successorsets previously modified
to support the closest argument.
This logic respect repository filtering. So hidden revision will be skipped by
this logic unless --hidden is specified. Since we only display the visible
predecessors, this template will not display anything in most case. It makes a
good candidate for inclusion in the default log output.
I updated the test-obsmarker-template.t test file introduced with the
predecessors template to test successorssets template.
Boris Feld <boris.feld@octobus.net> [Mon, 03 Jul 2017 14:22:28 +0200] rev 33275
template: add tests for more complex cases
We add new tests for improving the coverage of existing obs-markers related
template (predecessors) and the new one we are introducing (successorssets).