Siddharth Agarwal <sid0@fb.com> [Fri, 14 Nov 2014 11:33:52 -0800] rev 23332
ancestor.missingancestors: don't discard from basesvisit
We only actually care about whether revsvisit is empty, so we can let
basesvisit grow to arbitrary size.
It turns out that this actually helps performance. For a large repo with
hundreds of thousands of commits, hg perfrevset 'only(0, tip)' (basically the
worst case, involving a full DAG traversal) goes from 1.63 seconds to 1.50. hg
perfrevset 'only(tip, 0)' remains unchanged at 1.98 seconds.
Siddharth Agarwal <sid0@fb.com> [Sat, 15 Nov 2014 10:55:34 -0800] rev 23331
test-ancestor: use random testing for missing ancestors
We're going to make changes to the missing ancestor algorithm, and random
testing will give us much more confidence than a fixed set of tests.
Siddharth Agarwal <sid0@fb.com> [Sat, 15 Nov 2014 18:52:44 -0800] rev 23330
test-ancestor: define a main function
We're going to add to it in upcoming patches.
Siddharth Agarwal <sid0@fb.com> [Fri, 14 Nov 2014 14:50:03 -0800] rev 23329
test-ancestor: test iteration for lazyancestors
This has some test coverage in test-revlog-ancestry.py, but not very much.
Siddharth Agarwal <sid0@fb.com> [Fri, 14 Nov 2014 14:36:25 -0800] rev 23328
ancestor.lazyancestors: take parentrevs function rather than changelog
Principle of least privilege, and it also brings this in line with
missingancestors.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 16 Nov 2014 00:24:23 -0500] rev 23327
remove: avoid a bogus warning about no tracked files when removing '.'
Previously, any files relative to the root of the repo that match the -I
patterns would be deleted, but the command exited with 1 after printing a
warning:
$ hg remove -S -I 're:.*.txt' .
removing sub1/sub2/folder/test.txt
removing sub1/sub2/test.txt
not removing .: no tracked files
Matt Harbison <matt_harbison@yahoo.com> [Wed, 12 Nov 2014 23:15:20 -0500] rev 23326
remove: support remove with explicit paths in subrepos
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Nov 2014 21:36:19 -0500] rev 23325
remove: recurse into subrepositories with --subrepos/-S flag
Like 'forget', git and svn subrepos are currently not supported. Unfortunately
the name 'remove' is already used in the subrepo classes, so we break the
convention of naming the subrepo function after the command.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Nov 2014 13:50:43 +0900] rev 23324
cmdserver: protect pipe server streams against corruption caused by direct io
Because pipe-mode server uses stdio as IPC channel, other modules should not
touch stdio directly and use ui instead. However, this strategy is brittle
because several Python functions read and write stdio implicitly.
print 'hello' # should use ui.write()
# => ch = 'h', size =
1701604463 'ello', data = '\n'
This patch adds protection for such mistakes. Both stdio files and low-level
file descriptors are redirected to /dev/null while command server uses them.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Nov 2014 13:04:41 +0900] rev 23323
cmdserver: postpone creation of pipe server until run()
This makes it easy to swap file descriptors while running command server.
Yuya Nishihara <yuya@tcha.org> [Sat, 15 Nov 2014 12:43:35 +0900] rev 23322
cmdserver: use given streams as pipe channels like other commands
Because commandserver itself is an hg subcommand, it shouldn't use stdio
directly in principle.
Siddharth Agarwal <sid0@fb.com> [Fri, 14 Nov 2014 16:38:58 -0800] rev 23321
revset.only: use cl.findmissingrevs
ancestor.missingancestors is really an implementation detail.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 14 Nov 2014 09:33:28 -0800] rev 23320
manifestmerge: use already existing fl2 synonym for m2.flags(f)
Probably not a noticeable performance gain, but shortens the code
slightly.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 13 Nov 2014 23:12:15 -0800] rev 23319
merge: drop underscore prefix from _checkunknown()
The method has been called from commands.py since
3eab42088be4
(update: just merge unknown file collisions, 2012-02-09), so drop the
underscore prefix that suggests that it's private.
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 12 Nov 2014 14:47:48 +0000] rev 23318
transaction: drop special handling for phases and bookmarks generation
We are still doing double backups, but now that we have proper
location handling this is less of an issue. Dropping this simplifies
the code before we add some pending-related logic.
This also ensures we actually test the new 'location' mechanism.
Pierre-Yves David <pierre-yves.david@fb.com> [Fri, 17 Oct 2014 20:53:42 -0700] rev 23317
transaction: use 'location' instead of 'vfs' objects for file generation
The argument is now a location name. The location must be present in the
'vfsmap' provided to the transaction at creation time.
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 05 Nov 2014 01:59:32 +0000] rev 23316
transaction: use 'location' instead of 'vfs' in the addbackup method
This unlock the backup of file outside of store (eg: bookmarks).
Pierre-Yves David <pierre-yves.david@fb.com> [Fri, 14 Nov 2014 00:14:23 +0000] rev 23315
addbackup: handle file in subdirectory
The current naming scheme ('journal.backups.<file>') resulted is bad directory
name when 'file' was in a subdirectory. We now extract the directory name and
create the backupfile within it.
We plan to use file in a subdirectory for cachefile.
Pierre-Yves David <pierre-yves.david@fb.com> [Fri, 14 Nov 2014 14:54:55 +0000] rev 23314
addbackup: use the vfs for the backup destination too
The backup file location was always computed using the opener, bypassing the
'location' setting. (And making the feature broken.)
Pierre-Yves David <pierre-yves.david@fb.com> [Thu, 13 Nov 2014 11:17:36 +0000] rev 23313
transaction: set backupentries version to proper value
Now that all mechanisms are in place, we can advertise it with a
proper new version.
Pierre-Yves David <pierre-yves.david@fb.com> [Thu, 13 Nov 2014 11:17:09 +0000] rev 23312
transaction: support cache file in backupentries
We do not want to abort if anything wrong happen while handling a cache file.
Cache file have way to be invalidated and if old/bad version stay no
misbehavior will happen. Proper value will eventually be computed and the wrong
will be righten.
This changeset use the transaction reporter (usually writing on stderr) to write
details about failed cache handling. This will only apply to write operation
using a transaction. The usual update during read only operation will stay a
debug message.
I was on the way to bring these message back to debug level when I realised it
could be a feature. People with write access to the repository are likely to
have the power to fix error related to cache (and it is valuable to fix them).
So let the things as is for now.
Pierre-Yves David <pierre-yves.david@fb.com> [Fri, 17 Oct 2014 21:04:35 -0700] rev 23311
transaction: use the location value when doing backup
We finally use the 'location' value coupled with the 'vfsmap' to restore backup
for the right file.
Pierre-Yves David <pierre-yves.david@fb.com> [Fri, 17 Oct 2014 20:49:39 -0700] rev 23310
transaction: pass a vfs map to the transaction
The goal is to allow access to file outside ofthe store directory from the
transaction. The obvious target are the `bookmarks` file. But we can envision
usage for cache too.
We keep passing a main opener explicitly because a lot of code rely on this
default opener. The main opener (operating on store) is using an empty key ''.
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 05 Nov 2014 01:52:46 +0000] rev 23309
transaction: change the on disk format for backupentries
We need to store new data to improve the current transaction logic:
- location: We want to generate and backup file outside of the 'store' (eg:
bookmarks, or various cache files). This requires knowing and preserving where
each file is located. The value of this new field is a string. It will be used
as a key for a vfs mapping.
- cache: We would like to handle cache file in the transaction code. This
Will help to have cache consistent with the repository state and avoid
performance issue on big repository like Mozilla. However, failure to handle
cache file should not result in a transaction failure. We add a new field that
carry this information. The value is boolean, A True value mean any error
while handling this file can be ignored.
Those two mechanisms are not implemented yet, but they are now persisted in the
on disk file. Support for new mechanisms is coming in later changeset.
We update the file format now and will introduce the new features in later
changeset. The format version is set to 0 until we actually support the new feature.
This will prevent misunderstanding between incomplete and final client.
Support for reading both version 1 and (future) version 2 could be achieved
(using default value when reading version 1) but has not been seen as necessary
for now.
Augie Fackler <augie@google.com> [Thu, 13 Nov 2014 15:47:15 -0500] rev 23308
silenttestrunner: add environment variable to make tests noisy again
As I've been working on complicated extension code it's been handy to
be able to get standard unittest verbose output so I can find crashers
more efficiently.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 14 Nov 2014 05:58:59 -0800] rev 23307
largefiles: update comments to refer to the right overridden method
This cleans up leftovers from
b228ad1f79d7 (largefiles: override
calculateupdates instead of manifestmerge, 2014-03-02).
Siddharth Agarwal <sid0@fb.com> [Thu, 13 Nov 2014 21:36:38 -0800] rev 23306
revlog: cache chain info after calculating it for a rev (
issue4452)
This dumb cache works surprisingly well: on a repository with typical delta
chains ~50k in length, unbundling a linear series of 5000 revisions (changelogs
and manifests only) went from 60 seconds to 3.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 22 Oct 2014 21:38:30 -0700] rev 23305
manifest: add matches() method
Move the code in context._manifestmatches() into a new
manifest.matches(). It's a natural place for the code to live and it
allows other callers to easily use it. It should also make it easier
to optimize the new method in alternative implementations of the
manifest (same reasoning as with manifest.diff()).
Martin von Zweigbergk <martinvonz@google.com> [Wed, 12 Nov 2014 22:20:36 -0800] rev 23304
context.status: pass status tuple into _buildstatus
By passing a status tuple (instead of the current list), we can access
the status fields by name and make it a little more readable.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 12 Nov 2014 22:07:31 -0800] rev 23303
context.status: avoid de- and reconstructing status tuple
We can just modify the status tuple we got from dirstate.status()
instead of deconstructing it and constructing a new instance, thereby
simplifying the code a little.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 12 Nov 2014 16:51:11 -0800] rev 23302
context.status: make _dirstatestatus() return an status tuple
Letting _dirstatestatus() return an scmutil.status instance also means
that _buildstatus() will always return such an instance, so we can
remove the conversion from the call sites.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 12 Nov 2014 21:19:07 -0800] rev 23301
context.status: wipe deleted/unknown/ignored fields when reversed
It makes no sense to request reverse status (i.e. changes from the
working copy to its parent) and then look at the deleted, unknown or
ignored fields. If you do, you would get the result from the forward
status (changes from parent to the working copy). Instead of giving a
nonsensical answer to a nonsensical question, it seems a little saner
to return empty lists. It might be best if we could prevent the caller
accessing these lists, but it's doubtful it's worth the trouble.
Siddharth Agarwal <sid0@fb.com> [Wed, 12 Nov 2014 23:50:21 -0800] rev 23300
patch.trydiff: add support for noprefix
Siddharth Agarwal <sid0@fb.com> [Wed, 12 Nov 2014 23:29:14 -0800] rev 23299
mdiff.unidiff: add support for noprefix
Siddharth Agarwal <sid0@fb.com> [Thu, 13 Nov 2014 00:13:48 -0800] rev 23298
diff: add a --noprefix option
See previous patch descriptions for the motivation.
The tests reflect the current state of the world -- as we add support we'll see
changes in the test output.
Siddharth Agarwal <sid0@fb.com> [Thu, 13 Nov 2014 00:08:44 -0800] rev 23297
patch.diffopts: add support for noprefix
In an upcoming patch we'll enable support as an option to 'hg diff' as well.
The tests reflect the current state of the world -- as we add support we'll see
changes in the test output.
Siddharth Agarwal <sid0@fb.com> [Wed, 12 Nov 2014 23:47:25 -0800] rev 23296
patch.diffopts: allow a setting to be forced in plain mode
Upcoming patches will add an option that will almost certainly break diff
output parsers when enabled. Add support for forcing an option to something in
plain mode, as a fallback. Options passed in via the CLI are not affected,
though -- it is assumed that any script passing the option in explicitly knows
what it is doing.
Siddharth Agarwal <sid0@fb.com> [Wed, 12 Nov 2014 23:44:17 -0800] rev 23295
patch.diffopts: break get function into if statements
We're going to add another condition here, and with the current structure that
becomes just too confusing.
Siddharth Agarwal <sid0@fb.com> [Wed, 12 Nov 2014 23:25:32 -0800] rev 23294
mdiff.diffopts: add a new noprefix option
By popular demand, we introduce an option to disable the 'a/' and 'b/'
prefixes in diff output. This makes copying and pasting filenames from diff
output easier.
This option will be implemented and documented in upcoming patches. To ensure
that existing scripts that parse output don't break, we will ensure that this
prefix is disabled in plain mode. A straight 'hg export | hg import' without
HGPLAIN=1 will still be broken though, but there's little that can be done
about that.
Siddharth Agarwal <sid0@fb.com> [Wed, 12 Nov 2014 23:19:44 -0800] rev 23293
mdiff.diffopts: add doc comment for nobinary
Pierre-Yves David <pierre-yves.david@fb.com> [Sat, 08 Nov 2014 17:08:09 +0000] rev 23292
changelog: register changelog.i.a as a temporary file
The file is registered to make sure the transaction is cleaned up in all cases.
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 05 Nov 2014 09:27:08 +0000] rev 23291
transaction: allow registering a temporary transaction file
During the transaction, files may be created to store or expose data
involved in the transaction (eg: changelog index data are written in
a 'changelog.i.a' for hooks). But we do not have an official way to
record such file creation and make sure they are cleaned up. The lack
of clean-up is currently okay because there is a single file involved
and a single producer/consumer.
However, as we want to expose more data (bookmarks, phases, obsmarker)
we need something more solid. The 'backupentries' mechanism could
handle that. Temporary files can be encoded as a backup of nothing
'('', <temporarypath>)'. We "need" to attach it to the same mechanism
as we use to be able to use temporary transaction files outside of
.'store/' and 'backupentries' is expected to gain such feature.
This changeset makes it clear that we should rename 'backupentries' to
something more generic.
Pierre-Yves David <pierre-yves.david@fb.com> [Thu, 13 Nov 2014 10:22:47 +0000] rev 23290
transaction: always generate file on close
The conditionnal was buggy and file were only generated if "onclose" was
defined. By luck, "onclose" was always defined.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 09 Nov 2014 12:31:34 -0500] rev 23289
remove: move most of the implementation into cmdutils.remove()
This will allow access to the reusable parts from subrepos, similar to add(),
forget(), etc.
Siddharth Agarwal <sid0@fb.com> [Tue, 11 Nov 2014 20:08:19 -0800] rev 23288
revlog: increase I/O bound to 4x the amount of data consumed
This doesn't affect normal clones since they'd be bound by the CPU bound below
anyway -- it does, however, improve generaldelta clones significantly.
This also results in better deltaing for generaldelta clones -- in generaldelta
clones, we calculate deltas with respect to the closest base if it has a higher
revision number than either parent. If the base is on a significantly different
branch, this can result in pointlessly massive deltas. This reduces the number
of bases and hence the number of bad deltas.
Empirically, for a highly branchy repository, this resulted in an improvement
of around 15% to manifest size.
Siddharth Agarwal <sid0@fb.com> [Tue, 11 Nov 2014 20:01:19 -0800] rev 23287
revlog: bound based on the length of the compressed deltas
This is only relevant for generaldelta clones.
Siddharth Agarwal <sid0@fb.com> [Tue, 11 Nov 2014 19:54:36 -0800] rev 23286
revlog: compute length of compressed deltas along with chain length
In upcoming patches to the revlog, we're going to split up the notions of
bounding I/O and bounding CPU.
Siddharth Agarwal <sid0@fb.com> [Tue, 11 Nov 2014 21:41:12 -0800] rev 23285
revlog: store fulltext when compressed delta is bigger than it
This is a very silly case and not particularly likely to happen in the wild,
but it turns out we can hit it in a couple of places. As we tune the storage
parameters we're likely to hit more such cases.
The affected test cases all have smaller revlogs now.
Siddharth Agarwal <sid0@fb.com> [Tue, 11 Nov 2014 21:39:56 -0800] rev 23284
revlog: make a predicate clearer with parens
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 05 Nov 2014 13:06:24 +0000] rev 23283
transaction: extract backupentry registration in a dedicated function
We are about to use the 'backupentry' mechanism to allow cleaning up
transaction-related temporary files (such as 'changelog.i.a'). We start
by extracting the entry registration into its own method for easy reuse.
At that point, I would like to rename the backup-file related variable to
something generic but I'm a bit short of ideas.
Pierre-Yves David <pierre-yves.david@fb.com> [Sat, 08 Nov 2014 16:35:15 +0000] rev 23282
transaction: pass the transaction to 'postclose' callback
This mirrors the API for 'pending' and 'finalize' callbacks. I do not have
immediate usage planned for it, but I'm sure some callback will be happy to
access transaction related data.
Pierre-Yves David <pierre-yves.david@fb.com> [Sat, 08 Nov 2014 16:31:38 +0000] rev 23281
transaction: pass the transaction to 'finalize' callback
The callback will likely need to perform some operation related to the
transaction (eg: registering file update). So we better pass the current
transaction as the callback argument. Otherwise callback that needs it has to
rely on horrible weak reference trick.
This allow already allow us to slay a wild weak reference usage.
Pierre-Yves David <pierre-yves.david@fb.com> [Sat, 08 Nov 2014 16:27:50 +0000] rev 23280
transaction: pass the transaction to 'pending' callback
The callback will likely need to perform some operation related to the
transaction (eg: backing files up). So we better pass the current transaction as
the callback argument. Otherwise callback that needs it has to rely on horrible
weak reference trick.
The first foreseen user of this is changelog._writepending. We would like it to
register the temporary file it create for cleanup purpose.
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 05 Nov 2014 10:22:17 +0000] rev 23279
transaction: gather backupjournal logic together in the __init__
The initialisation of file-backup related variable were a bit scattered, we
gather them together.
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 05 Nov 2014 01:38:48 +0000] rev 23278
transaction: handle missing file in backupentries (instead of using entries)
The case where a backup of a missing file was requested was previously
handled by the 'entries' list. As the 'backupentries' is about to gain
ability to backup files outside of '.hg/store', we want it to be able
to handle the missing file too.
Reminder: using 'addbackup' on a missing file means that such file needs to be
deleted if we rollback the transaction.
Pierre-Yves David <pierre-yves.david@fb.com> [Wed, 05 Nov 2014 01:23:40 +0000] rev 23277
test-hup: use ls instead of echo
Having one file entry per line makes it easier to analyse diffs.