transaction: allow generating files with a suffix This will allow us to generate temporary pending files. Files generated with a suffix are assumed temporary and will be cleaned up at the end of the transaction.

transaction: fix some docstring grammar

transaction: accept a 'location' argument for registertmp This will allow generation of temporary files outside of store. This will be useful for bookmarks.

tests: handle differences between missing file error strings on Windows and Unix

run-tests: don't warn on unnecessary globs mandated by check-code.py When test output is processed, if os.altsep is defined (i.e. on Windows), TTest.globmatch() will cause a warning later on if a line has a glob that isn't necessary. Unfortunately, the regex checking in check-code.py doesn't have this context. Therefore we ended up with cases where the test would get flagged with a warning only on Windows because a glob was present, because check-code.py would warn if it wasn't. For example, from test-subrepo.t: $ hg -R issue1852a push `pwd`/issue1852c pushing to $TESTTMP/issue1852c (glob) The glob isn't necessary here because the slash is shown as it was provided. However, check-code mandates one to handle the case where the default path has backslashes in it. Break the cycle by checking against a subset of the check-code rules before flagging the test with a warning, and ignore the superfluous glob if it matches a rule. This change fixes warnings in test-largefiles-update.t, test-subrepo.t, test-tag.t, and test-rename-dir-merge.t on Windows. I really hate that the rules are copy/pasted here (minus the leading two spaces) because it would be nice to only update the rules once, in a single place. But I'm not sure how else to do it. I'm open to suggestions. Splitting some of the rules out of check-code.py seems wrong, but so does moving check-code.py out of contrib, given that other checking scripts live there. There are other glob patterns that could be copied over, but this is enough to make the current tests run on Windows.

update: remove unnecessary check for unknown files with --check As far as I and the test suite can tell, the checks in manifestmerge() already report the errors (whether or not --check is given), so we don't need to call merge.checkunknown(). Since this is the last call to the method, also remove the method.

merge with stable

tests: move a multi-statement debuglocks hook into a shell script for Windows Before this patch, a part of "test-push-hook-lock.t" fails unexpectedly on Windows environment, because semicolon (";") isn't recognized as the command separator by "cmd.exe". This is fixed the same way as a similar issue in 7c253c23de3b.

tests: fix globs for Windows test-largefiles-update.t, test-subrepo.t, test-tag.t, and test-rename-dir-merge.t still warn about no result returned because of unnecessary globs that test-check-code-hg.t wants, relating to output for pushing to, pulling from and moving X to Y.

run-tests: include quotes in the HGEDITOR value when storing sys.executable This fixes test-install.t on Windows that broke in 2122b82b6987 when shlex.split() was added to the debuginstall command: @@ -7,8 +7,11 @@ checking installed modules (*mercurial)... (glob) checking templates (*mercurial?templates)... (glob) checking commit editor... + Can't find editor 'c:\Python27\python.exe -c "(omitted)"' in PATH + (specify a commit editor in your configuration file) checking username... - no problems detected + 1 problems detected, please check your install! + [1] What happens is that shlex.split() on Windows turns this: c:\Python27\python.exe -c "import sys; sys.exit(0)" into this: ['c:Python27python.exe', '-c', 'import sys; sys.exit(0)'] While technically a regression, most programs on Windows live in some flavor of 'Program Files', and therefore the environment variable needs to contain quotes anyway to handle the space. This wasn't handled prior to the shlex() change, because it tested the whole environment variable to see if it was an executable, or split on the first space and tested again.

mq: when adding headers in plain mode, separate them from message (issue4453) c87f2a5a6e49 did a clean-up in one direction ... but we want it in the other direction.

mq: introduce insertplainheader - same naive implementation as before

mq: when setting message in plain mode, separate it from header (issue4453) Fix inconsistent handling of plain header separation in mq patcheader - and contrary to c87f2a5a6e49, do it in the direction of having an empty line between header and description. Plain patches are like mails and should thus have an empty line between headers and body in compliance with RFC 822 3.1.

setdiscovery: avoid a full changelog graph traversal We were definitely being suboptimal here: we were constructing two full sets, one with the full set of common nodes (i.e. a graph traversal) and one with all nodes. Then we subtract one set from the other. This whole process is O(commits) and causes discovery to be significantly slower than it should be. Instead, keep track of common incrementally and keep undecided as small as possible. This makes discovery massively faster on large repos: on one such repo, 'hg debugdiscovery' over SSH with one commit missing on the client and five on the server went from 4.5 seconds to 1.5. (An 'hg debugdiscovery' with no commits missing on the client, i.e. connection startup time, was 1.2 seconds.)

ancestor: add a way to remove ancestors of bases from a given set This and missingancestors can share state, which will turn out to be perfect for set discovery.

ancestor: add a way to add to bases of a missing ancestor object This will be useful for setdiscovery, since with that we incrementally add to our knowledge of common nodes.

ancestor: add a way to test whether a missing ancestor object has bases This is pretty trivial so there's no unit test coverage for it. This will be used by setdiscovery.

ancestor: remove now-unused missingancestors function Callers should use revlog.incrementalmissingrevs instead.

revlog: switch findmissing* methods to incrementalmissingrevs This will allow us to remove ancestor.missingancestors in an upcoming patch.

revlog: add a method to get missing revs incrementally This will turn out to be useful for discovery.

test-ancestor: add support for multiple tests against one incremental object In upcoming patches we'll add more operations to the object, and this prepares for testing those operations.

test-ancestor: move naive missing ancestor algorithm into a class This mirrors the change to the real missing ancestor algorithm in a previous patch.

ancestor.missingancestors: turn into a state-keeping class This allows multiple efficient missing ancestor queries against the same set of bases. In upcoming patches we'll also define ways to grow the set of bases. The fact that the test output hasn't changed establishes this patch's correctness.

ancestor.missingancestors: calculate start point after filtering revsvisit Any revs that are filtered out are also in basesvisit, which means they wouldn't be returned in the missing list anyway. There's no need to explore such revs or their ancestors. The 'if not revsvisit' check moves down because we can't call max() on an empty set.

ancestor.missingancestors: don't discard from basesvisit We only actually care about whether revsvisit is empty, so we can let basesvisit grow to arbitrary size. It turns out that this actually helps performance. For a large repo with hundreds of thousands of commits, hg perfrevset 'only(0, tip)' (basically the worst case, involving a full DAG traversal) goes from 1.63 seconds to 1.50. hg perfrevset 'only(tip, 0)' remains unchanged at 1.98 seconds.

test-ancestor: use random testing for missing ancestors We're going to make changes to the missing ancestor algorithm, and random testing will give us much more confidence than a fixed set of tests.

test-ancestor: define a main function We're going to add to it in upcoming patches.

test-ancestor: test iteration for lazyancestors This has some test coverage in test-revlog-ancestry.py, but not very much.

ancestor.lazyancestors: take parentrevs function rather than changelog Principle of least privilege, and it also brings this in line with missingancestors.

remove: avoid a bogus warning about no tracked files when removing '.' Previously, any files relative to the root of the repo that match the -I patterns would be deleted, but the command exited with 1 after printing a warning: $ hg remove -S -I 're:.*.txt' . removing sub1/sub2/folder/test.txt removing sub1/sub2/test.txt not removing .: no tracked files

remove: support remove with explicit paths in subrepos

remove: recurse into subrepositories with --subrepos/-S flag Like 'forget', git and svn subrepos are currently not supported. Unfortunately the name 'remove' is already used in the subrepo classes, so we break the convention of naming the subrepo function after the command.

cmdserver: protect pipe server streams against corruption caused by direct io Because pipe-mode server uses stdio as IPC channel, other modules should not touch stdio directly and use ui instead. However, this strategy is brittle because several Python functions read and write stdio implicitly. print 'hello' # should use ui.write() # => ch = 'h', size = 1701604463 'ello', data = '\n' This patch adds protection for such mistakes. Both stdio files and low-level file descriptors are redirected to /dev/null while command server uses them.

cmdserver: postpone creation of pipe server until run() This makes it easy to swap file descriptors while running command server.

cmdserver: use given streams as pipe channels like other commands Because commandserver itself is an hg subcommand, it shouldn't use stdio directly in principle.

revset.only: use cl.findmissingrevs ancestor.missingancestors is really an implementation detail.

manifestmerge: use already existing fl2 synonym for m2.flags(f) Probably not a noticeable performance gain, but shortens the code slightly.

merge: drop underscore prefix from _checkunknown() The method has been called from commands.py since 3eab42088be4 (update: just merge unknown file collisions, 2012-02-09), so drop the underscore prefix that suggests that it's private.

transaction: drop special handling for phases and bookmarks generation We are still doing double backups, but now that we have proper location handling this is less of an issue. Dropping this simplifies the code before we add some pending-related logic. This also ensures we actually test the new 'location' mechanism.

transaction: use 'location' instead of 'vfs' objects for file generation The argument is now a location name. The location must be present in the 'vfsmap' provided to the transaction at creation time.

transaction: use 'location' instead of 'vfs' in the addbackup method This unlock the backup of file outside of store (eg: bookmarks).

addbackup: handle file in subdirectory The current naming scheme ('journal.backups.<file>') resulted is bad directory name when 'file' was in a subdirectory. We now extract the directory name and create the backupfile within it. We plan to use file in a subdirectory for cachefile.

addbackup: use the vfs for the backup destination too The backup file location was always computed using the opener, bypassing the 'location' setting. (And making the feature broken.)

transaction: set backupentries version to proper value Now that all mechanisms are in place, we can advertise it with a proper new version.

transaction: support cache file in backupentries We do not want to abort if anything wrong happen while handling a cache file. Cache file have way to be invalidated and if old/bad version stay no misbehavior will happen. Proper value will eventually be computed and the wrong will be righten. This changeset use the transaction reporter (usually writing on stderr) to write details about failed cache handling. This will only apply to write operation using a transaction. The usual update during read only operation will stay a debug message. I was on the way to bring these message back to debug level when I realised it could be a feature. People with write access to the repository are likely to have the power to fix error related to cache (and it is valuable to fix them). So let the things as is for now.

transaction: use the location value when doing backup We finally use the 'location' value coupled with the 'vfsmap' to restore backup for the right file.

transaction: pass a vfs map to the transaction The goal is to allow access to file outside ofthe store directory from the transaction. The obvious target are the `bookmarks` file. But we can envision usage for cache too. We keep passing a main opener explicitly because a lot of code rely on this default opener. The main opener (operating on store) is using an empty key ''.

transaction: change the on disk format for backupentries We need to store new data to improve the current transaction logic: - location: We want to generate and backup file outside of the 'store' (eg: bookmarks, or various cache files). This requires knowing and preserving where each file is located. The value of this new field is a string. It will be used as a key for a vfs mapping. - cache: We would like to handle cache file in the transaction code. This Will help to have cache consistent with the repository state and avoid performance issue on big repository like Mozilla. However, failure to handle cache file should not result in a transaction failure. We add a new field that carry this information. The value is boolean, A True value mean any error while handling this file can be ignored. Those two mechanisms are not implemented yet, but they are now persisted in the on disk file. Support for new mechanisms is coming in later changeset. We update the file format now and will introduce the new features in later changeset. The format version is set to 0 until we actually support the new feature. This will prevent misunderstanding between incomplete and final client. Support for reading both version 1 and (future) version 2 could be achieved (using default value when reading version 1) but has not been seen as necessary for now.

silenttestrunner: add environment variable to make tests noisy again As I've been working on complicated extension code it's been handy to be able to get standard unittest verbose output so I can find crashers more efficiently.

largefiles: update comments to refer to the right overridden method This cleans up leftovers from b228ad1f79d7 (largefiles: override calculateupdates instead of manifestmerge, 2014-03-02).

revlog: cache chain info after calculating it for a rev (issue4452) This dumb cache works surprisingly well: on a repository with typical delta chains ~50k in length, unbundling a linear series of 5000 revisions (changelogs and manifests only) went from 60 seconds to 3.

manifest: add matches() method Move the code in context._manifestmatches() into a new manifest.matches(). It's a natural place for the code to live and it allows other callers to easily use it. It should also make it easier to optimize the new method in alternative implementations of the manifest (same reasoning as with manifest.diff()).

context.status: pass status tuple into _buildstatus By passing a status tuple (instead of the current list), we can access the status fields by name and make it a little more readable.

context.status: avoid de- and reconstructing status tuple We can just modify the status tuple we got from dirstate.status() instead of deconstructing it and constructing a new instance, thereby simplifying the code a little.

context.status: make _dirstatestatus() return an status tuple Letting _dirstatestatus() return an scmutil.status instance also means that _buildstatus() will always return such an instance, so we can remove the conversion from the call sites.

context.status: wipe deleted/unknown/ignored fields when reversed It makes no sense to request reverse status (i.e. changes from the working copy to its parent) and then look at the deleted, unknown or ignored fields. If you do, you would get the result from the forward status (changes from parent to the working copy). Instead of giving a nonsensical answer to a nonsensical question, it seems a little saner to return empty lists. It might be best if we could prevent the caller accessing these lists, but it's doubtful it's worth the trouble.

patch.trydiff: add support for noprefix

mdiff.unidiff: add support for noprefix

diff: add a --noprefix option See previous patch descriptions for the motivation. The tests reflect the current state of the world -- as we add support we'll see changes in the test output.

patch.diffopts: add support for noprefix In an upcoming patch we'll enable support as an option to 'hg diff' as well. The tests reflect the current state of the world -- as we add support we'll see changes in the test output.

patch.diffopts: allow a setting to be forced in plain mode Upcoming patches will add an option that will almost certainly break diff output parsers when enabled. Add support for forcing an option to something in plain mode, as a fallback. Options passed in via the CLI are not affected, though -- it is assumed that any script passing the option in explicitly knows what it is doing.

patch.diffopts: break get function into if statements We're going to add another condition here, and with the current structure that becomes just too confusing.

mdiff.diffopts: add a new noprefix option By popular demand, we introduce an option to disable the 'a/' and 'b/' prefixes in diff output. This makes copying and pasting filenames from diff output easier. This option will be implemented and documented in upcoming patches. To ensure that existing scripts that parse output don't break, we will ensure that this prefix is disabled in plain mode. A straight 'hg export | hg import' without HGPLAIN=1 will still be broken though, but there's little that can be done about that.

mdiff.diffopts: add doc comment for nobinary

changelog: register changelog.i.a as a temporary file The file is registered to make sure the transaction is cleaned up in all cases.

transaction: allow registering a temporary transaction file During the transaction, files may be created to store or expose data involved in the transaction (eg: changelog index data are written in a 'changelog.i.a' for hooks). But we do not have an official way to record such file creation and make sure they are cleaned up. The lack of clean-up is currently okay because there is a single file involved and a single producer/consumer. However, as we want to expose more data (bookmarks, phases, obsmarker) we need something more solid. The 'backupentries' mechanism could handle that. Temporary files can be encoded as a backup of nothing '('', <temporarypath>)'. We "need" to attach it to the same mechanism as we use to be able to use temporary transaction files outside of .'store/' and 'backupentries' is expected to gain such feature. This changeset makes it clear that we should rename 'backupentries' to something more generic.

transaction: always generate file on close The conditionnal was buggy and file were only generated if "onclose" was defined. By luck, "onclose" was always defined.

remove: move most of the implementation into cmdutils.remove() This will allow access to the reusable parts from subrepos, similar to add(), forget(), etc.

revlog: increase I/O bound to 4x the amount of data consumed This doesn't affect normal clones since they'd be bound by the CPU bound below anyway -- it does, however, improve generaldelta clones significantly. This also results in better deltaing for generaldelta clones -- in generaldelta clones, we calculate deltas with respect to the closest base if it has a higher revision number than either parent. If the base is on a significantly different branch, this can result in pointlessly massive deltas. This reduces the number of bases and hence the number of bad deltas. Empirically, for a highly branchy repository, this resulted in an improvement of around 15% to manifest size.

revlog: bound based on the length of the compressed deltas This is only relevant for generaldelta clones.

revlog: compute length of compressed deltas along with chain length In upcoming patches to the revlog, we're going to split up the notions of bounding I/O and bounding CPU.

revlog: store fulltext when compressed delta is bigger than it This is a very silly case and not particularly likely to happen in the wild, but it turns out we can hit it in a couple of places. As we tune the storage parameters we're likely to hit more such cases. The affected test cases all have smaller revlogs now.

revlog: make a predicate clearer with parens

transaction: extract backupentry registration in a dedicated function We are about to use the 'backupentry' mechanism to allow cleaning up transaction-related temporary files (such as 'changelog.i.a'). We start by extracting the entry registration into its own method for easy reuse. At that point, I would like to rename the backup-file related variable to something generic but I'm a bit short of ideas.

transaction: pass the transaction to 'postclose' callback This mirrors the API for 'pending' and 'finalize' callbacks. I do not have immediate usage planned for it, but I'm sure some callback will be happy to access transaction related data.

transaction: pass the transaction to 'finalize' callback The callback will likely need to perform some operation related to the transaction (eg: registering file update). So we better pass the current transaction as the callback argument. Otherwise callback that needs it has to rely on horrible weak reference trick. This allow already allow us to slay a wild weak reference usage.

transaction: pass the transaction to 'pending' callback The callback will likely need to perform some operation related to the transaction (eg: backing files up). So we better pass the current transaction as the callback argument. Otherwise callback that needs it has to rely on horrible weak reference trick. The first foreseen user of this is changelog._writepending. We would like it to register the temporary file it create for cleanup purpose.

transaction: gather backupjournal logic together in the __init__ The initialisation of file-backup related variable were a bit scattered, we gather them together.

transaction: handle missing file in backupentries (instead of using entries) The case where a backup of a missing file was requested was previously handled by the 'entries' list. As the 'backupentries' is about to gain ability to backup files outside of '.hg/store', we want it to be able to handle the missing file too. Reminder: using 'addbackup' on a missing file means that such file needs to be deleted if we rollback the transaction.

test-hup: use ls instead of echo Having one file entry per line makes it easier to analyse diffs.

largefiles: move "copyalltostore" invocation into "markcommitted" Before this patch, while "hg convert", largefiles avoids copying largefiles in the working directory into the store area by combination of setting "repo._isconverting" in "mercurialsink{before|after}" and checking it in "copytostoreabsolute". This avoiding is needed while "hg convert", because converting doesn't update largefiles in the working directory. But this implementation is not efficient, because: - invocation in "markcommitted" can easily ensure updating largefiles in the working directory "markcommitted" is invoked only when new revision is committed via "commit" of "localrepository" (= with files in the working directory). On the other hand, "commitctx" may be invoked directly for in-memory committing. - committing without updating the working directory (e.g. "import --bypass") also needs this kind of avoiding For efficiency of this kind of avoiding, this patch does: - move "copyalltostore" invocation into "markcommitted" - remove meaningless procedures below: - hooking "mercurialsink{before|after}" to (un)set "repo._isconverting" - checking "repo._isconverting" in "copytostoreabsolute" This patch invokes "copyalltostore" also in "_commitcontext", because "_commitcontext" expects that largefiles in the working directory are copied into store area after "commitctx". In this case, the working directory is used as a kind of temporary area to write largefiles out, even though converted revisions are committed via "commitctx" (without updating normal files).

largefiles: avoid printing messages while transplanting by "_lfstatuswriters" Putting "lambda *msg, **opts: None" (= avoid printing messages always) into "_lfstatuswriters" while transplanting makes explicit passing "printmessage = False" for "updatelfiles()" useless. This patch also removes setting/unsetting "repo._istransplanting" in "overridetransplant", because there is no code path referring it.

largefiles: update standins only at the 1st commit of "transplant --continue" Before this patch, "hg transplant --continue" may record incorrect standins, because largefiles extension always avoid updating standins while transplanting, even though largefiles in the working directory may be modified manually at the 1st commit of "hg transplant --continue". But, on the other hand, updating standins should be avoided at subsequent commits for efficiency reason. To update standins only at the 1st commit of "hg transplant --continue", this patch uses "automatedcommithook", which updates standins by "lfutil.updatestandinsbymatch()" only at the 1st commit of resuming. Even after this patch, "repo._istransplanting = True" is still needed to avoid some status report while updating largefiles in "lfcommands.updatelfiles()". This is reason why this patch omits not "repo._istransplanting = True" in "overriderebase" but examination of "getattr(repo, "_istransplanting", False)" in "updatestandinsbymatch".

largefiles: avoid redundant "updatelfiles" invocation in "overridetransplant" At "hg transplant --merge REV", largefiles newly coming from the 2nd parent (= REV) are marked as "a"(dded) by "patch.patch()", and have to be marked as "n"(ormal) after commit. But until changeset 3100d1cbce32, such largefiles were still marked as "a" unexpectedly even after commit, because no additional entry is added to filelog of such largefiles and they aren't listed in "repo[newnode].files()" in this case: "newnode" is one of newly committed changeset (= result of "repo.commit()"). "updatelfiles" invocation in "overridetransplant" shadows this problem by forcibly synchronizing lfdirstate to dirstate. Now, "updatelfiles" invocation in "overridetransplant" is redundant, because changeset 3100d1cbce32 made "markcommitted" use "ctx.files()" to get targets of "synclfdirstate" instead of "repo[newnode].files()".

merge with stable

util.system: remove unused handling of onerr=ui In our code, onerr is None or util.Abort. It smells bad to overload ui and exception class.

util.system: use ui.system() in place of optional ui.fout parameter

ui: introduce util.system() wrapper to make sure ui.fout is used This change is intended to avoid future problem of data corruption under command server. out=ui.fout is mandatory as long as command server uses stdout as IPC channel.

hook: remove redundant code to redirect http hook output to client stream out=ui and out=ui.fout should be the same here. ui.fout was introduced at afccc64eea73, which was not available when out=ui was added at c37f35d7f2f5.

hgk: forward command output to ui.fout consistently Nobody would want to run hgk in command server, but it should work in principle. This fixes possible data corruption of command-server channel.

merge with stable

Added signature for changeset 643c58303fb0

Added tag 3.2.1 for changeset 643c58303fb0

run-tests: use a try/except ladder instead of looking for a specific version This ensures we get json instead of simplejson in as many places as possible.

hghave: use a less brittle have-json check

sortdict: add insert method Future patches will allow extensions to choose which order a namespace should output in the log, so we add a way for sortdict to insert to a specific location.