transaction: handle missing file in backupentries (instead of using entries) The case where a backup of a missing file was requested was previously handled by the 'entries' list. As the 'backupentries' is about to gain ability to backup files outside of '.hg/store', we want it to be able to handle the missing file too. Reminder: using 'addbackup' on a missing file means that such file needs to be deleted if we rollback the transaction.

test-hup: use ls instead of echo Having one file entry per line makes it easier to analyse diffs.

largefiles: move "copyalltostore" invocation into "markcommitted" Before this patch, while "hg convert", largefiles avoids copying largefiles in the working directory into the store area by combination of setting "repo._isconverting" in "mercurialsink{before|after}" and checking it in "copytostoreabsolute". This avoiding is needed while "hg convert", because converting doesn't update largefiles in the working directory. But this implementation is not efficient, because: - invocation in "markcommitted" can easily ensure updating largefiles in the working directory "markcommitted" is invoked only when new revision is committed via "commit" of "localrepository" (= with files in the working directory). On the other hand, "commitctx" may be invoked directly for in-memory committing. - committing without updating the working directory (e.g. "import --bypass") also needs this kind of avoiding For efficiency of this kind of avoiding, this patch does: - move "copyalltostore" invocation into "markcommitted" - remove meaningless procedures below: - hooking "mercurialsink{before|after}" to (un)set "repo._isconverting" - checking "repo._isconverting" in "copytostoreabsolute" This patch invokes "copyalltostore" also in "_commitcontext", because "_commitcontext" expects that largefiles in the working directory are copied into store area after "commitctx". In this case, the working directory is used as a kind of temporary area to write largefiles out, even though converted revisions are committed via "commitctx" (without updating normal files).

largefiles: avoid printing messages while transplanting by "_lfstatuswriters" Putting "lambda *msg, **opts: None" (= avoid printing messages always) into "_lfstatuswriters" while transplanting makes explicit passing "printmessage = False" for "updatelfiles()" useless. This patch also removes setting/unsetting "repo._istransplanting" in "overridetransplant", because there is no code path referring it.

largefiles: update standins only at the 1st commit of "transplant --continue" Before this patch, "hg transplant --continue" may record incorrect standins, because largefiles extension always avoid updating standins while transplanting, even though largefiles in the working directory may be modified manually at the 1st commit of "hg transplant --continue". But, on the other hand, updating standins should be avoided at subsequent commits for efficiency reason. To update standins only at the 1st commit of "hg transplant --continue", this patch uses "automatedcommithook", which updates standins by "lfutil.updatestandinsbymatch()" only at the 1st commit of resuming. Even after this patch, "repo._istransplanting = True" is still needed to avoid some status report while updating largefiles in "lfcommands.updatelfiles()". This is reason why this patch omits not "repo._istransplanting = True" in "overriderebase" but examination of "getattr(repo, "_istransplanting", False)" in "updatestandinsbymatch".

largefiles: avoid redundant "updatelfiles" invocation in "overridetransplant" At "hg transplant --merge REV", largefiles newly coming from the 2nd parent (= REV) are marked as "a"(dded) by "patch.patch()", and have to be marked as "n"(ormal) after commit. But until changeset 3100d1cbce32, such largefiles were still marked as "a" unexpectedly even after commit, because no additional entry is added to filelog of such largefiles and they aren't listed in "repo[newnode].files()" in this case: "newnode" is one of newly committed changeset (= result of "repo.commit()"). "updatelfiles" invocation in "overridetransplant" shadows this problem by forcibly synchronizing lfdirstate to dirstate. Now, "updatelfiles" invocation in "overridetransplant" is redundant, because changeset 3100d1cbce32 made "markcommitted" use "ctx.files()" to get targets of "synclfdirstate" instead of "repo[newnode].files()".

merge with stable

util.system: remove unused handling of onerr=ui In our code, onerr is None or util.Abort. It smells bad to overload ui and exception class.

util.system: use ui.system() in place of optional ui.fout parameter

ui: introduce util.system() wrapper to make sure ui.fout is used This change is intended to avoid future problem of data corruption under command server. out=ui.fout is mandatory as long as command server uses stdout as IPC channel.

hook: remove redundant code to redirect http hook output to client stream out=ui and out=ui.fout should be the same here. ui.fout was introduced at afccc64eea73, which was not available when out=ui was added at c37f35d7f2f5.

hgk: forward command output to ui.fout consistently Nobody would want to run hgk in command server, but it should work in principle. This fixes possible data corruption of command-server channel.

merge with stable

Added signature for changeset 643c58303fb0

Added tag 3.2.1 for changeset 643c58303fb0

run-tests: use a try/except ladder instead of looking for a specific version This ensures we get json instead of simplejson in as many places as possible.

hghave: use a less brittle have-json check

sortdict: add insert method Future patches will allow extensions to choose which order a namespace should output in the log, so we add a way for sortdict to insert to a specific location.

sortdict: add iteritems method Future patches will start using sortdict for log operations where order is important. Adding iteritems removes the headache of having to remember to use items() if the object is a sortdict.

addremove: add back forgotten files (BC) After running "hg forget README && hg addremove", README will still be reported as removed, while "hg forget README && hg add README" adds it back so it gets reported as clean. It seems like they should behave the same. Furthermore, it seems like no files should remain untracked after 'hg addremove && hg commit' (or 'hg commit -A'). For these reasons, change the behavior of addremove so it does add forgotten files back. The problem is with scmutil._interestingfiles(), which reports the file as removed, so scmutil.addremove() does not add it. Fix by teaching _interestingfiles() to report forgotten files separately from removed files and make addremove() add forgotten files back. However, do not treat forgotten files as sources for rename detection. Note that since removed and forgotten files are treated the same before this change, forgotten files were considered sources for rename detection. Also update the other caller, marktouched(), in the same way as addremove().

add: add back forgotten files even when not matching exactly (BC) I accidentally did 'hg forget .' and tried to undo the operation with 'hg add .'. I expected the files to be reported as either modified or clean, but they were still reported as removed. It turns out that forgotten files are only added back if they are listed explicitly, as shown by the following two invocations. This makes it hard to recover from the mistake of forgetting a lot of files. $ hg forget README && hg add README && hg status -A README C README $ hg forget README && hg add . && hg status -A README R README The problem lies in cmdutil.add(). That method checks that the file isn't already tracked before adding it, but it does so by checking the dirstate, which does have an entry for forgotten files (state 'r'). We should instead be checking whether the file exists in the workingctx. The workingctx is also what we later call add() on, and that method takes care of transforming the add() into a normallookup() on the dirstate. Since we're changing repo.dirstate into wctx, let's also change repo.walk into wctx.walk for consistency (repo.walk calls wctx.walk, so we're simply inlining the call).

context.status: explain "caching reasons" more fully Where we "load earliest manifest first for caching reasons", elaborate on what "caching reasons" refers to. Text provided by Matt in http://thread.gmane.org/gmane.comp.version-control.mercurial.devel/73235/focus=73578.

localrepo: rename revlog.maxchainlen to format.maxchainlen This is more consistent with other option names, as spotted by Pierre-Yves. Thanks!

revlog: add config variable for limiting delta-chain length The current heuristic for deciding between storing delta and full texts is based on ratio of (sizeofdeltas)/(sizeoffulltext). In some cases (for example a manifest for ahuge repo) this approach can result in extremely long delta chains (~30,000) which are very slow to read. (In the case of a manifest ~500ms are added to every hg command because of that). This commit introduces "revlog.maxchainlength" configuration variable that will limit delta chain length.

debugrevlog: fix computing chain length in debugrevlog -d The chain length was computed correctly only when generaldelta feature was enabled. Now it's fixed. When generaldelta is disabled the base revision in revlog index is not the revision we have delta against - it's always previous revision. Instead of incorrect chainbaseandlen in command.py we are now using two single-responsibility functions in revlog.py: - chainbase(rev) - chainlen(rev) Only chainlen(rev) was missing so it was written to mimic the way the chain of deltas is actually found during file reconstruction.

transaction: factorise append-only file registration The addition is done in two different places but differs slightly. We factorise this addition to ensure it is consistent in all places.

transaction: document `tr.add`

transaction: drop backupentries logic from startgroup and endgroup The `startgroup` and `endgroup` methods are used in a very specific context to wrap a very specific operation (revlog truncation). It does not make sense to perform any other operations during such a "group" (eg:file backup). There is currently no user of backupfile during a "group" so we drop the group-specific code and restrict authorized operations during "group".

transaction: document startgroup and endgroup These enigmatic methods are only used in repair. We document them to clarify there purpose and user.

transaction: mark backup-related attributes private As the transaction is gaining more functions and attributes, it is important to clarify what is part of the public API.

transaction: document the contents of `tr.backupentries` Now that all items are known we can document it.

transaction: drop the third item in `tr.backupentries` This third item is always None and never used.

rebase: fix rebase with no common ancestors (issue4446) The new rebase revset didn't check for the case when there are no common ancestors. Now it does. The new behavior should be the same as the pre-3.2 behavior. Added a test.

test-run-tests: accept more levels of precision and trailing ws (issue4440) simplejson produces slightly different output from the built-in json module, specifically: * It uses 0.000 instead of 0.0000 * It likes to put a trailing space after a comma This change works around both of those variations.

merge with stable

changegroup.cg2packer: lookup 'group' via inheritance chain This lets extensions insert themselves in the class hierarchy.

context.status: only filter suspect symlinks in the dirstate status We don't care about filtering out symlinks that have already been committed with full content, only those that have been accidentally resolved in the working directory.

context.status: inline _poststatus() By inlining _poststatus() into _buildstatus(), it becomes clearer that it is only called for the workingctx.

context.status: remove now-empty _prestatus()

context.status: call _dirstatestatus() from within _buildstatus() By making the call to _dirstatestatus() within _buildstatus(), it becomes clearer that it's called only for the workingctx.

context.status: move manifest caching trick to _buildstatus() In basectx._buildstatus(), we read the manifests for the two revisions being compared. For "caching reasons" unknown to me, it is better to read the earlier manifest first, which basectx._prestatus() takes care of. However, if the 'self' context is a committablectx and the 'other' context is the parent of the working directory (as in the very common case of plain "hg status"), there is no need to read any manifests at all -- all that's needed is the dirstate status. To avoid reading the manifests, _prestatus() is overridden in committablectx and avoids calling its super method, and _buildstatus() calls its super method only if the 'other' context is not the parent of the working directory. It seems easier to follow what's happening if we move the pre-fetching to _buildstatus() just before the place where the manifests are fetched. We just need to add an extra check that the revision is not None to handle the case that was previously handled by subclass overriding. That also makes it safe for committablectx._prestatus() to call its parent, although the latter now becomes empty, so we won't bother.

context.status: remove unused arguments from _matchstatus()

context.status: remove overriding in workingctx The workingctx method simply calls the super method. The only effect it has is that it uses a different default argument for the 'other' argument. The only in-tree caller is patch.diff, which always passes an argument to the method, so it should be safe to remove the overriding. Having the default argument depend on the type seems rather dangerous anyway.

synthrepo: when adding files, ensure new path is not a directory

synthrepo: synthesized dates must be positive, fit in 32-bit signed ints

discovery: test coverage for issue4438 / 86c35b7ae300 / 73cfaa348650 The randomness in the discovery protocol made this problem hard to reproduce. The test mocks random.sample to make sure we hit the problem every time. The set iteration order also made the output unstable ... but with the issue fixed, it is stable.

hgweb: fix a crash when using web.archivesubrepos A matcher is required when enabling the subrepo option on archival.archive(), because that calls match.narrowmatcher(), which accesses fields on the object. It's therefore probably a bad idea to default the matcher to None on archive(), but that's a fix for default.

tests: introduce a subrepository to test-archive.t This will be used in an upcoming patch to add coverage for web.archivesubrepos.

test-status-rev: add tests for plain dirstate and inter-revision status We have tests for the status across from '.^' to the working copy. It makes sense to have the similar tests for the inter-revision status between '.^' and '.' and for the dirstate status in the same place.

test-status-rev: remove unnecessary initial commit The initial commit was there when we had a group of tests that compared against an empty base, but since those tests no longer exist, we can drop the empty commit.

test-status-rev: use one glob for each expected status It's getting a little hard to read the ~30 calls to 'hg status' with one per file. Instead, let's use one glob for each expected status. For example, modified files can be listed with 'glob:content1_*_content[23]-tracked'. That also nicely becomes an explanation for why each status is expected.

test-status-rev: remove duplicate tests The second group of tests in test-status-rev compare to an empty revision. The first group of tests that compare to the first commit should be testing all the same states with the missing_* files, so drop the second group of tests.

changegroup: sparsely populate fnodes Previously, fnodes had a key and empty dict value for every element in changedfiles. This is somewhat wasteful. Empty dicts in CPython consume a lot more memory than you would expect - 280 bytes. On mozilla-central, which has ~190,000 files/fnodes keys, the previous loop populating fnodes allocated 91,924 KB of memory, most of that for the empty dicts. With this patch in place, our peak RSS during mozilla-central clone drops: before: 364,356 KB after: 326,008 KB delta: -38,348 KB When combined with the previous patch, total peak RSS decrease is now 190,116 KB.

changegroup: don't store unused value on fnodes (issue4443) The contents of fnodes are only accessed once per key. It is wasteful to cache the value since nobody will use it. Before this patch, the caching of unused data in fnodes was effectively causing a memory leak during the file streaming part of bundle creation. On mozilla-central (which has ~190,000 entries in fnodes), this patch has a significant impact on RSS at the end of generate(): before: 516,124 KB after: 364,356 KB delta: -151,768 KB The origin of this code can be traced back to 627cd7842e5d and has been with us since the 2.7 release.

changegroup: don't define lookupmf() until it is needed lookupmf() is currently defined earlier than when it is needed. Future patches further refactoring this code will be easier to read when lookupmf() is in its new home.

mail: actually use the verifycert config value The mail module only verifies the smtp ssl certificate if 'verifycert' is enabled (the default). The 'verifycert' can take three possible values: - 'strict' - 'loose' - any "False" value, eg: 'false' or '0' We tested the validity of the third value, but never converted it to actual falseness, making 'False' an equivalent for 'loose'. This changeset fixes it.

exchange: use the postclose API on transaction As with changegroup, we should wait for the transaction to be really closed before scheduling hook execution.

changegroup: use the 'postclose' API on transaction The post-transaction hooks run after the lock release (because hooks may want to touch the repository), but they must only run if the transaction is successfully closed. We use the new 'addpostclose' method on transaction to register a callback installing this post-lock-release call.

transaction: allow registering a post-close callback The addchangegroup code considers the transaction done after a 'tr.close()' call and schedules the hook's execution for after lock release. In the nested transaction case, the transaction is not yet committed and we must delay this scheduling. We add an 'addpostclose' method (like the 'addpending' and 'addfinalize' ones) that registers code to be run if the transaction is successfully committed.

exchange: swap "push" for "pull" in pulloperation docstring