synthrepo: when adding files, ensure new path is not a directory

synthrepo: synthesized dates must be positive, fit in 32-bit signed ints

discovery: test coverage for issue4438 / 86c35b7ae300 / 73cfaa348650 The randomness in the discovery protocol made this problem hard to reproduce. The test mocks random.sample to make sure we hit the problem every time. The set iteration order also made the output unstable ... but with the issue fixed, it is stable.

hgweb: fix a crash when using web.archivesubrepos A matcher is required when enabling the subrepo option on archival.archive(), because that calls match.narrowmatcher(), which accesses fields on the object. It's therefore probably a bad idea to default the matcher to None on archive(), but that's a fix for default.

tests: introduce a subrepository to test-archive.t This will be used in an upcoming patch to add coverage for web.archivesubrepos.

test-status-rev: add tests for plain dirstate and inter-revision status We have tests for the status across from '.^' to the working copy. It makes sense to have the similar tests for the inter-revision status between '.^' and '.' and for the dirstate status in the same place.

test-status-rev: remove unnecessary initial commit The initial commit was there when we had a group of tests that compared against an empty base, but since those tests no longer exist, we can drop the empty commit.

test-status-rev: use one glob for each expected status It's getting a little hard to read the ~30 calls to 'hg status' with one per file. Instead, let's use one glob for each expected status. For example, modified files can be listed with 'glob:content1_*_content[23]-tracked'. That also nicely becomes an explanation for why each status is expected.

test-status-rev: remove duplicate tests The second group of tests in test-status-rev compare to an empty revision. The first group of tests that compare to the first commit should be testing all the same states with the missing_* files, so drop the second group of tests.

changegroup: sparsely populate fnodes Previously, fnodes had a key and empty dict value for every element in changedfiles. This is somewhat wasteful. Empty dicts in CPython consume a lot more memory than you would expect - 280 bytes. On mozilla-central, which has ~190,000 files/fnodes keys, the previous loop populating fnodes allocated 91,924 KB of memory, most of that for the empty dicts. With this patch in place, our peak RSS during mozilla-central clone drops: before: 364,356 KB after: 326,008 KB delta: -38,348 KB When combined with the previous patch, total peak RSS decrease is now 190,116 KB.

changegroup: don't store unused value on fnodes (issue4443) The contents of fnodes are only accessed once per key. It is wasteful to cache the value since nobody will use it. Before this patch, the caching of unused data in fnodes was effectively causing a memory leak during the file streaming part of bundle creation. On mozilla-central (which has ~190,000 entries in fnodes), this patch has a significant impact on RSS at the end of generate(): before: 516,124 KB after: 364,356 KB delta: -151,768 KB The origin of this code can be traced back to 627cd7842e5d and has been with us since the 2.7 release.

changegroup: don't define lookupmf() until it is needed lookupmf() is currently defined earlier than when it is needed. Future patches further refactoring this code will be easier to read when lookupmf() is in its new home.

mail: actually use the verifycert config value The mail module only verifies the smtp ssl certificate if 'verifycert' is enabled (the default). The 'verifycert' can take three possible values: - 'strict' - 'loose' - any "False" value, eg: 'false' or '0' We tested the validity of the third value, but never converted it to actual falseness, making 'False' an equivalent for 'loose'. This changeset fixes it.

exchange: use the postclose API on transaction As with changegroup, we should wait for the transaction to be really closed before scheduling hook execution.

changegroup: use the 'postclose' API on transaction The post-transaction hooks run after the lock release (because hooks may want to touch the repository), but they must only run if the transaction is successfully closed. We use the new 'addpostclose' method on transaction to register a callback installing this post-lock-release call.

transaction: allow registering a post-close callback The addchangegroup code considers the transaction done after a 'tr.close()' call and schedules the hook's execution for after lock release. In the nested transaction case, the transaction is not yet committed and we must delay this scheduling. We add an 'addpostclose' method (like the 'addpending' and 'addfinalize' ones) that registers code to be run if the transaction is successfully committed.

exchange: swap "push" for "pull" in pulloperation docstring

exchange: prepare kwargs for bundle2 part generation exactly once

exchange: fix indentation in _pullchangeset

dagutil: fix id/ix typos in docstrings

patchbomb: extract 'getpatchmsgs' closure into its own function Keep marching toward the promised land of simplification!

patchbomb: extract 'makeintro' closure into its own function Keep marching toward the promised land of simplification!

patchbomb: extract 'getbundlemsgs' closure in its own function Keep marching toward the promised land of simplification!

patchbomb: extract 'getdescription' closure in its own function Keep marching toward the promised land of simplification!

patchbomb: extract 'getbundle' closure in its own function Keep marching toward the promised land of simplification!

patchbomb: extract 'getpatches' closure in its own function Keep marching toward the promised land of simplification!

patchbomb: extract 'getoutgoing' closure into its own function The patchbomb command is a gigantic 300 line function full of closures. As a first step to simplify it in smaller bits, I'm extracting the closures into full featured functions. The first victim is 'getoutgoing'. It gains a docstring in the process.

bundle2: handle empty 'b2x:changegroup' value in push and pull Changeset e4dc2b0be056 added advertising of supported changegroup version through the new 'b2x:changegroup' capability. However, this capability is not new and has been around since 3.1 with an empty value. This makes new clients unable to push to 3.2 servers through bundle2 as they cannot find a common changegroup version to use from and empty list. Treating empty 'b2x:changegroup' value as old client fixes it.

bundle2: drop duplicated definition of 'b2x:exchange' This bundle2 capability is going to be dynamically computed in 'getrepocaps'. We do not need to include it in the static value.

convert: use git diff-tree -Cn% instead of --find-copies=n% for older git The option --find-copies was added in a later git version than the one included in Debian squeeze-lts (1.7.2.5), probably around 1.7.4.

changelog: rely on transaction for finalization Instead of calling 'cl.finalize()' by hand (possibly at a bogus time) we register it in the transaction during 'delayupdate' and rely on 'tr.close()' to call it at the right time.

transaction: allow registering a finalization callback The new 'addfinalize' method allows people to register a callback to be triggered when the transaction is closed. This aims to get rid of explicit calls to 'changelog.finalize'. This also obsoletes the 'onclose' function but removing it is not in the scope of this series.

changelog: handle writepending in the transaction The 'delayupdate' method now takes a transaction object and registers its '_writepending' method for execution in 'transaction.writepending()'. The hook can then use 'transaction.writepending()' directly. At some point this will allow the addition of other file creation during writepending.

transaction: add 'writepending' logic The contents of the transaction must be flushed to disk before running a hook. But it must be flushed to a special file so that the normal reader does not use it. This logic is currently in the changelog only. We add some facility to register such operations in the transaction itself.

changelog: rework the delayupdate mechanism The current way we use the 'delayupdate' mechanism is wrong. We call 'delayupdate' right after the transaction retrieval, then we call 'finalize' right before calling 'tr.close()'. The 'finalize' call will -always- result in a flush to disk, making the data available to all readers. But the 'tr.close()' may be a no-op if the transaction is nested. This would result in data: 1) exposed to reader too early, 2) rolled back by other part of the transaction after such exposure So we need to end up in a situation where we call 'finalize' a single time when the transaction actually closes. For this purpose we need to be able to call 'delayupdate' and '_writepending' multiple times and 'finalize' once. This was not possible with the previous state of the code. This changeset refactors the code to makes this possible. We buffer data in memory as much as possible and fall-back to writing to a ".a" file after the first call to '_writepending'.

merge with stable

bookmarks: fix formatting of exchange message (issue4439) The message formatting was crashing when doing explicit pulling `hg pull -B X`. This changeset fix it and improved the test coverage.

test-status-rev: document one more broken test The status for missing_content2_content2-untracked doesn't get reported at all. Since the file does exist in the working copy, it should reported as unknown. Document that in the test.

test-status-rev: use common script for generating file history Start using the generate-working-copy-states.py script that's shared with test-revert.t, instead of creating the states manually in the test. This adds several states that are currently missing. We will start checking those states later.

test-status-rev: use same names as from generate-working-copy-states To prepare for using generate-working-copy-states.py for generating the files and their content, let's start by renaming the files according to the naming scheme used by that script.

test-revert: move embedded script to its own file Move the gen-revert-cases.py out of test-revert.t into its own file so we can reuse it from other tests (specifically test-status-rev.t).

test-revert: simplify generation of files With the recent change in naming of the generated files, it becomes much easier to generate the files by iterating over all the possible states than over the state transitions.

merge with stable

discovery: indices between sample and yesno must match (issue4438) 3ef893520a85 changed 'sample' from a list to a set. The iteration order is thus undefined and the yesno indices are not stable. To solve this, repeat the listification and comment from elsewhere in the code. Note: the randomness in the discovery protocol can make this problem hard to reproduce.

discovery: limit 'all local heads known remotely' to real 'all' (issue4438) 3ef893520a85 made it possible that the initial head check didn't include all heads. If that is the case, don't use the early exit just because this random sample happened to be 'all known'. Note: the randomness in the discovery protocol can make this problem hard to reproduce.

largefiles: avoid printing messages while rebasing by "_lfstatuswriters" Putting "lambda *msg, **opts: None" (= avoid printing messages always) into "_lfstatuswriters" while rebasing makes explicit passing "printmessage = False" for "updatelfiles()" useless. This patch also removes setting/unsetting "repo._isrebasing" in "overriderebase", because there is no code path referring it.

largefiles: get function to write status messages via "getstatuswriter()" This patch makes "updatelfiles()" get appropriate function to write largefiles specific status messages via "getstatuswriter()". This patch introduces None as "print messages if needed", because True (forcibly writing) and False (forcibly ignoring) are already used for "printmessage" of "updatelfiles". Subsequent patch will move "avoid printing messages only while automated committing" decision from caller of "updatelfiles()" into "getstatuswriter()".

largefiles: introduce "_lfstatuswriters" to customize status reporting "lfutil.getstatuswriter" is the utility to get appropriate function to write largefiles specific status out from "repo._lfstatuswriters". This patch uses "stack" with an element instead of flag like "_isXXXXing" or so, because: - the former works correctly even when customizations are nested, and - ensuring at least one element can ignore empty check