changegroup: store source and url in the `hookargs` dict We store the source and url of the current data into `transaction.hookargs` this let us inherit it from upper layers that may have created a much wider transaction. We have to modify bundle2 at the same time to register the source and url in the transaction. We have to do it in the same patch otherwise, the `addchangegroup` call would fill these values and the hook calling will crash because of the duplicated 'source' and 'url' arguments passed to the hook call.

prechangegroup: use hook argument from the transaction There can be useful data in there (eg: bundle2 related one)

addchangegroup: call `prechangegroup` hook after transaction retrieval We want to reused some possible information stored in the transaction `hookargs` dict that may be stored by something handling the transaction at an upper level (eg: bundle2) So we move the running of the hooks after transaction creation. This has no visible effects (but an empty transaction roolback if the hook fails) because nothing had happened in the transaction yet.

addchangegroup: get the `node` argument of `incoming` hook from transaction The transaction is now carrying hook-related informations. So we use it to retrieve the `node` argument. This will also carry around all kinds of other useful informations (like: "are we in a bundle2 processing")

dicthelpers: delete now that they are no longer used

manifest: transpose pair of pairs from diff() It makes more sense for the file nodeids and returned from diff() to be ((n1,fl1),(n2,fl2)) than ((n1,n2),(fl1,fl2)), so change it to the former.

manifest: for diff(), only iterate over files, not flags From manifest.diff(), we return a dict from filename to pairs of pairs of file nodeids and flags (values of the form ((n1,n2),(fl1,fl2))). To create this dict, we currently generate one dict for files (with (n1,n2) values) and one for flags (with (fl1,fl2) values) and then join these dicts. Missing files are represented by None and missing flags by '', but due to the dict joining, the inner pairs themselves can also be None. The only caller, merge.manifestmerge(), then unpacks these values while checking for None values. By inlining the calls to dicthelpers and simplifying it to only iterate over files (ignoring flags-only differences), we can simplify life for our caller.

manifest: repurpose flagsdiff() into (node-and-flag)diff() The manifestdict class already has a method for diff flags between two manifests (presumably because there is no full access to the private _flags field). The only caller is merge.manifestmerge(), which also wants a diff of files between the same manifests. Let's combine the code for diffing files and flags into a single method on manifestdict. This puts all the manifest diffing in one place and will allow for further simplification. It might also be useful for it to be encapsulated in manifestdict if we later decide to to shard manifests. The docstring is intentionally unclear about missing entries for now.

util: add a file handle wrapper class that does hash digest validation It is going to be used for the remote-changegroup feature in bundle2.

util: add a helper class to compute digests It is going to be used for the remote-changegroup feature in bundle2.

bundle2: merge return values when bundle contains multiple changegroups A bundle2 may contain multiple parts adding changegroups, in which case there are multiple operation records for changegroups, each with its own return value. Those multiple return values are aggregated in a single cgresult value for the whole operation. As can be seen in the associated test case, the situation with hooks is not really the best, but without deeper thoughts and changes, we can't do much better. Hopefully, things will be improved before bundle2 is enabled by default. In the meanwhile, multiple changegroups is not expected to be in widespread use, and even less expected to be used for pushes. Also, not many clients cloning or pulling bundle2 with multiple changesets are not expected to have changegroup hooks anyways.

changegroup: use a copy of hookargs when invoking the changegroup hook addchangegroup creates a runhook function that is used to invoke the changegroup and incoming hooks, but at the time the function is called, the contents of hookargs associated with the transaction may have been modified externally. For instance, bundle2 code affects it with obsolescence markers and bookmarks info. It also creates problems when a single transaction is used with multiple changegroups added (as per an upcoming change), whereby the contents of hookargs are that of after adding a latter changegroup when invoking the hook for the first changegroup.

tests: pull common http server setup out of individual tests There are currently two different tests using roughly the same code to create temporary scripts acting as HTTP servers. As there is going to be at least one more in an upcoming change, factor those out in a standalone dumbhttp.py script.

util: move md5 back next to sha1 and allow to call it without an argument This effectively backs out changeset 908c5906091b. The API change is done so that both util.sha1 and util.md5 can be called the same way. The function is moved in order to use it for md5 checksumming for an upcoming bundle2 feature.

test-bundle2-exchange: do not drop HG_NODE from the hook output HG_NODE is precisely something we would like to test.

obsolete: update test-commit-amend to use obsolete option This test actually used the obs.py file as part of the test, so we need to fix up the test a little more than usual to work with the new obsolete option flags.

obsolete: update tests to use obsolete options The obsolete._enabled flag has become a config option. This updates all but one of the tests to use the minimal number of flags necessary for them to pass. For most tests this is just 'createmarkers', for a couple tests it's 'allowunstable', and for even fewer it's 'exchange'.

obsolete: prevent options from being used without createmarkers exchange and allowunstable should only be enabled if createmarkers is enabled, so check for that and raise an exception if that's not the case.

obsolete: add exchange option This adds an option that enables obsolete marker exchange.

obsolete: add allowunstable option This option allows the creation of unstable commits. This allows things like amending in the middle of a stack of commits, etc.

obsolete: add createmarkers option The basic obsolete option is allowing the creation of obsolete markers. This does not enable other features, such as allowing unstable commits or exchanging obsolete markers.

obsolete: add readonly flag to obstore constructor Previously, obstore read the obsolete._enabled flag to determine whether to allow writes to the obstore. Since obsolete._enabled will be moving into a repo specific config, we can't read it globally, and therefore must pass the information into the constructor.

obsolete: add isenabled function for option checking Previously, obsolete used the module level _enabled flag to determine whether it was on or off. We need a bit more granular control, so we'll be introducing toggle options. The isenabled() function is how you check if a particular option is enabled for the given repository. Future patches will add options such as 'createmarkers', 'allowunstable', and 'exchange' to enable various features of obsolete markers.

obsstore: fix defaultformat option passing The obsstore format passing was not actually being passed to the obsstore. This fixes it.

tests: use $PYTHON instead of hardcoding python This makes running the testsuite with pypy possible.

checklink: always close the NamedTemporaryFile This fixes test-patchbomb.t when using pypy with --pure.

hgignore: ignore the PyCharm workspace folder

revset: better naming of variables containing the value of a single argument Calling them args is not helpful.

manifest: add docstring to text() method

manifest: rename ambiguously-named set to setflag Just makes it a little clearer what this method does.

bookmarks: inform transaction-related hooks that some bookmarks were moved We do not have enough information to provide any finer data, but this is still useful information.

phases: inform transaction-related hooks that a phase was moved We do not have enough information to provide finer data, but this is still useful information.

test-bundle2: also test the argument of the changegroup hook We also track execution of the changegroup hook. The important information here is to make sure the information that the transaction was processing a bundle2 is passed to hook. This will let most hooks disable themselves while waiting for the hook concluding bundle2 processing (the one we discovered to be not called for pull in the previous changesets).

test-bundle2: test that we got appropriate hook called with appropriate data We can notice that this transaction wide hook is only happening during push and it is missing changegroup-related information. We'll want to fix this but this is not what this patch is about.

pull: use `stepsdone` instead of `todosteps` The push process uses a `stepsdone` attribute instead of a `todosteps` one (with the logic swapped). We unify the two process by picking the `stepsdone` version. I feel like `stepsdone` better fits extensions that would want to extend the push exchange process.

pull: make discovery phase extensible We apply the same approach as for push and make the discovery extensible. There is only one user in core right now, but we already know we'll need something smarter for obsmarkers. In fact the evolve extension could use this to cleanly extend discovery. The main motivation for this change is consistency between push and pull.

sshpeer: forward stdout of remote "hg init" to appropriate output channel Otherwise, commandserver channel could be corrupted.

revlog: support importing censored file revision tombstones This change allows a revision log to not fail integrity checks when applying a changegroup delta (eg from a bundle) results in a censored file tombstone. The tombstone is inserted as-is, so future integrity verification will observe the tombstone. Deltas based on the tombstone will also remain correct. The new code path is encountered for *exactly* the cases where _addrevision is importing a tombstone from a changegroup. When committing a file containing the "magic" tombstone text, the "text" parameter will be non-empty and the checkhash call is not executed (and when committing, the node will be computed to match the "magic" tombstone text).

verify: report censored nodes if configured policy is abort

context: handle censored data in an on-disk file context based on config Two possible behaviors are defined for handling censored data: abort, and ignore. When we ignore censored data we return an empty file to callers requesting the file data.

manifest: add fastdelta method to manifestdict This is another step closer to alternate manifest implementations that can offer different hashing algorithms.

manifest: move _search to module level and rename to _msearch The rename is intended to provide a slight hint that it is manifest-specific.

manifest: move manifestdict-to-text encoding to manifest class A future patch will introduce a new format, with a new class.

localrepo: access status fields by name rather than index

subrepo: use separate instances of empty lists in status We do modify the lists that make up the status in several places, so it seems risky to use the same instance of a list for several different status types. Use a separate empty list for each type instead.

summary: make status code more readable In commands.summary(), we currently zip a list of labels with a list of statuses. This means the order of the status list has to match the list of the labels, which in turn means the status elements have to be inserted into specific places in the list. Let's instead group the labels and status data we want to display in a single list of pairs.

strip: make checklocalchanges() return full status tuple By making checklocalchanges() return the full instance of the status class instead of just the first 4 elements of it, we can take advantage of the field names and not require the caller to remember the element indices.

fileset: access status fields by name rather than index

histedit: access status fields by name rather than index

shelve: access status fields by name rather than index

record: access status fields by name rather than index It is safe to pass the full status to patch.diff() since it does its own slicing.

purge: access status fields by name rather than index

largefiles: access status fields by name rather than index

keyword: access status fields by name rather than index

hgcia: access status fields by name rather than index

context: store status class instead of plain tuple in self._status This improves readability a bit by allowing us to refer to statuses by name rather than index.

status: update and move documentation of status types to status class The various status types are currently documented on the dirstate.status() method. Now that we have a class for the status types, it makese sense to document the status types there instead. Only leave the bits related to lookup/unsure in the status() method documentation.

status: update various other methods to return new class

status: create class for status lists Callers of various status() methods (on dirstate, context, repo) get a tuple of 7 elements, where each element is a list of files. This results in lots of uses of indexes where names would be much more readable. For example, "status.ignored" seems clearer than "status[4]" [1]. So, let's introduce a simple named tuple containing the 7 status fields: modified, added, removed, deleted, unknown, ignored, clean. This patch introduces the class and updates the status methods to return instances of it. Later patches will update the callers. [1] Did you even notice that it should have been "status[5]"? (tweaked by mpm to introduce the class in scmutil and only change one user)

lfutil: avoid creating unnecessary copy of status tuple In lfdirstatestatus(), the status tuple gets deconstructed, the lists get updated, and then an identical status tuple gets created and returned. Change it so we simply return the original tuple.