largefiles: restore standins from non branch-tip parent at rollback correctly Before this patch, "hg rollback" can't restore standins correclty, if: - old parent of the working directory is rollback-ed, and - new parent of the working directory is not branch-tip "overriderollback" uses "merge.update" as a kind of "revert" utility to restore only standins with "node=None", and this makes "merge.update" choose "branch-tip" revision as the updating target unexpectedly. Then, "merge.update" restores standins from the branch-tip revision regardless of the parent of the working directory after rollback and this may cause unexpected behavior. This patch invokes "merge.update" with "node='.'" to restore standins from the parent revision of the working directory. In fact, this "merge.update" invocation will be replaced in the subsequent patch to fix another problem, but this change is usefull to inform reason why such complicated case should be tested.

largefiles: omit restoring standins if working parent is not rollbacked For efficiency, this patch omits restoring standins and updating lfdirstate, if the parent of the working directory is not rollbacked. This patch adds the test not to confirm whether restoring is skipped or not, but to detect unexpected regression in the future: it is difficult to distinguish between skipping and perfectly restoring.

repoview: fix 0L with pack/unpack for 2.4

help: add pad function to template help Commit aa51392da507 introduced a pad function for use in templates, but did not add the corresponding documentation to 'hg help templating'.

histedit: use str.startswith to check for comments in action list

histedit: drop duplicate line extracting keep option

import: avoid editor invocation when importing with "--exact" for exact-ness Before this patch, external editor is invoked when imported patch has no commit message, even if "--exact" is specified. Then, exact-ness is broken, because empty commit message causes failure of committing. This patch avoids editor invocation at importing with "--exact" for exact-ness, because commit message in the patch should be kept as it is in such case, even if it is empty.

import: disallow meaningless combination of "--exact" and "--edit" Before this patch, "hg import" allows combination of "--exact" and "--edit", even though editing commit message breaks exact-ness. This patch disallows meaningless combination of "--exact" and "--edit".

config: highlight parse error caused by leading spaces (issue3214) Added "unexpected leading whitespace" message to parse error when .hgrc has a line that starts with whitespace. Helps new users unfamiliar with syntax of rc file.

test-config: add tests for invalid syntax This was not tested and there is more to come in the next patch.

debugobsolete: add a --rev argument This argument can be used to list markers relevant to a set of revisions. We add a test for this option and the relevant computation in the same move.

obsolete: rename `allmarkers` to `getmarkers` This function is going to gain a "nodes" argument to filter the markers to the ones relevant to <nodes> only.

debugobsolete: add a way to record parent information We add a ``--record-parents`` flag to debugobsolete. This can be used to record parent information in the marker when the precursors are known locally. This will be useful to test the "relevant markers" computation.

obsstore: add relevantmarkers method We add a ``relevantmarkers`` method to fetch all markers that seem relevant to a set of nodes. See function documentation about how this set is computed. This will let us exchange only the markers that seem "relevant" to the set of changesets related to a push or a pull. The approach used to define "relevant" has been successfully tested in evolve for 6 months.

obsstore: keep track of children information We use the new `parents` field to build a dictionary of markers that touch children of a node. This will be used to link prune markers to a set of exchanged nodes.

push: check if local and remote support evolution during discovery We can now directly prevent any evolution-related operation from happening by using an empty set for outgoing markers. So we detect if no transfers should occur and use an empty set in this case.

obsstore: drop outdated comment This comment was associated with a now-defunct line.

cvsps: add two more tiebreakers in cscmp test-convert-cvs.t has been a little flaky for a while now. Add an extra tiebreaker in cscmp so that all the cases in the test will sort reliably. Without this patch, test-convert-cvs.t failed after 346 runs. With this patch, I stopped trying to get it to fail after 615 runs. While not conclusive, that makes me pretty optimistic that this is a working fix.

annotate: abort early if no file is specified This change is intended to move the getdate function near the opmap table.

purge: avoid full walks when directories aren't purged If match.traversedir is not None, we're forced to do full walks. However when we aren't purging directories we don't need to set match.traversedir to anything. This speeds up non-full walks such as the one hgwatchman makes possible. For mozilla-central with hgwatchman enabled, 'hg purge --files' goes from 0.88 seconds to 0.22.

clone: for local clones, copy over filtered branchcaches as well (issue4286) Local clones copy/hardlink over files directly, so the branchcaches should all be valid. There's a slight chance that a read operation would update one of the branchcaches, but we hold a lock over the source repo so they shouldn't cause an invalid branchcache to be copied over, just a different, valid one.

clone: for local clones, copy branchcache from the right location (issue4286) The unfiltered branchcache is in .hg/cache/branch2, not .hg/store/cache/branch2.

obsolete: avoid 2-argument form of enumerate, which was new in Python 2.6

repoview: use util.sha1() instead of hashlib.sha1() 45b5cd948a4d accidentally broke Python 2.4 compatibility, this fixes it.

debugobsolete: display parents information from markers Now that we have a new field, we need a way to visualize it.

obsmarkers: add a `parentnodes` method to retrieve parent information

obsstore: also store the 'parents' field on disk We now store the `parents` field on disk. We use the same strategy as for `date`: We stick it into the metadata. This is slow and dirty, but this is also the only way we currently have. At some point we'll have a new obsstore format to store this properly.

obsstore: drop 'date' from the metadata dictionary We are extracting the `date` information from the metadata at read time. However, we failed to remove it from the metadata returned in the markers. This is now fixed.

createmarkers: automatically record the parent of pruned changesets We need this information to build the set of relevant markers during exchanges. This can only be done at the `createmarkers` level since the `obsstore.create` function does not have a repo and therefore has no access to the parent information.

obsstore: add a `parents` argument to obsstore.create We need a way to pass the information to the function. Some guru told me that what's arguments are made for.

obsstore: add a `parents` field This field is intended to store the parent of the precursor. This is useful to attach pruned changesets to a set of exchanged changesets. We currently just add the fields with a None value. None stands for "no data recorded".

obsstore: add some documentation about the marker fields As the number of fields grow, it makes sense to start documenting their contents.

transplant: change "editform" to distinguish merge commits from others "editform" argument for "getcommiteditor" is decided according to the format below: EXTENSION[.COMMAND][.ROUTE] - EXTENSION: name of extension - COMMAND: name of command, if there are two or more commands in EXTENSION - ROUTE: name of route, if there are two or more routes in COMMAND This patch newly adds "normal" and "merge" as ROUTE, to distinguish merge commits from other. This patch adds 4 test patterns to test combination of "merge"(x2) and "--continue"(x2).

rebase: change "editform" to distinguish merge commits from others "editform" argument for "getcommiteditor" is decided according to the format below: EXTENSION[.COMMAND][.ROUTE] - EXTENSION: name of extension - COMMAND: name of command, if there are two or more commands in EXTENSION - ROUTE: name of route, if there are two or more routes in COMMAND This patch newly adds "merge" as ROUTE, to distinguish merge commits from other. This patch passes bool as "ctxorbool" to "mergeeditform", because working context has always 2 parents at this point. Dropping the second parent of non-merging commits is executed in "concludenode". Unlike other patches in this series (e.g. for "hg commit"), this patch doesn't add "normal.normal"/"normal.merge" style ROUTEs, because there is no "merge" case in "collapse" ROUTE.

import: change "editform" to distinguish merge commits from others "editform" argument for "getcommiteditor" is decided according to the format below: COMMAND[.ROUTE] - COMMAND: name of command - ROUTE: name of route, if there are two or more routes in COMMAND This patch uses "normal.normal" and "normal.merge" as ROUTE of "editform" instead of "normal", to distinguish merge commits from other in "hg import" without "--bypass" case. This patch assumes "editform" variations for "hg import" below: import.normal.normal import.normal.merge import.bypass.normal import.bypass.merge Unlike other patches in this series, this patch uses "editor.sh" instead of "checkeditform.sh" for the name of the script to check "HGEDITFORM", because it has to do more than checking "HGEDITFORM". To invoke editor forcibly in "test-import-merge.t", this patch creates the patch not having patch description as "merge.nomsg.diff".

commit: change "editform" to distinguish merge commits from other (--amend) "editform" argument for "getcommiteditor" is decided according to the format below: COMMAND[.ROUTE] - COMMAND: name of command - ROUTE: name of route, if there are two or more routes in COMMAND This patch uses "amend.normal" and "amend.merge" as ROUTE of "editform" instead of "amend", to distinguish merge commits from other in "hg commit --amend" case.

commit: change "editform" to distinguish merge commits from others "editform" argument for "getcommiteditor" is decided according to the format below: COMMAND[.ROUTE] - COMMAND: name of command - ROUTE: name of route, if there are two or more routes in COMMAND This patch uses "normal.normal" and "normal.merge" as ROUTE of "editform" instead of "normal", to distinguish merge commits from others in "hg commit" without "--amend" case. This patch assumes "editform" variations for "hg commit" below: commit.normal.normal commit.normal.merge commit.amend.normal commit.amend.merge "mergeeditform" is factored out for subsequent patches. It takes "ctxorbool" argument, because context object can't be passed in some cases.

test-ssh: verify that stderr from remote is printed (issue4336) The issue fixed in the previous patch was uncovered by implementing an extension that printed additional output locally before the push command completed. This test emulates that. If this change is applied before the previous patch, the test will fail on Linux, with the local output being printed before the "remote: " lines.

posix: implement readpipe using non-blocking I/O (issue4336) On Linux, fstat().st_size of a pipe always returns 0, even if the pipe has data available for reading. This meant that reading from and subsequently printing the stderr pipe content after wireproto commands over SSH meant that available data wasn't being printed. We now implement pipe reading on POSIX by doing a non-blocking read for all available data.

platform: implement readpipe() Reading all available data from a pipe has a platform-dependent implementation. This patch establishes platform.readpipe() by copying the inline implementation in sshpeer.readerr(). The implementations for POSIX and Windows are currently identical. The POSIX implementation will be changed in a subsequent patch.

exchange: remove duplicated addition to pushop.stepdone Not sure how it got there but it is useless.

exchange: drop superfluous parens

push: add bookmarks to the unified bundle2 push We use the `pushkey` part to exchange bookmark updates within the unified bundle2 push. Note that this only applies on update (moving a bookmark known on both sides) since bookmark export (creation of a new bookmark on remote) is apparently done outside of the _push function.

test-bundle2: add bookmark movement to the push test If we add bookmarks to bundle2, we need a way to test the new code. Tests are changed beforehand to highlight that inclusion of bookmarks in bundle does not introduce any behavior changes.

push: use stepsdone to control bookmark push If bookmark are to be integrated in the unified bundle2, we need a way to disable the old-style push.

push: move bookmark discovery with other discovery steps The discovery of necessary bookmark updates is now done within the "discovery phase". This opens the door to the inclusion of bookmarks in a unified bundle2 push.

pushbookmark: split an ultra-long line into a saner version We make a temporary variable for the remote bookmark data and we do not expand all elements from `bookmark.compare` since we are going to use only one.

revert: drop dead code for untracked files Untracked files are handled through status now.

revert: handle unknown files through status This will allow us to drop the code dedicated to this special case.

revert: drop more dead code Now that we detect all clean files, we do not need this clause anymore.

revert: add a message to noop action This prepares for the arrival of a second "not touching file" action: revert of an untracked file.

revert: simplify loop conditional The two breaks can be joined into one. The code gains one level of indent.

revert: explode the action tuple in the for loop noop is about to gain a message.

revert: use actions[...] in all disptable cases 1. Special cases are not special enough 2. There are two cases where nothing is done and a message is displayed. This prepares it.

revset: factorize backup decision The conditional controlling the creation of backup is fairly big. We move config related decisions outside of the loop.

revert: use a flat dispatch table Now that the table is simpler, remove one level of depth from it. This simplifies its usage in the for loop.

pushbookmark: do not attempt to update bookmarks if the push failed (BC) Before this patch, there was always an attempt to update bookmark even if prior steps of the push failed. I cannot see a good semantic reason to do so. We disable this possibility to simplify the push flow with bundle2. Bookmarks will be included in the bundle and fail with other steps.

pushbookmark: remove a <cond> and <val> or <other> construct We make the conditional explicit for the sake of readability.

pushbookmark: stop unrolling ancestors Now that ancestors has the same boolean property as a list, we can stop unrolling the set of ancestors. This should provide a significant speedup to this step as ancestor objects are smart and lazy.

ancestors: add a __nonzero__ method This allows using the object in a conditional the same way we can use list.

push: update bookmarks within the remote lock Updating bookmarks is part of the push. It should be done within the same lock as the other steps of the push.

hgweb: refresh repository using URL not path (issue4323) hgweb detects out-of-date repository instances (using a highly suspect mechanism that should probably be fixed) and obtains a new repository object if needed. This patch changes the repository object copy to use the repo URL (instead of path). This preserves more information about the source repository and allows bundles to be served through hgweb. A test verifying that bundles can now be served properly via `hg serve` has been added.

obsmarker: add `date` as an explicit field The markers are now 5-item tuples (concluded by the date). The obsstore.fields contents have been updated accordingly. There is no change to the on-disk format yet, so the date has to be extracted every time we read binary markers and re-injected each team we write them. This introduces a slowdown that will be solved when a new version of the format is added. Such a slowdown was already introduced by the evolve extension anyway.

obsstore: add fields attribute to track each field in a marker We are going to increase the amount of data explicitly stored in obsolescence markers. This mean we are going to have a longer tuple and some values will be shuffled around. So we add a ``fields`` attribute to the obsstore class to keep track of what entry is what. This will be useful for extensions and for documentation purpose.

debugobsolete: explicitly display date in the output As the date is becoming a first-class citizen, we are displaying it in an explicit field. As a bonus it is now readable by humans.

obsolete: add a date argument to the `createmarkers` function The function is now just passing the value to create markers.

debugobsolete: use the new date argument on obsstore.create Now that we have this new argument, we can just use it.

obsstore: add an explicit `date` argument to obsstore.create The date will become an official field in the markers (and ultimately in the on-disk format). We start by making it an official argument for the function.

obsolete: explicitly pass metadata argument using keyword argument We are about to add more arguments to this function (date, parents, etc). Passing metadata as a keyword argument gives us more flexibility when adding them.

obsmarker: add a `flags` method We introduce a proper method to access the flag information.

test-treediscovery: update output after merge Recent change in the push process introduced an extra listing of the phase name space before the push (on default). Meanwhile on default, a fix introduced a new test with debug output. We update the test output to be correct.

revert: add an XXX about rename tracking We check for rename information in the dirstate. This is probably not enough to preserve this behavior when using an explicit target rev. I just spotted this while working on this code, but this is outside the scope of my series so I'm just adding a comment to highlight this suspicious situation.

revert: inline a now useless closure Now that a single call site remains, we can just inline its content.

revert: remove code killed by the double status All those checks were here to catch cases where files were not modified in dirstate but were different in the target revision. This is now properly handled by calling status on the target node too.

revert: drop `missingmodified` set There cannot be any elements in this set since: dsmodified &= modified

revert: detect files added during a merge In case of merge, file that are actually added can be reported as modified. This is currently handled by special-case code. We detect it beforehand instead. This will lets use remove the special-case code at some point in the future.

revert: simplify handling of `added` files There are multiple possible cases for added files. But it's all handled by magic much lower in the stack. We document them, simplify the codes and move on.

merge with stable

rebase: use "rebase.collapse" as "editform" for "--collapse" always Before this patch, if both "--message" and "--collapse" are specified for "hg rebase", "rebaes.normal" is used as "editform" unexpectedly. Unlike patches before and after in this series for improvement, this is bug fix patch.

ui: invoke editor for committing with HGEDITFORM environment variable At the external editor invocation for committing, the value specified as "editform" for "cmdutil.getcommiteditor" is in "HGEDITFORM". This enables external editor to do own customization according to commit types.

cleanup: name unused variables using convention of leading _ This helps checker tools ... and readability for those who knows and follows the convention.

cleanup: rename check-translation.py checker function - don't hide global var

cleanup: remove some unused / duplicate imports

cleanup: fix some list comprehension redefinitions of existing vars In all the remaining cases the comprehension variable is used for the same thing as a previous loop variable. This will mute some pyflakes "list comprehension redefines" warnings.

cleanup: avoid local vars shadowing imports This will mute some pyflakes "import ... shadowed by loop variable" warnings.

cleanup: avoid _ for local unused tmp variables - that is reserved for i18n _ is usually used for i18n markup but we also used it for I-don't-care variables. Instead, name don't-care variables in a slightly descriptive way but use the _ prefix to designate unused variable. This will mute some pyflakes "import '_' ... shadowed by loop variable" warnings.

cleanup: make sure we always access members of imported modules This will make sure we get import errors, even if demandimport is enabled. This will also mute some pyflakes 'imported but unused' warnings.

largefiles: update lfdirstate for unchanged largefiles during linear merging Before this patch, linear merging of modified largefiles causes an unexpected result, if (1) largefile collides with same-name normal one in the target revision and (2) "local" largefile is chosen, even though branch merging between such revisions works correctly. Expected result of such linear merging is marking the largefile as (re-)"added", but the actual result is marking it as "modified". The standin of modified "local largefile" is not changed by linear merging, and updating/merging update lfdirstate entries only for largefiles of which standins are changed. This patch adds the code path to update lfdirstate only for largefiles of which standins are not changed. In this case, "synclfdirstate" should be invoked with True as "normallookup" argument always to force using "normallookup" on dirstate for "n" files, because "normal" may mark target files as "clean" unexpectedly. To reduce cost of "lfile not in filelist", this patch converts "filelist" to a "set" object: "filelist" is used only in (1) the newly added code path and (2) the next line of "filelist = set(filelist)". This is a temporary way to fix with less changes. For fundamental resolution of this kind of problems in the future, "lfdirstate" should be updated with "dirstate" simultaneously during "merge.update" execution: maybe by hooking "recordupdates" (+ total refactoring around lfdirstate handling)

largefiles: keep largefiles from colliding with normal one during linear merge Before this patch, linear merging of modified or newly added largefile causes unexpected result, if (1) largefile collides with same name normal one in the target revision and (2) "local" largefile is chosen, even though branch merging between such revisions doesn't. Expected result of such linear merging is: (1) (not yet recorded) largefile is kept in the working directory (2) largefile is marked as (re-)"added" (3) colliding normal file is marked as "removed" But actual result is: (1) largefile in the working directory is unlinked (2) largefile is marked as "normal" (so treated as "missing") (3) the dirstate entry for colliding normal file is just dropped (1) is very serious, because there is no way to restore temporarily modified largefiles. (3) prevents the next commit from adding the manifest with correct "removal of (normal) file" information for newly created changeset. The root cause of this problem is putting "lfile" into "actions['r']" in linear-merging case. At liner merging, "actions['r']" causes: - unlinking "target file" in the working directory, but "lfile" as "target file" is also largefile itself in this case - dropping the dirstate entry for target file "actions['f']" (= "forget") does only the latter, and this is reason why this patch doesn't choose putting "lfile" into it instead of "actions['r']". This patch newly introduces action "lfmr" (LargeFiles: Mark as Removed) to mark colliding normal file as "removed" without unlinking it. This patch uses "hg debugdirstate" instead of "hg status" in test, because: - choosing "local largefile" hides "removed" status of "remote normal file" in "hg status" output, and - "hg status" for "large2" in this case has another problem fixed in the subsequent patch

largefiles: add test for large/normal conflict at linear merging Before this patch, there is no explicit test for it: test-issue3084.t seems to test such conflict only at branch merging. This patch uses "[debug] dirstate.delaywrite" feature for the tests expecting "M" status of largefiles, to confirm certainly whether files are marked unexpectedly as "clean".

largefiles: put whole "hgmerge" process into the same "wlock" scope Before this patch, there are two distinct "wlock" scopes below in "hgmerge": 1. "merge.update" via original "hg.merge" function 2. "updatelfiles" specific "wlock" scope (to synchronize largefile dirstate) But these should be executed in the same "wlock" scope for consistency, because users of "hg.merge" don't get "wlock" explicitly before invocation of it. - merge in commands This patch puts almost all of the original "hgmerge" implementation into "_hgmerge" to reduce changes.

largefiles: put whole "hgupdaterepo" process into the same "wlock" scope Before this patch, there are two distinct "wlock" scopes below in "hgupdaterepo": 1. "merge.update" via original "hg.updaterepo" function 2. "updatelfiles" specific "wlock" scope (to synchronize largefile dirstate) In addition to them, "dirstate.walk" is executed between these "wlock" scopes. But these should be executed in the same "wlock" scope for consistency, because many (indirect) users of "hg.updaterepo" don't get "wlock" explicitly before invocation of it. "hg.clean" is invoked without "wlock" from: - mqrepo.restore in mq - bisect in commands - update in commands "hg.update" is invoked without "wlock" from: - clone in mq - pullrebase in rebase - postincoming in commands (used in "hg pull -u", "hg unbundle") - update in commands This patch puts almost all original "hgupdaterepo" implementation into "_hgupdaterepo" to reduce changes.

annotate: inline definition of decorate() functions

annotate: rewrite long short-circuit statement by if-elif-else

revert: use modified information from both statuses Using status information against the target ensures we are catching all files with modifications that need reverting. We still need to distinguish fresh modifications for backup purpose. test-largefile is affected because it reverted a file that needs no content change.

revert: drop special case handling for file unknown in parent We had a special case for file not caught by any categories. It was aimed at files missing in wc and wc's parent but existing in the target revision. This is now properly handled using status information.

revert: use "remove" information from both statuses Using status information against the target to make sure we are catching all files that need to be re-added. We still need to distinguish fresh removal because they use a different message.

revert: process removed files missing in target as clean If a file does not exist in target and is marked as removed in the dirstate, we can mark it as clean. There are no changes needed to revert it.

revert: also track clean files Tracking clean files is the simplest way to be able to reports files that need no changes. So we explicitly retrieve them. This fixes a couple of test outputs where the lack of changes was not reported.

revert: triage "deleted" files into more appropriate categories Status can return file as "deleted". This is only a special case related to working directory state: file is recorded as tracked but no file exists on disk. This will never be a state obtainable from manifest comparisons. "Deleted" files have another working directory status shadowed by the lack of file. They will -alway- be touched by revert. The "lack of file" can be seen as a modification. The file will never match the same "content" as in the revert target. From there we have two options: 1. The file exists in the target and can be seen as "modified". 2. The file does not exist in the target and can be seen as "added". So now we just dispatch elements from delete into appropriate categories.

unshelve: silence internal revert This prepares for upcoming revert changes.

tests: fixup issue markers to make check-commit happy

incoming: don't request heads that already are common Pull would send a getbundle command where common heads were sent both as common and head, even though there is no reason to request a common head. The request was thus twice as big as necessary and more likely to hit HTTP header size limits. Instead, don't request heads that already are common. This is fixed in bundlerepo.getremotechanges . It could perhaps also have been fixed in discovery.findcommonincoming but that would have a bigger impact.

tests: improve test coverage for discovery and actual parameters for pulling

changectx: ancestor should only prefer merge.preferancestor if it is a revision The value '*' currently designates that bid merge should be used. The best way to test bid merge is to set preferancestor=* in the configuration file ... but then it would abort with unknown revision '*' when other code paths ended up in changectx.ancestor . Instead, just skip and ignore the value '*' when looking for a preferred ancestor.

merge: show the scary multiple ancestor hint for merges only, not for updates Updates with uncommited changes will always only have one ancestor - the parent revision. Updates between existing revision should (and will) always give the same result no matter which ancestor is used. The warning is thus only relevant when doing a "real" merge.

discovery: prevent crash on unknown remote heads with old repo (issue4337) When a remote is not capable of the `branchmap` wireproto command, we denote incoming heads with None. This leads to a crash in the code in charge of displaying the list of unknown remote heads. We now properly detect this case and display a shorter message in this case. The reason for this `set([None])` value is now documented.

obsstore.create: add a simple safeguard against cyclic markers We detect when there is a cycle in the marker itself (precursors being listed as successors).

debugobsolete: catch ValueError that may be raised by obsstore.create There are already a couple of errors that obsstore.create can raise and we are going to introduce a cycle check too.

check-code: extend try/except/finally check for multiple except clauses

repoview: fix try/except/finally for py2.4

merge with stable