rebase: fix a typo in test-rebase-obsolete We had left a lonely single quote where it shouldn't be!

merge with stable

hgweb: unify import style of error classes It will be enforced by the import checker.

hgweb: do not import templatefilters.revescape and websub as symbol It will be enforced by the import checker.

hgweb: alias ui module as uimod It will be enforced by the import checker.

mergestate.commit: factor out making the list of records Will aid in testing.

mergestate: move _read() call to read constructor With this patch, mergestate.clean() will no longer abort when it encounters an unsupported merge type. However we hold off on testing it until backwards compatibility is in place.

debugmergestate: explain why we create mergestate objects directly We would normally use the read() constructor, but in this case it's fine because - we implement our own reading layer, so the extra parsing done by read() is unnecessary - read() can raise an exception for unsupported merge state records, but here we'd like to handle that separately - debugmergestate needs to be privy to mergestate internals anyway

debugdirstate: convert keyword parameters to **opts Removes the named arguments and replaces them by accessing opts. This will be used in the next patch in the series because we'll be adding more flags to debugdirstate

convert: changed test's progress output format to ignore estimate On my machine, whenever I run all test with a high -j value, test-convert-git.t would consistently fail by displaying an estimate. This patch removes that value from the output.

mq: use repo._bookmarks.recordchange instead of repo._bookmarks.write Before this patch, mq was using repo._bookmarks.write. This patch replaces this code with the recommended way of saving bookmarks changes: repo._bookmarks.recordchange.

mq: indentation change to make the next patch more legible We put the code to be indented in the next patch in a "if True:" block to make it easier to review.

bookmarks: use repo._bookmarks.recordchange instead of repo._bookmarks.write We move from the old api repo._bookmarks.write to the new api repo._bookmarks.recordchange.

localrepo: put bookmark move following commit in one transaction Before this patch, making a commit on a local repo could move a bookmark and both operations would not be grouped as one transaction. This patch makes both operations part of one transaction. This is necessary to switch to the new api to save bookmarks repo._bookmarks.recordchange if we don't want to change the current behavior of rollback. Dirstate change happening after the commit is done is now part of the transaction mentioned above. This leads to a change in the expected output of several tests. The change to test-fncache happens because both lock are now released in the same finally clause. The lock release is made explicitly buggy in this test. Previously releasing lock would crash triggering release of wlock that crashes too. Now lock release crash does not directly result in the release of wlock. Instead wlock is released at garbage collection time and the error raised at that time "confuses" python.

test-resolve.t: switch to mergestate.read() See previous patches for why we're doing this.

localrepo.commit: switch to mergestate.read() See previous patches for why we're doing this.

fileset: switch to mergestate.read() See previous patches for why we're doing this.

commands.summary: switch to mergestate.read() See previous patch for why we're doing this. We do this with a bit of care -- it would be bad form for 'hg summary' to abort completely if we encounter an unsupported merge record. Instead just warn about that and continue with the rest of the summary.

commands.resolve: switch to mergestate.read() See previous patches for why we're doing this.

shelve: switch to mergestate.read() See previous patches for why we're doing this.

mergestate: add a constructor that reads state from disk At the moment it's the same as just creating a new mergestate, but we'll soon move the _read call out of __init__ and in here.

merge.applyupdates: switch to mergestate.clean() See the previous patches for why we're doing this.

localrepo: switch to mergestate.clean() See the previous patches for why we're doing this.

strip: switch to mergestate.clean() See the previous patches for why we're doing this.

mergestate: add a constructor that sets up a clean merge state Eventually, we'll move the read call out of the constructor. This will: - avoid unnecessary reads when we're going to nuke the merge state anyway - avoid raising an exception if there's an unsupported merge record 'clean' seems like a good name for it because I wanted to avoid anything with the word 'new' in it, and 'reset' is more an action performed on a merge state than a way to get a new merge state. Thanks to Martin von Zweigbergk for feedback about naming this.

mergestate: raise structured exception for unsupported merge records We're going to catch this exception in 'hg summary' to print a better error message. This code is pretty untested, so there are no changes to test output. In upcoming patches we're going to test the output more thoroughly.

error: add a structured exception for unsupported merge records We're going to use this in summary to print a better error message.

dirstate: back out 502b56a9e897 Superseded by the parent of this commit.

osutil: make statfiles check for interrupts periodically This is a simpler and faster fix for issue4878 than the contortions performed in 502b56a9e897.

osutil: don't leak on statfiles error Found using the power of the mighty eyeball.

histedit: extracts _isdirtywc function Checking if working copy is dirty was done in few places, this patch extracts it in _isdirtywc procedure.

node: add 'nullhex', hex-encoded nullid We're going to need this for upcoming changes, because the merge state stores nodes as hex strings.

filemerge: introduce class whose objects represent files not in a context Most code is going to barf at the return values here (particularly from data and size), so we restrict it to the filemerge code. This is already somewhat supported via: ctx.filectx(f, fileid=nullid) Indeed, for add/add conflicts (ancestor doesn't have the file) we use precisely that. However, that is broken in subtle ways: - The cmp() function in filectx returns False (identical) for such a filectx when compared to a zero-length file. - size() returns 0 rather than some sort of value indicating that the file isn't present. - data() returns '' rather than some sort of value indicating that the file isn't present. Given the relatively niche use of such filectxes, this seems to be the simplest way to fix all these issues.

filectx: add isabsent method This will indicate whether this filectx represents a file that is *not* in a changectx. This will be used by merge and filemerge code to know about when a conflict is a change/delete conflict. While this is kind of hacky, it is the least bad of all the alternatives. Other options considered but rejected include: - isinstance(fctx, ...) -- not very Pythonic, doesn't support duck typing - fctx.size() is None -- the 'size()' call on workingfilectxes causes a disk stat - fctx.filenode() == nullid -- the semantics around filenode are incredibly confusing. In particular, for workingfilectxes, filenode() is always None no matter whether the file is present on disk or in either parent. Having different behavior for None versus nullid in the merge code is just asking for pain. Thanks to Pierre-Yves David for early review feedback here.

filectx: allow custom comparators We're going to introduce other sorts of filectxes very soon, and we'd like the cmp method to function properly (i.e. commutatively) for them. The only way to make that happen is for this cmp method to call into that specialized one if that defines a custom comparator.

debugmergestate: print out record type for files We're going to add a separate record type for change/delete conflicts soon. We need to make sure they get stored with the correct record type so that older versions of Mercurial correctly abort when they see change/delete records.

merge.applyupdates: don't return early if merge driver's conclude failed Somewhat silly oversight -- this prevented the progress bar from being reset.

convert: use repo._bookmarks.recordchange instead of repo._bookmarks.write Before this patch, convert was using repo._bookmarks.write, a deprecated API for saving bookmarks. This patch changes the use of repo._bookmarks.write to repo._bookmarks.recordchange.

convert: indentation change to make the next patch more legible We put the code to be indented in the next patch in a "if True:" block to make it easier to review.

strip: renaming local variables Renaming local variables to be more precise, i want to store a different list of bookmarks and it would be hard to understand what marks represents in that change therefore renaming it to repomarks. Renamed bookmarks(module) to bookmarksmod as to free up bookmarks which will be used when pluralizing bookmark.

test-rename-merge2: make selected output less arbitrary It's unclear why everything from the first 'updating:' line should be ignored. The arbitrariness makes it confusing that changing the code so e.g. the 'getting 8/f' line is printed later makes it disappear completely from the ouput. The list of 'preserving x for resolve of y' seems convered by the subsequent for loop in the test case. Perhaps it's only copies that are of interests, so let's keep only that part.

test-convert-git: drop a git warning that doesn't occur on Windows I do see this on Linux with 1.7.7.6, but not on Windows with 1.9.5.msysgit.0. Adding a (?) didn't work to conditionally ignore the line; I'm not sure if the (glob) interferes with that.

tests: fix globs for Windows

pull: return 255 value on update failure (issue4948) (BC) Before this patch hg pull -u on UpdateAbort error returned 0 value to the system. This patch fixes this by reraising UpdateAbort with updated error msg.

filemerge: remove leading space from " no tool found..." I could not find or see a reason for the unusual formatting. The lines following it in the test cases are not indented.

test-encoding: enable fuzz testing of utf8b roundtrip This test was written by David R. MacIver <david@drmaciver.com> at the London sprint. We can enable it as the bug in utf8b encoder/decoder has been fixed.

import-checker: reset context to verify convention in function scope I got the following error by rewriting hgweb/webcommands.py to use absolute_import. It is false-positive because the import line appears in "help" function: hgweb/webcommands.py:1297: higher-level import should come first: mercurial This patch makes the import checker aware of the function scope and apply rules recursively.

import-checker: allow import of child modules from package root I got the following error by rewriting hgweb/__init__.py to use absolute_import, which is obviously wrong: Import cycle: mercurial.hgweb.__init__ -> mercurial.hgweb.__init__ "from foo import bar" should not make a cycle if "foo" is a package and if "bar" is a module or a package. On the other hand, it should be detected as a cycle if "bar" is a non-module name. Both cases are doc-tested already, so this patch does not add new doctest.

encoding: extend test cases for utf8b This adds a round-trip helper and a few tests of streams that could cause synchronization problems in the encoder.

merge: make 'cd' and 'dc' actions store the same arguments as 'm' We're going to treat these conflicts similarly to merge conflicts, and this change to the way we store things in memory makes future code a lot simpler.

merge: stop emptying out 'cd' and 'dc' actions (1) These aren't currently read from anywhere, so emptying this out is pointless. (2) These *will* be read from later in upcoming patches, and not emptying them out will be important then.

rebase: add returning value from pullrebase function So far pullrebase function has always returned None value, no matter what orig function returned. This behaviour made impossible for pull to change returned value from mercurial process (it has always ended with 0 value by default). This patch makes pullrebase returning with returned value from orig.

resolve: restore .orig only after merge is fully complete (issue4952) Previously, we'd restore the .orig file after the premerge is complete but before the merge was complete. This would lead to the .orig file potentially containing merge conflict markers in it, as a leftover from the last merge attempt.

monoblue: add a space to separate navigation links in file log footer Before: '(0)-60tip', after: '(0) -60 tip'.

merge: move messages about possible conflicts a litte earlier I actually wanted to reduce the amount of code around the call to applyupdates(), so I tried moving these warnings a little earlier, and I think it makes the output make a little more sense (see changes to test cases).

import-checker: include lineno in warning message This makes it easy to look for imports in function scope.

import-checker: extract function to generate a formatted warning This is the stub to embed lineno. See the next patch for details.

import-checker: make it executable for convenience

test-fileset: glob the updated file count for Windows stability Some of the prior changes are platform dependent, involving symlinks, exec bits and non portable names. An update 10 lines above also globbed this value to avoid differences, since this line isn't of particular interest- the filesets are.

tests: tolerate differences between Linux and Windows error strings These are related to differences in how missing files and network connection failures are displayed. I opted to combine the strings in one line instead of using '#if windows' blocks around entire commands in order to avoid future changes being accidentally missed in the Windows sections. Globbing away the entire output seemed wrong, as it could mask other failures. The raw messages involved are: Linux Windows "* not known" <-> "getaddrinfo failed" "Connection refused" <-> "No connection could be made because the target machine actively refused it" "No such file or directory" <-> "The system cannot find the file specified" Issue 4941 indicates that NetBSD has yet another string for "* not known". Also, the histedit test shows that the missing file is printed first on Windows, last on Linux. That is controlled in windows.py:posixfile if we care to change it.

test-largefiles-cache: add glob for Windows

tests: make 'f' tool open files in binary mode when hexdumping On Windows, the various bundle hexdumps were being truncated at the first occurrence of 0x1a. The line endings in the sequence generation then needs to be standardized on LF so that the file is the same for the tool tests.

merge.applyupdates: only attempt to merge files in mergeactions This only makes a difference when a merge driver is active -- in that case we don't want to try and merge all the files, just the ones still unresolved after the merge driver's preprocess step is over.

filemerge: rename _symlinkcheck to _mergecheck We're going to be adding other checks in here soon.

test-merge-prompt.t: rename to test-merge-changedelete.t Because: (a) the only prompts we test in this file are change/delete prompts, and (b) we're going to be expanding the scope of this test to cover future improvements to change/delete conflicts, including avoiding prompts altogether.

merge with stable

tags: create new sortdict for performance reasons sortdict internally maintains a list of keys in insertion order. When a key is replaced via __setitem__, we .remove() from this list. This involves a linear scan and array adjustment. This is an expensive operation. The tags reading code was calling into sortdict.__setitem__ for each tag in a read .hgtags revision. For repositories with thousands of tags or thousands of .hgtags revisions, the overhead from list.remove() noticeable. This patch creates a new sortdict() so __setitem__ calls don't incur a list.remove. This doesn't appear to have any performance impact on my Firefox repository. But that's only because tags reading doesn't show up in profiles to begin with. I'm still waiting to hear from a user with over 10,000 tags and hundreds of heads on the impact of this patch.

largefiles: specify where .orig files are kept This patch let's the user specify where .orig files are kept using the cmdutil.origpath function

mq: let the user choose where .orig files are kept This patch uses cmdutil.origpath to let the user decide where .orig files are kept when backup files are created.

shelve: choose where .orig file locations are kept This patch uses cmdutil.origpath to let the user set the location of the orig files.

filemerge: choose where .orig files are kept Having .orig files litter your working copy is a common complaint. This patch uses cmdutil.orig to let the user determine where those files should reside.

git-subrepos: revert respects specified location to save .orig files This patch makes revert in git subrepos save .orig files where the user wants them by using the cmdutil.origpath function.

resolve: choose where .orig file locations are kept Having your working copy littered with .orig files is a common HG complaint. This patch uses the cmd.origpath helper function to let the user specify where they should be kept when using the resolve command.

revert: allow configuring the .orig file location

cmdutil: add origbackuppath helper

discovery: pass pushop to _nowarnheads Previously, we passed only enough information to _nowarnheads to perform the old calculation. However, extensions may want to use more information in this calculation. Now that pushop is extensible, extensions will often want access to the additional fields they have added to the pushop, so pass that through. Specifically, the remotenames extension will use this information to allow pushing new heads that are bookmarked without requiring --force.

exchange: pass pushop to discovery.checkheads Previously, we passed a bunch of parameters to discovery.checkheads, but all of the arguments can be fetched out of pushop, which may contain a lot more useful information for extensions now that pushop is extensible.

merge with stable

share: wrap bmstore._writerepo for transaction sensitivity (issue4940) 46dec89fe888 made 'bmstore.write()' transaction sensitive, to restore original bookmarks correctly at failure of a transaction. For example, shelve and unshelve imply steps below: before 46dec89fe888: 1. move active bookmark forward at internal rebasing 2. 'bmstore.write()' writes updated ones into .hg/bookmarks 3. rollback transaction to remove internal commits 4. restore updated bookmarks manually after 46dec89fe888: 1. move active bookmark forward at internal rebasing 2. 'bmstore.write()' doesn't write updated ones into .hg/bookmarks (these are written into .hg/bookmarks.pending, if external hook is spawn) 3. rollback transaction to remove internal commits 4. .hg/bookmarks should be clean, because it isn't changed while transaction running: see (2) above But if shelve or unshelve is executed in the repository created with "shared bookmarks" ("hg share -B"), this doesn't work as expected, because: - share extension makes 'bmstore.write()' write updated bookmarks into .hg/bookmarks of shared source repository regardless of transaction activity, and - intentional transaction failure at the end of shelve/unshelve doesn't restore already updated .hg/bookmarks of shared source This patch makes share extension wrap 'bmstore._writerepo()' instead of 'bmstore.write()', because the former is used to actually write bookmark changes out.

test: use generaldelta in 'test-clonebundle.t' Using generaldelta has effects on the generated bundle.

test: use generaldelta in 'test-ssh.t' This has effects on the capabilities string.

test: use generaldelta in 'test-ssh-bundle1.t' This has effect on the capabilities string.

test: use generaldelta in 'test-strip.t' This has minor effects on test output.

test: use generaldelta in 'test-rebase-newancestors.t' This has some minor effects on test output.

test: use general delta for 'test-commit-amend.t' Using general delta introduces harmless changes to some of the test output.

test: wider matching in 'test-clone-r.t' The test is already somewhat resistant to changes from general delta. We make it wider to catch expected differences.

test: use generaldelta in 'test-bundle-type.t' Generaldelta changes some of the default targets for 'hg bundle'. All cases are already properly tested but some ambiguous specifications are affected.

test: use generaldelta in 'test-patchbomb.t' Using generaldelta changes the default type of bombed bundle.

test: use generaldelta for test-bundle.t This impacts tests of the 'packed' feature. We can safely accept the new output because the 'v1' format is not restricted to old revlog format, the requirements are properly advertised to the client.

test: use generaldelta in 'test-rebase-conflict.t' Doing this has some minor impacts on test output since the rebase backup bundle moves to bundle2.

tests: test generaldelta configuration in test-init.t We turn the feature on for the whole test and test that it can properly be disabled through the config.

test: enable generaldelta early in 'test-treemanifest.t' Having generaldelta on results in minor test output changes (as we are staring at the revlog).

test: enable generaldelta in 'test-copy.t' Doing so has a minimal impact on the test.

test: enable generaldelta in 'test-debugcommand.t' The test logic is not affected by generaldelta, so we move it to generaldelta early. Some of the fuzzy matching had to be extended to cover "delta".

test: use generaldelta in test-manifestv2.t Generaldelta and bundle2 have no real influence on this test. So we can safely switch it early.

test: explicitly use generaldelta in 'test-rebase-mq-skip.t' This induces some trivial changes in the tests. This happen in its own changeset for clarity purposes.

test: using generaldelta in 'test-lfconvert.t' This leads to a harmless test change.

test: enforce generaldelta format with the right option The option that will become true by default is now 'usegeneraldelta' We have to adjust the change made in b75fed55f6dc to actually achieve its goal.

test: enforce non-generaldelta base repository in 'test-generaldelta' This test check the difference between various configurations we have to pin the type of some repositories to ensure the test is still correct when we change the default.

test: use generaldelta in 'test-hgweb-command.t' Using generaldelta has a meaningless impact on the capabilities string.

stream: sort stream capability before serialisation We want that capability to be stable in our testing. This is currently not an issue because the set is size 1, but this will be once generaldelta related data gets in there.

test: spread capabilities replies in 'test-hgweb-commands' Any changes to the capabilities string was very hard to review. So we introduce two tests. A small one that validates the format and a second one to validate the content, where each capability is on its own line to make changes simpler to review.

phase: improve retractboundary perf The existing retractboundary implementation computed the new boundary by walking all descendants of all existing roots and computing the new roots. This is O(commits since first root), which on long repos can be hundreds of thousands of commits. The new algorithm only updates roots that are greater than the new root locations. For common operations like commit on a repo with the earliest root several hundred thousand commits ago, this makes retractboundary go from 1 second to 0.008 seconds. I tested it by running the test suite with both implementations and checking that the root results were always the identical. There was some discussion on IRC about the safety of this (i.e. what if the new nodes are already part of the phase, etc). I've looked into it and believe this patch is safe: 1) The old existing code already filters the input nodes to only contain nodes that require retracting (i.e. we only make node X a new root if the old phase is less than the target phase), so there's no chance of us adding a unnecessary root to the phase (unless the input root is made unnecessary by another root in the same input, but see point #3). 2) Another way of thinking about this is: the only way the new algorithm would be different from the old algorithm is if it added a root that is a descendant of an old root (since the old algorithm would've caught this in the big "roots(%ln::)". At the beginning of the function, when we filter out roots that already meet the phase criteria, the *definition* of meeting the phase criteria is "not being a descendant of an existing root". Therefore, by definition none of the new roots we are processing are descendants of an existing root. 3) If two nodes are passed in as input, and one node is an ancestor of the other (and therefore the later node should not be a root), this is still caught by the 'roots(%ln::)' revset. So there's no chance of an extra root being introduced that way either.

merge with stable

format: introduce 'format.usegeneraldelta` This option will make repositories created as general delta by default but will not make Mercurial aggressively recompute deltas for all incoming bundle. Instead, the delta contained in the bundle will be used. This will allow us to start having general delta repositories created everywhere without triggering massive recomputation costs for all new clients cloning from old servers.

scmutil: extract general delta config handling in a function General delta is currently controlled by a single option, we will introduce a new one in the next changeset. We extract the logic in a function while it is simple.

test: test-bundle-type.t needs to work more universally The cut and head utilities on Solaris have weird differences from the GNU versions. The f helper script does a dump more nicely than those tools, anyway.

bash_completion: add -p|--patch|--stat support for shelve

test-convert-git: silence commit output for stablity When running the tests with 1.7.7.6, I get 'files' and 'insertions' instead of the singular forms, and there is also an additional '0 deletions(-)' at the end. Since this doesn't seem important to the test, silence it.

rebase: preserve the 'intermediate-source' attribute of grafts Preserving the 'source' attribute of grafts started with a69a77a80900, which predates the introduction of 'intermediate-source' in 51930a7180bd by a year and a half. It looks like not preserving this was an oversight. On a related note, notice how the source value of 32af76 is no longer visible in the graph above this test. Is it reasonable to import the sha1 translation from evolve.py:relocate() into scmutil or similar, and use that to fixup these attributes as well as the commit message? (I realize that evolve is still experimental, but I don't see a way to do this from the evolve extension.)

merge with stable

verify: add a hook that can let extensions manipulate file lists Without a hook of this nature, narrowhg[0] clones always result in 'hg verify' reporting terrible damage to the entire repository history. With this hook, we can ignore files that aren't supposed to be in the clone, and then get an accurate report of any damage present (or not) in the repo. 0: https://bitbucket.org/Google/narrowhg

resolve: don't abort when file is missing A file being missing is a completely valid situation in which the user may want to re-resolve merge conflicts. Mercurial already maintains backups of local data, so this turns out to be easy to handle.

filemerge: treat EOF at prompt as fail, not abort Previously we'd abort the merge entirely if there was an EOF at the prompt. This is unnecessary -- it's much better to simply fail and treat the file as unresolved instead.

test-merge-tools.t: add 'hg resolve --list' output to tests In upcoming patches we're going to be making some improvements to the ':prompt' merge tool, and one of them is recovering appropriately from exceptions. It is important in such cases to make sure that the merge state is written out properly. I went through the output and verified that it all makes sense.

error: add structured exception for EOF at prompt We'll catch this exception for promptchoice queries to provide better error handling.

test-merge-prompt.t: unset HGMERGE We're going to test behavior with and without HGMERGE, so disable it for the test.

hgweb: compute changeset parents and children for log pages lazily Log pages, i.e. changelog, filelog and search results page computed children and parents for each changeset shown, because spartan hgweb style shows this info. Turns out, computing all this is heavy and also unnecessary for log pages in all other hgweb styles. Luckily, templates allow an easy way to do computations on demand: just pass the heavy part of code as a callable and it will be only called when needed. Here are some benchmarks on the mercurial repository (best of 3): time wget http://127.0.0.1:8021/ before: 0m0.050s after: 0m0.040s time wget http://127.0.0.1:8021/?revcount=960 before: 0m1.164s after: 0m0.389s time wget http://127.0.0.1:8021/log/tip/mercurial/commands.py before: 0m0.047s after: 0m0.042s time wget http://127.0.0.1:8021/log/tip/mercurial/commands.py?revcount=960 before: 0m0.830s after: 0m0.434s

filemerge: move :merge-local/other symlink check to precheck In upcoming patches we're going to add more checks to the function currently named _symlinkcheck. This consolidates all those checks.

test-merge-types.t: add tests for :merge-local/:merge-other with symlinks This had no test coverage.