push: elevate phase transaction to cover entire operation This patch series is intended to allow bundle2 push reply part handlers to make changes to the local repository; it has been developed in parallel with an extension that allows the server to rebase incoming changesets while applying them. Most pushes already open a transaction in order to sync phase information. This diff replaces that transaction with one that spans the entire push operation. This transaction will be used in a later patch to guard repository changes made during the reply handler.

pull: extract transaction logic into separate object This patch series is intended to allow bundle2 push reply part handlers to make changes to the local repository; it has been developed in parallel with an extension that allows the server to rebase incoming changesets while applying them. Aside from the transaction logic, the pulloperation class is used primarily as a logic-free data structure for storing state information. This diff extracts the transaction logic into its own class that can be shared with push operations.

record: don't honor format-changing diffopts (issue4459) record does support the whitespace options, so honor those. In upcoming patches we'll replace all the other uses.

patch.difffeatureopts: add a feature for format-changing diffopts Typically, callers that are interested in this set are also interested in the others. We still add it as a first-class flag for convenience.

patch.difffeatureopts: add a feature for whitespace diffopts These aren't exactly format-breaking features -- just ones for which patches applied to a repo will produce incorrect commits, In any case, some commands like record and annotate only care about this feature.

patch.difffeatureopts: add a feature for diff.git This deserves to be its own feature -- indeed, this is often the only feature several commands care about.

patch: rename diffopts to diffallopts This is meant to be used when all features are explicitly requested. diffopts is kept as an alias for now -- it will go away soon.

patch: add a new function to initialize diffopts by feature Not all callers are interested in all diffopts -- for example, commands like record (which use diff internally) break when diffopts like noprefix are enabled. This function will allow us to add flags that callers can use to enable only the features they're interested in.

patch.diffopts: use a dict for initialization In upcoming patches we'll conditionally add to and remove from the dict.

largefiles: split the creation of a normal matcher out of its install method Refactoring addremove to support subrepos will need the ability to keep passing the same matcher and narrowing it, instead of monkey patching scmutil's matcher.

addremove: print relative paths when called with -I/-X (BC) For "hg addremove 'glob:*.py'", we print any paths added or removed as relative to the current directory, but when "hg addremove -I 'glob:*.py'" is used, we use the absolute path (relative from the repo root). It seems like they should be the same, so change it so we use relative paths in both cases. Continue to use absolute paths when no patterns are given.

merge with stable

merge with i18n

i18n-pt_BR: synchronized with 3bd577a3283e

i18n-ja: synchronized with edf29f9c15f0

Added signature for changeset 902554884335

Added tag 3.2.2 for changeset 902554884335

merge: before cd/dc prompt, check that changed side really changed Before, merging would in some cases ask "wrong" questions about "changed/deleted" conflicts ... and even do it before the resolve phase where they can be postponed, re"resolved" or answered in bulk operations. Instead, check that the content of the changed file really did change. Reading and comparing file content is expensive and should be avoided before the resolve phase. Prompting the user is however even more expensive. Checking the content here is thus better. The 'f in ancestors[0]' should not be necessary but is included to be extra safe.

largefiles: don't show largefile/normal prompts if one side is unchanged

tests: add test-issue3084.t cases for 'changed but same' as for 'unchanged' Use suffix -same for cases where file changed but content is the same - that is the case where manifestmerge doesn't detect that a file is unchanged. (The suffix -id is already used for cases where the file didn't change - that is the trivial case where manifestmerge detects that the file is unchanged.) These new tests are good but the results are bad. There shouldn't be any merge conflicts or prompts when one side didn't change.

tests: clean-up of largefiles tests in test-issue3084.t Prepare for adding more test cases to the systematic testing, moving the test from ac3b3a2d976d to another section.

pushkey: gracefully handle prepushkey hook failure (issue4455) This allow to gracefully report the failure of the bookmark push and carry on. Before this change set. Local push would plain quit and wireprotocol would failed in various ungraceful way.

hook: raise a more specialized HookAbort exception when a hook fails We need to gracefully handle some aborts for pushkey, especially because it leads to a user-facing crash over the wireprotocols. So we need a more specialized exception to catch.

help: fix typo in files help

revset: fix first and last for generatorset (issue4465) The code was just plain wrong.

mq: fix update of headers that occur in the "wrong" order Mq tried to insert headers in the right order. Sometimes it would stop searching before checking all headers and it could thus duplicate a header instead of replacing it.

subrepo: add status support for ignored files in git subrepos Retrieving the status of a git subrepo did not show ignored files. Using 'git ls-files', we can retrieve these files and display the correct status.

manifest: document the extra letter in working copy manifest node As the second developer to get confused by this in November, I'm adding some documentation for the next poor soul.

hgweb: send proper HTTP response after uncaught exception This patch fixes a bug where hgweb would send an incomplete HTTP response. If an uncaught exception is raised when hgweb is processing a request, hgweb attempts to send a generic error response and log that exception. The server defaults to chunked transfer coding. If an uncaught exception occurred, it was sending the error response string / chunk properly. However, RFC 7230 Section 4.1 mandates a 0 size last chunk be sent to indicate end of the entity body. hgweb was failing to send this last chunk. As a result, properly written HTTP clients would assume more data was coming and they would likely time out waiting for another chunk to arrive. Mercurial's own test harness was paving over the improper HTTP behavior by not attempting to read the response body if the status code was 500. This incorrect workaround was added in ba6577a19656 and has been removed with this patch.

tests: test rebase of merge of target ancestors

tests: test coverage of parent order with p1 outside rebase set

tests: add test for rebasing merges with ancestors of the rebase destination This shows sub-optimal behaviour. The user gets a merge prompt that it is very hard to justify.

merge: 0 is a valid ancestor different from None Most internal functions can take either a hash or an integer. Merge did however not handle 0 as revision 0. Now it does.

merge with stable

revert: look for copy information for all local modifications Renaming a file over an existing one marks the file as modified. So we track rename source in modified file too.

rename: properly report removed and added file as modified (issue4458) The result of 'hg rm' + 'hg rename' disagreed with the one from 'hg rename --force'. We align them on 'hg move --force' because it agrees with what 'hg status' says after the commit. Stopping reporting a modified file as added puts an end to the hg revert confusion in this situation (issue4458). However, reporting the file as modified also prevents revert from restoring the copy source. We fix this in a later changeset. Git diff also stop reporting the add in the middle of the chain as add. Not sure how important (and even wrong) it is.

manifest: fix a bug where working copy file 'add' mark was buggy Because the same dictionary was used to (1) get node from parent and (2) store annotated version, we could end up with buggy values. For example with a chain of renames: $ hg mv b c $ hg mv a b The value from 'b' would be updated as "<old-a>a", then the value of c would be updated as "<old-b>a'. With the current dictionary sharing this ends up with: '<new-c>' == '<old-a>aa' This value is double-wrong as we should use '<old-b>' and a single 'a'. We now use a read-only value for lookup. The 'test-rename.t' test is impacted because such a chained added file is suddenly detected as such.

merge: be precise about what merged into what in short desc

docker: add Docker files for running an Apache mod_wsgi server I frequently find myself wanting to run hgweb in a production-like environment, with a real HTTP server and multiple WSGI workers. This patch introduces a Docker environment for running Mercurial under Apache + mod_wsgi. With just a few command executions, it is possible to spin up a Docker container running hgweb. The container is tailored for Mercurial developers wanting to run Mercurial from a source checkout. It is **not** meant to be something suitable for production use. The container provides a default hgweb environment with an empty repository that allows pushes. You can thus start a container and push your favorite repository there for quick testing. The container is designed to allow customizations. Users can provide their own hgweb configurations and mount existing directories containing repositories into the container. The behavior of the container and how to control things is documented in the README.rst file.

merge: use None as filename for base in 'both created' conflicts Instead of using a file that we know is not in the common ancestor's maniffest, let's use None. This is safe as the only place that cares about the value (applyupdates) already checks if the item exists in the ancestor.

merge: break out "both renamed a -> b" case We can further limit the scope of the 2-way merge case by breaking out the case where the file was not created from scratch on both sides but rather renamed in the same way (and is therefore a 3-way merge). This involves copying some code, but it makes it clearer which case the "Note:" in the code refers to.

merge: separate out "both created" cases When 'f' is not in 'ma', 'a' will be 'nullid' and all the if/elif conditions that check whether some one nodeid is equal to 'a' will fail, and the else-clause will instead apply. We can make that more explicit by creating a separate 'm' action for the case where 'a' is 'nullid'. While it does mean copying some code, perhaps it makes it a little clearer which codepaths are possible, and which cases the "Note:" in the code refers to. It also lets us make the debug action messages a little more specific.

merge: indent to prepare for next patch

merge with stable

factotum: convert Plan 9 quoted string to Python string

plan9: setup check for current python build

add: check for the existence of a file matched inexactly before adding it The change in 10697f29af2b created a problem on Windows and OS X: --- /usr/local/mercurial/tests/test-issue660.t +++ /usr/local/mercurial/tests/test-issue660.t.err @@ -47,6 +47,8 @@ Should succeed - shadow removed: $ hg add b + adding b/b + b/b does not exist! Prior to the failing 'hg add', the file 'b/b' was added and committed, then 'b' was recursively deleted from the filesystem, file 'b' was created and the delete was recorded with 'hg rm --after'. This add is attempting to record the existence of file 'b'. A filesystem that is not case sensitive prevents dirstate.walk() from skipping its step 3, and step 3 has the effect of inserting removed files into the walk list. The Linux code doesn't run through step 3, and didn't exhibit the problem. It's not clear why a non case sensitive filesystem triggers step 3, given that the path normalization occurs in step 2. Prior to 10697f29af2b, part of the check here was 'f not in repo.dirstate' instead of 'f not in wctx'. Files in the 'r' state are filtered out of context.__contains__() but not dirstate.__contains__(). Therefore the removed file name wasn't added to the list of files to add when checking against dirstate. That change was to allow removed files to be readded, but adding a file that doesn't exist is nonsensical. If the user specifies a missing file, it will be an exact match and will still fail.

tests: choose the proper environment variable style for the platform Windows was printing out 'commit $HG_NODE' instead of the actual hash.

tests: fix another multi-statement hook for Windows The double quotes are necessary, otherwise it tries to pipe into a program named 'short'. An '&' could serve as a command separator on Windows instead of ';', but I don't see any obvious way to swap these depending on the platform. In this case though, there really wasn't a need for multiple statements.

run-tests: set a default largefiles usercache in the default hgrc file This fixes a test failure introduced in 4be754832829 on Windows and OS X, where the cached largefile wasn't being deleted because the named .cache directory didn't exist. It only existed on Linux because the test suite sets $HOME to the directory of the test being run, and Linux uses $HOME/.cache by default. Most of the other largefiles tests explicitly set this value at the top of their scripts, but test-largefiles-update.t didn't pick that up when it was created. Those scripts that do set a value will override this. We could just set the parameter in the test-largefiles-update.t script, but there are a few other non obvious tests that exercise largefiles too. These largefiles end up being cached in the user's real cache, so proper hygiene dictates that this not be left to each individual test script.

merge: remove obsolete check for untracked files in 'dm' action Since 0776a6cababe (merge: don't use unknown(), 2012-02-09), untracked files are no longer included in the manifest diff, so there is no need to check exclude them when renaming files for directory moves with the 'dm' action.

merge: remove dead assignment in applyupdates()

merge: move calculateupdates() before applyupdated() calculateupdates() happens before applyupdates(), so move it before in the code. That also moves it close to manifestmerge(), which is a good location as calculateupdates() is the only caller of manifestmerge().

merge: remove unused variables from _checkcollision()

largefiles: avoid exec-bit examination on the platform being unaware of it Changeset 24600c9d7f4e introduced the examination of exec bit of largefiles in "hg status --rev REV" case, but it doesn't avoid it on the platform being unaware of exec-bit (e.g. on NTFS of Windows).

merge with stable

changegroup: fix file linkrevs during reorders (issue4462) Previously, if reorder was true during the creation of a changegroup bundle, it was possible that the manifest and filelogs would be reordered such that the resulting bundle filelog had a linkrev that pointed to a commit that was not the earliest instance of the filelog revision. For example: With commits: 0<-1<---3<-4 \ / --2<--- if 2 and 3 added the same version of a file, if the manifests of 2 and 3 have their order reversed, but the changelog did not, it could produce a filelog with linkrevs 0<-3 instead of 0<-2, which meant if commit 3 was stripped, it would delete that file data from the repository and commit 2 would be corrupt (as would any future pulls that tried to build upon that version of the file). The fix is to make the linkrev fixup smarter. Previously it considered the first manifest that added a file to be the first commit that added that file, which is not true. Now, for every file revision we add to the bundle we make sure we attach it to the earliest applicable linkrev.

merge: consistently use single quotes for non-user-facing strings Because I'm getting tired of searching for both 'O' and "O".

localrepo: add a currenttransaction method This method returnx the current transaction or None: it will allow a cache writer to hook in an existing transaction.

repoview: extract actual hidden cache writing in its own function This will allow the generation of this cache within the transaction. Relying on the transaction will reduce the chance of reader seeing bad cache.

templates: fix broken "less" & "more" links in paper style (issue4460) "/search", which is an invalid command in hgweb, was mistakenly used for "[show] more [revsets]" and "[show] less [revsets]" links on search page in templates "paper" (and those which inherit paper, such as coal) before and worked fine until 6e1fbcb18a75, which made hgweb more strict about invalid commands.

push: stop independent usage of bundle2 in syncphase (issue4454) The phase-syncing code was using bundle2 if the remote supported it. It was doing so without regard to bundle2 activation on the client. Moreover, the phase push is now properly included in the unified bundle2 push, so having extra code in syncphase should be useless. If the remote is bundle2-enabled, the phases should already be synced. The buggy verification code was leading to a crash when a 3.2 client was pushing to a 3.1 server. The real bundle2 path detected that their versions were incompatible, but the syncphase code failed to, sending an incompatible bundle2 to the server. We drop the useless and buggy code as a result. The "else" clause is de-indented in the process.

dirstate: speed up repeated missing directory checks In a mozilla repo with tip at bb3ff09f52fe, hg update tip~1000 && time hg revert -nq -r tip . displays ~4:20 minutes. With tip~100, it runs in ~11 s. With revision 100000, it did not finish in 12 minutes. Revert calls dirstate.status() with a matcher that matches each file in the target revision. The main problem [1] lies in dirstate._walkexplicit(), which looks for matching deleted directories by checking whether each path is prefix of any path in the dirstate. With m files in the dirstate and n files in the target revision that are not in the dirstate, this is clearly O(m*n). Let's improve by keeping a lazily initialized set of all the directories in the dirstate, so the time becomes O(m+n). After this patch, the 4:20 minutes become 5.5 s, while for a single missing path, it slows down from 1.092 s to 1.150 s (best of 4). The >12 min case becomes 5.8 s. [1] A narrower optimization would be to make revert take the fast path for '.' and '--all'.

revert: access status fields by name rather than index For better readability.

subrepo: remove "_getstorehashcachepath" referred by no other code paths

subrepo: replace direct file APIs around "writelines" by "vfs.writelines" This patch also replaces "self._getstorehashcachepath" (building absolute path up) by "self._getstorehashcachename" (building relative path up), because "vfs.writelines" requires relative path.

vfs: add "writelines" This patch allows "writelines" to take "mode" and "notindexed" arguments, because subsequent patch for subrepo requires both.

vfs: add "notindexed" argument to invoke "ensuredir" with it in write mode This patch uses "False" as default value of "notindexed" argument, even though "vfs.makedir()" uses "True" for it, because "os.mkdir()" doesn't set "_FILE_ATTRIBUTE_NOT_CONTENT_INDEXED" attribute to newly created directories.

subrepo: replace direct file APIs around "readlines" by "vfs.tryreadlines" This patch also replaces "self._getstorehashcachepath" (building absolute path up) by "self._getstorehashcachename" (building relative path up), because "vfs.tryreadlines" requires relative path. This patch makes "_readstorehashcache()" return "[]" (returned by "vfs.tryreadlines()"), when cache file doesn't exist, even though "_readstorehashcache()" returned '' (empty string) in such case before this patch. "_readstorehashcache()" is invoked only by the code path below in "_storeclean()": for filehash in self._readstorehashcache(path): if filehash != itercache.next(): clean = False break In this case, "[]" and '' don't differ from each other, because both of them cause avoiding iteration of "for loop".

vfs: add "readlines" and "tryreadlines" This patch allows "readlines" and "tryreadlines" to take "mode" argument, because "subrepo" requires to read files not in "rb" (binary, default for vfs) but in "r" (text) mode in subsequent patch.

subrepo: add "_cachestorehashvfs" to handle cache store hash files via vfs This "vfs" object will be used by subsequent patches to handle cache store hash files without direct file APIs. This patch decorates "_cachestorehashvfs" with "@propertycache" to delay vfs creation, because it is used only for cooperation with other repositories. In this patch, "/" is used as the path separator, even though "self._repo.join" uses platform specific path separator (e.g. "\\" on Windows). But it is reasonable enough, because "store" and other management file handling already include such implementation, and they work well.

subrepo: remove "_calcfilehash" referred by no other code paths

subrepo: replace "_calcfilehash" invocation by "vfs.tryread" "_calcfilehash" can be completely replaced by simple "vfs.tryread" invocation. def _calcfilehash(filename): data = '' if os.path.exists(filename): fd = open(filename, 'rb') data = fd.read() fd.close() return util.sha1(data).hexdigest() Building absolute path "absname" up by "self._repo.join" for files in "filelist" is avoided, because "vfs.tryread" does so internally.

subrepo: replace "os.path.exists" by "exists" via wvfs of the parent Existance of specified "path" should be examined by "exists" via wvfs of the parent repository, because the working directory of the parent repository may be in UTF-8 mode. Wide API should be used via wvfs in such case. In this patch, "/" is used as the path separator, even though "path" uses platform specific path separator (e.g. "\\" on Windows). But it is reasonable enough, because "store" and other management file handling already include such implementation, and they work well.

subrepo: avoid redundant "util.makedirs" invocation "util.makedirs" for the (sub-)repository root of "hgsubrepo" is also executed in the constructor of "localrepository", if "create" is True and ".hg" of it doesn't exist. This patch avoids redundant "util.makedirs" invocation in the constructor of "hgsubrepo".

merge: remove confusing comment about --force manifestmerge() has a piece of code that's roughly: if not force and different: abort else: # if different: old untracked f may be overwritten and lost ... The comment only talks about what happens when 'different' is true, and in combination with the if-block above, that must mean that it is only about what happens when 'force and different'. It seems quite fine that files are overwritten when 'force' is true, so let's remove the comment. As it stands, it can easily be interpreted as a TODO (which is how I interpreted it at first).

phases: read pending data when appropriate If we are called by a hook and pending data exists, read those.

bookmark: read pending data when appropriate If we are called by a hook and pending data exists, read it.

test-bundle2: check visible data in pre/post-transaction hooks We are about to make bookmarks and phases available for hooks. Therefore we need a witness for this new availability. We introduce the new hooks in a distinct changeset to reduce the noise in the ones with actual changes.

transaction: write pending generated files Such file are generated with a .pending prefix. It is up to the reader to implement the necessary logic for reading pending files. We add a test to ensure pending files are properly cleaned-up in both success and error cases.

transaction: have _generatefile return a boolean The function returns True if any files were generated. This will be used to know if any pending files have been written.

transaction: allow generating files with a suffix This will allow us to generate temporary pending files. Files generated with a suffix are assumed temporary and will be cleaned up at the end of the transaction.

transaction: fix some docstring grammar

transaction: accept a 'location' argument for registertmp This will allow generation of temporary files outside of store. This will be useful for bookmarks.

tests: handle differences between missing file error strings on Windows and Unix

run-tests: don't warn on unnecessary globs mandated by check-code.py When test output is processed, if os.altsep is defined (i.e. on Windows), TTest.globmatch() will cause a warning later on if a line has a glob that isn't necessary. Unfortunately, the regex checking in check-code.py doesn't have this context. Therefore we ended up with cases where the test would get flagged with a warning only on Windows because a glob was present, because check-code.py would warn if it wasn't. For example, from test-subrepo.t: $ hg -R issue1852a push `pwd`/issue1852c pushing to $TESTTMP/issue1852c (glob) The glob isn't necessary here because the slash is shown as it was provided. However, check-code mandates one to handle the case where the default path has backslashes in it. Break the cycle by checking against a subset of the check-code rules before flagging the test with a warning, and ignore the superfluous glob if it matches a rule. This change fixes warnings in test-largefiles-update.t, test-subrepo.t, test-tag.t, and test-rename-dir-merge.t on Windows. I really hate that the rules are copy/pasted here (minus the leading two spaces) because it would be nice to only update the rules once, in a single place. But I'm not sure how else to do it. I'm open to suggestions. Splitting some of the rules out of check-code.py seems wrong, but so does moving check-code.py out of contrib, given that other checking scripts live there. There are other glob patterns that could be copied over, but this is enough to make the current tests run on Windows.

update: remove unnecessary check for unknown files with --check As far as I and the test suite can tell, the checks in manifestmerge() already report the errors (whether or not --check is given), so we don't need to call merge.checkunknown(). Since this is the last call to the method, also remove the method.

merge with stable

tests: move a multi-statement debuglocks hook into a shell script for Windows Before this patch, a part of "test-push-hook-lock.t" fails unexpectedly on Windows environment, because semicolon (";") isn't recognized as the command separator by "cmd.exe". This is fixed the same way as a similar issue in 7c253c23de3b.

tests: fix globs for Windows test-largefiles-update.t, test-subrepo.t, test-tag.t, and test-rename-dir-merge.t still warn about no result returned because of unnecessary globs that test-check-code-hg.t wants, relating to output for pushing to, pulling from and moving X to Y.

run-tests: include quotes in the HGEDITOR value when storing sys.executable This fixes test-install.t on Windows that broke in 2122b82b6987 when shlex.split() was added to the debuginstall command: @@ -7,8 +7,11 @@ checking installed modules (*mercurial)... (glob) checking templates (*mercurial?templates)... (glob) checking commit editor... + Can't find editor 'c:\Python27\python.exe -c "(omitted)"' in PATH + (specify a commit editor in your configuration file) checking username... - no problems detected + 1 problems detected, please check your install! + [1] What happens is that shlex.split() on Windows turns this: c:\Python27\python.exe -c "import sys; sys.exit(0)" into this: ['c:Python27python.exe', '-c', 'import sys; sys.exit(0)'] While technically a regression, most programs on Windows live in some flavor of 'Program Files', and therefore the environment variable needs to contain quotes anyway to handle the space. This wasn't handled prior to the shlex() change, because it tested the whole environment variable to see if it was an executable, or split on the first space and tested again.

mq: when adding headers in plain mode, separate them from message (issue4453) c87f2a5a6e49 did a clean-up in one direction ... but we want it in the other direction.

mq: introduce insertplainheader - same naive implementation as before

mq: when setting message in plain mode, separate it from header (issue4453) Fix inconsistent handling of plain header separation in mq patcheader - and contrary to c87f2a5a6e49, do it in the direction of having an empty line between header and description. Plain patches are like mails and should thus have an empty line between headers and body in compliance with RFC 822 3.1.

setdiscovery: avoid a full changelog graph traversal We were definitely being suboptimal here: we were constructing two full sets, one with the full set of common nodes (i.e. a graph traversal) and one with all nodes. Then we subtract one set from the other. This whole process is O(commits) and causes discovery to be significantly slower than it should be. Instead, keep track of common incrementally and keep undecided as small as possible. This makes discovery massively faster on large repos: on one such repo, 'hg debugdiscovery' over SSH with one commit missing on the client and five on the server went from 4.5 seconds to 1.5. (An 'hg debugdiscovery' with no commits missing on the client, i.e. connection startup time, was 1.2 seconds.)

ancestor: add a way to remove ancestors of bases from a given set This and missingancestors can share state, which will turn out to be perfect for set discovery.