sshpeer: introduce a "doublepipe" class This class is responsible for ensuring we still process the server output streamed through the ssh's 'stderr' pipe during the initial wait for other protocol streams. It currently only works on posix system because of its use of 'select.select'.

util: add a simple poll utility We'll use it to detect when a sshpeer have server output to be displayed. The implementation is super basic because all case support is not the focus of this series.

crecord: fix blue artifact bug coming back from help screen Before this patch, with record's curses interface: - When one pressed the ? key to see the help menu and then leaves the help menu - Then, blue artifacts would remain on the screen This patch forces a screen clear / refresh after coming back from the help menu. Since we don't have UI test for crecord I checked on my machine that the bug does not reproduce.

scmutil: consistently return subrepos relative to ctx1 from itersubrepos() Previously, if a subrepo was added in ctx2 and then compared to another without it (ctx1), the subrepo for ctx2 was returned amongst all of the ctx1 based subrepos, since no subrepo exists in ctx1 to replace it in the 'subpaths' dict. The two callers of this, basectx.status() and cmdutil.diffordiffstat(), both compare the yielded subrepo against ctx2, and thus saw no changes when ctx2's subrepo was returned. The tests here previously didn't mention 's/a' for the 'p1()' case. This appears to have been a known issue, because some diffordiffstat() comments mention that the subpath disappeared, and "the best we can do is ignore it". I originally ran into the issue with some custom convert code to flatten a tree of subrepos causing hg.putcommit() to abort, but this new behavior seems like the correct status and diff behavior regardless. (The abort in convert isn't something users will see, because convert doesn't currently support subrepos in the official repo.)

context: introduce the nullsub() method Ultimately, this will be used by scmutil. The subrepo module already imports it, so it can't import the subrepo module to access the underlying method.

subrepo: introduce the nullsubrepo() method This will be used in an upcoming patch. It's a one-off use, but seems better to be contained in the subrepo module, than for the next patch to overwrite the _ctx and _state fields in another module. '' is used as the default revision in subrepo.state() if it can't be found, so it seems like a safe choice.

ssl: prompt passphrase of client key file via ui.getpass() (issue4648) This is necessary to communicate with third-party tools through command-server channel. This requires SSLContext backported to Python 2.7.9+. It doesn't look nice to pass ui by sslkwargs, but I think it is the only way to do without touching various client codes including httpclient (aka http2). ui is mandatory if certfile is specified, so it has no default value. BTW, test-check-commit-hg.t complains that ssl_wrap_socket() has foo_bar naming. Should I bulk-replace it to sslwrapsocket() ?

https: do not inherit httplib.HTTPSConnection that creates unused SSLContext HTTPSConnection of Python 2.7.9 creates SSLContext in __init__, which involves a password prompt for decrypting the private key. This means the password was asked twice, one for unused SSLContext, and next for our ssl function. Because our httpsconnection replaces connect() method at all, we can simply drop httplib.HTTPSConnection. Instead, class and instance attributes are copied from it. HTTPSConnection of Python 2.7.8 and 2.6.9 seem to have no such problem. https://hg.python.org/cpython/file/v2.7.9/Lib/httplib.py#l1183

test-https: test basic functions of client certificate authentication Because hgweb doesn't support client certificates, I just patched it to require client certificates that are signed and verified by the server certificate. This won't be ideal for production servers, but should be okay for the test. The encrypted key file will be used by future patches. I couldn't figure out a way to redirect a password prompt provided by OpenSSL, so it isn't tested for now.

phases: add `hg help phases` hint to failures to edit public commits There were a couple of locations that were missing this hint after an edition of some kind failed due to the public phase.

phases: rewrite "immutable changeset" to "public changeset" The phrase "cannot edit immutable changeset" is kind of tautological. Of course unchangeable things can't be changed. We instead mention "public" and provide a hint so that we can point to the actual problem. Even in cases where some operation other than edition cannot be performed, "public" gives the root cause that results in the "immutable" effect. There is a precedent for saying "public" instead of "immutable", for example, in `hg commit --amend`.

revlog: raise an exception earlier if an entry is too large (issue4675) Before we were relying on _pack to error out when trying to pass an integer that was too large for the "i" format specifier. Now we check this earlier so we can form a better error message. The error message unfortunately must exclude the filename at this level of the call stack. The problem is that this name is not available here, and the error can be triggered by a large manifest or by a large file itself. Although perhaps we could provide the name of a revlog index file (from the revlog object, instead of the revlogio object), this seems like too much leakage of internal data structures. It's not ideal already that an error message even mentions revlogs, but this does seem unavoidable here.

wix: move library.zip and all *.pyd into a lib/ folder This makes the root install folder (on Windows) nice and tidy. The only files left in the root folder are: hg.exe python27.dll COPYING.rtf ReadMe.html the last of which was probably out-of-date 7 years ago

sshpeer: run the ssh command unbuffered This is necessary to use non-blocking IO base on polling. Such polling is needed to restore real time output with ssh peer. Changeset fce065538bcf is talking about 5x regression on Mac OS X when playing with this value. So we introduced our own buffering layer in previous changesets. This seems to keep the regression away (we are even issuing much less read).

sshpeer: use a 'bufferedinputpipe' for standard output of the ssh process We need this pipe to still be buffered when will switch to unbuffered pipe. (switch motivated by the need of using polling to restore real time output from ssh server). This is the only pipe that needs to be wrapped because this is the one who do extensive usage of 'readline'. The stderr pipe of the process is alway read in non blocking raw chunk, so it won't benefit from the buffering.

util: introduce a bufferedinputpipe utility To restore real time server output through ssh, we need to using polling feature (like select) on the pipes used to communicate with the ssh client. However we cannot use select alongside python level buffering of these pipe (because we need to know if the buffer is non-empty before calling select). However, unbuffered performance are terrible, presumably because the 'readline' call is issuing 'read(1)' call until it find a '\n'. To work around that we introduces our own overlay that do buffering by hand, exposing the state of the buffer to the outside world. The usage of polling IO will be introduced later in the 'sshpeer' module. All its logic will be very specific to the way mercurial communicate over ssh and does not belong to the generic 'util' module.

test: drop all the forced use of bundle2 Using bundle2 during exchange is now the default, we do not need all this explicit enabling of bundle2 exchange in test anymore.

bundle2: use bundle2 by default All the test change have been isolated and validated. We have free to turn on bundle2 as the default exchange protocol. "To reach a port we must set sail – Sail, not tie at anchor Sail, not drift."

wireprotocol: distinguish list and set in getbundle argument The 'bundlecaps' argument is expected to be a set, but 'listkeys' is expected to be a list where ordering matters. We introduce a new 'scsv' argument type for the 'set' version and move 'csv' to the 'list' version. 'test-ssh.t' is changed because this introduced an instability in the order we were producing listkeys parts.

exchange: support transferring .hgtags fnodes mapping On Mozilla's mozilla-beta repository .hgtags fnodes resolution takes ~18s from a clean cache on my machine. This means that the first time a user runs `hg tags`, `hg log`, or any other command that displays or accesses tags data, a ~18s pause will occur. There is no output during this pause. This results in a poor user experience and perception that Mercurial is slow. The .hgtags changeset to filenode mapping is deterministic. This patch takes advantage of that property by implementing support for transferring .hgtags filenodes mappings in a dedicated bundle2 part. When a client advertising support for the "hgtagsfnodes" capability requests a bundle, a mapping of changesets to .hgtags filenodes will be sent to the client. Only mappings of head changesets included in the bundle will be sent. The transfer of this mapping effectively eliminates one time tags cache related pauses after initial clone. The mappings are sent as binary data. So, 40 bytes per pair of SHA-1s. On the aforementioned mozilla-beta repository, 659 * 40 = 26,360 raw bytes of mappings are sent over the wire (in addition to the bundle part headers). Assuming 18s to populate the cache, we only need to transfer this extra data faster than 1.5 KB/s for overall clone + tags cache population time to be shorter. Put into perspective, the mozilla-beta repository is ~1 GB in size. So, this additional data constitutes <0.01% of the cloned data. The marginal overhead for a multi-second performance win on clones in my opinion justifies an on-by-default behavior.

bundle2: part handler for processing .hgtags fnodes mappings .hgtags fnodes cache entries can be expensive to compute, especially if there are hundreds of even thousands of them. This patch implements support for receiving a bundle2 part that contains a mapping of changeset to .hgtags fnodes. An upcoming patch will teach the server to send this part, allowing clients to bypass having to redundantly compute these values. A number of tests changed due to the client advertising the "hgtagsfnodes" capability.

changegroup: rename _computeoutgoing to computeoutgoing We're going to use this function from another module in an upcoming patch. Drop the _ prefix to mark it as non-private.

merge with stable

test-treemanifest: add globs for Windows

test-import: sync with 6084926366b9 for platforms without execbit

hgwebdir: avoid redundant repo and directory entries when 'web.name' is set Previously, when 'web.name' was set on a subrepo parent and 'web.collapse=True', the parent repo would show in the list with the configured 'web.name', and a directory with the parent repo's filesystem name (with a trailing slash) would also appear. The subrepo(s) would unexpectedly be excluded from the list of repositories. Clicking the directory entry would go right to the repo page. Now both the parent and the subrepos show up, without the additional directory entry. The configured hgweb paths used '**' for finding the repos in this scenario. A couple of notes about the tests: - The area where the subrepo was added has a comment that it tests subrepos, though none previously existed there. One now does. - The 'web.descend' option is required for collapse to work. I'm not sure what the previous expectations were for the test. Nothing changed with it set, prior to adding the code in this patch. It is however required for this test. - The only output changes are for the hyperlinks, obviously because of the 'web.name' parameter. - Without this code change, there would be an additional diff: --- /usr/local/mercurial/tests/test-hgwebdir.t +++ /usr/local/mercurial/tests/test-hgwebdir.t.err @@ -951,7 +951,7 @@ /rcoll/notrepo/e/ /rcoll/notrepo/e/e2/ /rcoll/notrepo/f/ - /rcoll/notrepo/f/f2/ + /rcoll/notrepo/f/ Test repositories inside intermediate directories I'm not sure why the fancy name doesn't come out, but it is enough to demonstrate that the parent is not listed redundantly, and the subrepo isn't skipped.

Added signature for changeset ed18f4acf435

Added tag 3.4.1 for changeset ed18f4acf435

merge with stable

templatekw: compare target context and its parent exactly (issue4690) Before this patch, template keywords `{file_mods}`, `{file_adds}` and `{file_dels}` use values gotten by `repo.status(ctx.p1().node(), ctx.node())`. But this doesn't work as expected if `ctx` is `memctx` or `workingcommitctx`. Typical case of templating with these contexts is customization of the text shown in the commit message editor by `[committemplate]` configuration. In this case, `ctx.node()` returns None and it causes comparison between `ctx.p1()` and `workingctx`. `workingctx` lists up all changed files in the working directory even at selective committing. BTW, `{files}` uses `ctx.files()` and it works as expected. To compare target context and its parent exactly, this patch passes `ctx.p1()` and `ctx` without `node()`-nize. This avoids unexpected comparison with `workingctx`. This patch uses a little redundant template configurations in `test-commit.t`, but they are needed to avoid regression around problems fixed by a4958cdb2202 and 1e6fb8db666e: accessing on `ctx` may break `ctx._status` field.

test: use bundle2 in test-pull-http This test has a minor output change when using bundle2. We make this change early to reduce the noise in the final change.

test: use bundle2 in test-push-http Now that we have a bundle1 version of this test, we can move the original to bundle2. This lets us handle the ouput change from using the bundle2 protocol earlier.

test: copy test-push-http.t to testpush-http-bundle1.t We want to keep both code paths tested. The test is a bit too extensive to simply introduce dual testing in it, so we make a copy for each protocol version.

tests: add (?) flag for optional lines When the test engine fails to match output on a line marked with (?), it will simply continue to the next expected line and try again. This allows simplifying tests that have either version-specific or non-fixed behavior, for instance: $ coin-flip heads (?) tails (?) (There's no form of back-tracking attempted, so optional matches should be specific.)

templatekw: display active bookmark more consistently (issue4552) (BC) Previously, the template keyword '{activebookmark}' would only display the active bookmark if it was also pointing to the working directory's parent. Meanwhile, the '{active}' subkeyword of the '{bookmarks}' keyword displays the active bookmark regardless of whether it also points to the working directory's parent. This is confusing. Consider the output of these two templates: $ hg log -T '{activebookmark}\n' -r indent $ hg log -T '{bookmarks % "{bookmark}"}\n' -r indent indent This is the current behavior that can arise after, eg, a pull moves a bookmark out from under you. After this patch, the first template will also return the active bookmark that points to a revision, even if it is not the current parent of the working directory. A test has been added to show the new behavior.

revrange: build spanset from x:y range This slightly improves the performance in the optimal case: % hg log -R mozilla-central -r0:tip -l1 --time (before) time: real 0.050 secs (user 0.040+0.000 sys 0.010+0.000) (after) time: real 0.020 secs (user 0.000+0.000 sys 0.010+0.000)

revrange: build balanced tree of addsets from revisions (issue4565) This reduces the stack depth from O(n) to O(log(n)). Therefore, repeated -rREV options will never exceed the Python stack limit. Currently it depends on private revset._combinesets() function. But at some point, we'll be able to drop the old-style parser, and revrange() can be completely rewritten without using _combinesets(): trees = [parse(s) for s in revs] optimize(('or',) + trees) # combine trees and optimize at once ... Blockers that prevent eliminating old-style parser: - nullary ":" operator - revrange(repo, [intrev, ...]), can be mapped to 'rev(%d)' ? - revrange(repo, [binnode, ...]), should be banned ?

revrange: clean up meaningless reconstruction of sets They just exist for deduplication that was removed by the previous patch.

revrange: drop unnecessary deduplication of revisions Because "l" is a smartset, duplicated entries are omitted by addsets.

summary: move the parents phase marker to commit line (issue4688) The phase of the pending commit depends on the parent of the working directory and on the phases.newcommit configuration. First, this information rather depend on the commit line which describe the pending commit. Then, we only want to be advertised when the pending phase is going to be higher than the default new commit phase. So the format will change from $ hg summary parent: 2:ab91dfabc5ad foo parent: 3:24f1031ad244 tip bar branch: default commit: 1 modified, 1 unknown, 1 unresolved (merge) update: (current) phases: 1 secret (secret) to parent: 2:ab91dfabc5ad foo parent: 3:24f1031ad244 tip bar branch: default commit: 1 modified, 1 unknown, 1 unresolved (merge) (secret) update: (current) phases: 1 secret

tags: support setting hgtags fnodes cache entries An upcoming patch will teach bundle2 to transfer .hgtags fnodes values. To support this, we need to support inserting values into the cache. Add functionality to do that.

tags: support reading tags cache without populating An upcoming patch will teach the bundle2 protocol to transfer .hgtags fnodes to the client. We don't want this to incur any extra work at serve time. Create an optional cache query mode that doesn't populate the cache as a side-effect.

check-commit: make foo_bar naming regexp less greedy \s is equivalent to the character class [ \t\n\r\f\v]. Using \s+ in a regular expression against input with multiple lines may match across multiple lines. For the regexp in question, "\+\s+" would match "+\n " and similar sequences, leading to false positives for functions that were included in diff context, after a modified hunk.

test: use bundle2 in test-http Now that we have a bundle1 version of this test, we can move the original to bundle2. This lets us handle the ouput change from using the bundle2 protocol earlier.

test: copy test-http.t to test-http-bundle1.t We want to keep both code paths tested. The test is a bit too extensive to simply introduce dual testing in it so we make a copy for each protocol version.

test: use bundle2 in test-ssh Now that we have a bundle1 version of this test, we can move the main version to bundle2. This lets us handle the ouput change from using the bundle2 protocol earlier.

test: copy test-ssh.t to test-ssh-bundle1.t We want to keep both code paths tested. The test is a bit too extensive to simply introduce dual testing in it so we make a copy for each protocol version.

test: lock test-unbundlehash to bundle1 usage It is testing a bundle1 specific behavior. Bundle2 has its own way there. See inline comment for details.

test: use bundle2 in test-acl This test makes extensive use of --debug so moving to bundle2 based exchange has a massive impact. We do it early to reduce the noise create by a future usage of bundle2 as the default protocol.

test: use both bundle formats in test-pull-http It is valuable to have both formats tested.

test: use bundle2 in test-http-proxy The proxy test does not care about what protocol is used, but the new protocol implies different traffic (and therefore different log output). We switch it to bundle2 early to minimise the noise of using bundle2 for exchange by default.

tests: use bundle2 for test-hook Using bundle2 has an effect on which hooks are run when. We turn it on for test-hooks early to reduce the noise of switching the default exchange to bundle2.

pull: only prefetch bookmarks when using bundle1 All bundle2 servers now support the 'listkeys' part(1), so we'll always be able to fetch bookmarks data at the same time as the changeset. This should be enough to avoid the one race condition that this bookmark prefetching is trying to work around. It even allows future server to make sure everything is generated from the same "transaction" if they become capable of such. The current code was already overwriting the prefetched value with the one in bundle2 anyway. Note that this is not preventing all race conditions in related to bookmark in 'hg pull' it makes nothing better and nothing worse. Reducing the number of listkeys calls will reduce the latency on pull. The pre-fetch is also moved into a discovery step because it seems to belong there. (1) Because all servers not speaking 'pushkey' parts are compatible with the 'HG2X' protocol only.

pull: document the race condition with bookmark name It seems valuable to document this in-place to help the next poor soul looking at this code to understand what kind of beast he is currently facing.

pull: only list remote bookmarks if -B is used to populate pulled heads Listing remote bookmarks results in network traffic and latency. This should be avoided when possible.

i18n-ja: synchronized with 8594d0b3018e

hg: explicitly check that peer lookup object has instance() if call failed If a "thing" is callable but raises TypeError for some reason, a callable object would be returned. Thereafter, unfriendly traceback would be displayed: Traceback (most recent call last): ... File "mercurial/hg.pyc", line 119, in _peerorrepo obj = _peerlookup(path).instance(ui, path, create) AttributeError: 'function' object has no attribute 'instance' Instead, we should show the reason why "thing(path)" didn't work: Traceback (most recent call last): ... File "hggit/__init__.py", line 89, in _local p = urlcls(path).localpath() TypeError: 'NoneType' object is not callable If a "thing" is not callable, it must be a module or an object that implements instance(). If that module didn't have instance(), the error message would be "<unloaded module 'foo'> object is not callable". It doesn't make perfect sense, but it isn't so bad as it can blame which module went wrong.

extensions: show traceback on load failure if --traceback flag is set Before this patch, there was no handy way to investigate the reason why extension couldn't be loaded. If ui.debug is set, tracebacks of both "hgext.foo" and "foo" are displayed because the first ImportError could occur at very deep dependency module.

ui: flush stdout before writing traceback to stderr Sometimes a traceback message is paired with ui.debug(). This patch makes sure that these messages are displayed in the right order.

treemanifest: visit directory 'foo' when given e.g. '-X foo/ba?' For globs like 'foo/ba?', match._roots() will return 'foo'. Since visitdir(), excludes directories in the excluded roots, it would skip the entire foo directory. This is incorrect, since 'foo/ba?' doesn't mean that everything in foo/ should be exluded. Note that visitdir() is called only from the treemanifest class, so this only affects tree manifests. Fix by adding roots to the set of excluded roots only if there are no excluded patterns. Since 'glob' is the default pattern type for globs, we also need to update some -X patterns in the tests to be of 'path' type to take advantage of the visitdir tricks. For consistency, also update the -I patterns. It seems a little unfortunate that 'foo' in 'hg files -X foo' is considered a pattern because of the implied 'glob' type, but improving that is left for another day.

phases: fix bug where native phase computation wasn't called I forgot to include this change as a previous diff and the native code to compute the phases was never called. The AttributeError was silently caught and the pure implementation was used instead.

merge with stable

record: add default value for operation argument This patch is part of a series of patches to change the recording ui to reflect the operation currently running (commit, shelve, revert ...). This patch adds the default value of the operation argument for record's standard and curses interface to match what is displayed in the interface as of today.

record: precise documentation This patch improves the documentation of the recordfilter function to explain that we need a translated string for the 'operation' argument.