Added signature for changeset 77eaf9539499

Added tag 4.1.3 for changeset 77eaf9539499

dispatch: protect against malicious 'hg serve --stdio' invocations (sec) Some shared-ssh installations assume that 'hg serve --stdio' is a safe command to run for minimally trusted users. Unfortunately, the messy implementation of argument parsing here meant that trying to access a repo named '--debugger' would give the user a pdb prompt, thereby sidestepping any hoped-for sandboxing. Serving repositories over HTTP(S) is unaffected. We're not currently hardening any subcommands other than 'serve'. If your service exposes other commands to users with arbitrary repository names, it is imperative that you defend against repository names of '--debugger' and anything starting with '--config'. The read-only mode of hg-ssh stopped working because it provided its hook configuration to "hg serve --stdio" via --config parameter. This is banned for security reasons now. This patch switches it to directly call ui.setconfig(). If your custom hosting infrastructure relies on passing --config to "hg serve --stdio", you'll need to find a different way to get that configuration into Mercurial, either by using ui.setconfig() as hg-ssh does in this patch, or by placing an hgrc file someplace where Mercurial will read it. mitrandir@fb.com provided some extra fixes for the dispatch code and for hg-ssh in places that I overlooked.

progress: retry ferr.flush() and .write() on EINTR (issue5532) See the inline comment how this could mitigate the issue. I couldn't reproduce the exact problem on my Linux machine, but there are at least two people who got EINTR in progress.py, and it seems file_write() of Python 2 is fundamentally broken [1]. Let's make something in on 4.2. [1]: https://hg.python.org/cpython/file/v2.7.13/Objects/fileobject.c#l1850

progress: extract stubs to restart ferr.flush() and .write() on EINTR

filemerge: optionally strip quotes from merge marker template (BC) For consistency with the other template options. Quotes are necessary if you want to preserve leading/trailing whitespace, which would be stripped by config parser.

commit: optionally strip quotes from commit template (BC) For consistency with the other template options.

graphlog: optionally strip quotes from graphnode template (BC) For consistency with the other template options.

dispatch: ignore further SIGPIPE while handling KeyboardInterrupt I got the following error by running "hg log" and quitting the pager immediately. Any output here may trigger another SIGPIPE, so only thing we can do is to swallow the exception and exit with an error status. Traceback (most recent call last): File "./hg", line 45, in <module> mercurial.dispatch.run() File "mercurial/dispatch.py", line 83, in run status = (dispatch(req) or 0) & 255 File "mercurial/dispatch.py", line 167, in dispatch req.ui.warn(_("interrupted!\n")) File "mercurial/ui.py", line 1224, in warn self.write_err(*msg, **opts) File "mercurial/ui.py", line 790, in write_err self._write_err(*msgs, **opts) File "mercurial/ui.py", line 798, in _write_err self.ferr.write(a) File "mercurial/ui.py", line 129, in _catchterm raise error.SignalInterrupt mercurial.error.SignalInterrupt Perhaps this wasn't visible before de5c9d0e02ea because the original stderr handle was restored very late.

worker: print traceback for uncaught exception unconditionally This is what a Python interpreter would do if there were no os._exit().

worker: propagate exit code to main process Follows up 86cd09bc13ba.

dispatch: print traceback in scmutil.callcatch() if --traceback specified Otherwise, traceback wouldn't be printed for a known exception occurred in worker processes.

dispatch: mark callcatch() as a private function

templatefilters: fix crash by string formatting of '{x|splitlines}' Before, it crashed because mapping['templ'] was missing. As it didn't support the legacy list template from the beginning, we can simply use hybridlist().

templatekw: factor out showdict() helper Make it less cryptic for common cases.

templatekw: have showlist() take mapping dict with no **kwargs expansion (API) See the previous commit for why. splitlines() does not pass a mapping dict, which would probably mean the legacy template didn't work from the beginning.

templatekw: change _showlist() to take mapping dict with no **kwargs expansion There was a risk that a template keyword could conflict with an argument name (e.g. 'name', 'values', 'plural', etc.) Let's make it less magical.

templatekw: rename 'args' to 'mapping' in showlist() The name 'args' provides no information. Call it 'mapping' as in templater.py.

templatekw: eliminate unnecessary temporary variable 'names' from _showlist() Replace 'names' with the optional argument 'plural'.

color: update the help with the new default The default is now "auto" we update the help to match reality.

upgrade: register all format variants in a list Now that all known format variants exists outside of the function, we can gather them in a lists. This build a single entry point other code can use (current target: extensions). The repository upgrade code is updated to simply use entries from this list. As a side effect this will also allow extensions to register their own format variants, to do this "properly" we should introduce a "registrar" for this category of object. However I prefer to keep this series simple, and that will be adventure for future time.

upgrade: move descriptions and selection logic in individual classes Our goal here is to get top level definition for all the format variants. Having them defined outside of the function enabled other users of that logic. They are two keys components of a format variant: 1) the name and various descriptions of its effect, 2) the code that checks if the repo is using this variant and if the config enables it. That second items make us pick a class-based approach, since different variants requires different code (even if in practice, many can reuse the same logic). Each variants define its own class that is then used like a singleton. The class-based approach also clarify the definitions part a bit since each are simple assignment in an indented block. The 'fromdefault' and 'fromconfig' are respectively replaced by a class attribute and a method to be called at the one place where "fromconfig" matters. Overall, they are many viable approach for this, but this is the one I picked.

upgrade: introduce a 'formatvariant' class The 'deficiency' type has multiple specificities. We create a dedicated class to host them. More logic will be added incrementally in future changesets.

upgrade: implement '__hash__' on 'improvement' class The pythonomicon request its implementation.

upgrade: implement '__ne__' on 'improvement' class The pythonomicon request its implementation.

color: also enable by default on windows I've not found anything related to color + windows on the bug tracker. So I'm suggesting we get bolder and turn it on for windows too in the release candidate. We can always backout this changeset if we find serious issue on windows.

color: turn on by default (but for windows) Color support is all in core for a couple of months. I've browsed the bug tracker without finding any blocker bug. So I'm moving forward and enable color on by default before '4.2-rc'. In the worse case, having it on in the release candidate will help us to find blocker bug and we can turn it off for the final release. I remember people talking about issue with Windows during the freeze so I'm keeping it off by default on that OS. We could do various cleaning of the color used and the label issued. However the label are probably already in our backward compatibility envelope since the color extensions has been around since for ever and I do not think the color choice themself should be considered BC. So I think we should rather gives color to all user sooner than later. A couple of test needs to be updated to avoid having color related control code spoil the tested output.

pager: stop using the color extension in tests The feature is in core so let us use the core config knob directly.

bundle2: ignore errors seeking a bundle after an exception (issue4784) Many have seen a "stream ended unexpectedly" error. This message is raised from changegroup.readexactly() when a read(n) operation fails to return exactly N bytes. I believe most occurrences of this error in the wild stem from the code changed in this patch. Before, if bundle2's part applicator raised an Exception when processing/applying parts, the exception handler would attempt to iterate the remaining parts. If I/O during this iteration failed, it would likely raise the "stream ended unexpectedly" exception. The problem with this approach is that if we already encountered an I/O error iterating the bundle2 data during application, then any further I/O would almost certainly fail. If the original stream were closed, changegroup.readexactly() would obtain an empty string, triggering "stream ended unexpectedly" with "got 0." This is the error message that users would see. What's worse is that the original I/O related exception would be lost since a new exception would be raised. This made debugging the actual I/O failure effectively impossible. This patch changes the exception handler for bundle2 application to ignore errors when seeking the underlying stream. When the underlying error is I/O related, the seek should fail fast and the original exception will be re-raised. The output changes in test-http-bad-server.t demonstrate this. When the underlying error is not I/O related and the stream can be seeked, the old behavior is preserved.

error: rename RichIOError to PeerTransportError This is a more descriptive name. RichIOError was introduced just hours ago, so it doesn't need to be marked as BC.

httppeer: don't send empty Vary request header As part of writing test-http-bad-server.t, I noticed that some requests include an empty Vary HTTP request header. The Vary HTTP request header indicates which headers should be taken into account when determining if a cached response can be used. It also accepts the special value of "*". The previous code unconditionally added a Vary header. This could lead to an empty header value. While I don't believe this violates the HTTP spec, this is weird and just wastes bytes. So this patch changes behavior to only send a Vary header when it has a value. Some low-level wire protocol byte reporting tests changed. In some cases, the exact point of data termination changed. However, the behavior being tested - that clients react when the connection is closed in the middle of an HTTP request line or header - remains unchanged.

badserverext: explicitly flush each log write No idea why, but without this, there was no content in the error logs that are `cat`d in the test on Windows.

test-http-bad-server: conditionalize error output for Windows

test-tag: make hook runnable on Windows Simply setting the hook to *.sh causes Windows to open a dialog asking how to open *.sh files.

obsolescence: add test for the "branch replacement" logic during push, case D7 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case D5 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case D3 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case B8 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case B7 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case B6 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case B5 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case B4 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

obsolescence: add test for the "branch replacement" logic during push, case B2 Mercurial checks for the introduction of new heads on push. Evolution comes into play to detect if existing branches on the server are being replaced by some of the new one we push. This changeset adds test for the improved "branch replacement" logic introduce in an earlier commits. This tests initially lived in the evolve extensions. Since we now have the code handling this logic in core, it make sense to have the tests in core too. See inline documentation for details about the test case added in this changeset.

checkheads: upgrade the obsolescence postprocessing logic (issue4354) The previous logic had many shortcoming (eg: looking at the head only, not handling prune, etc...), the new logic use a more robust approach: For each head, we check if after the push all changesets exclusive to this heads will be obsolete. If they are, the branch considered be "replaced". To check if a changeset will be obsolete, we simply checks: * the changeset phase * the existence of a marker relevant to the "pushed set" that affects the changesets.. This fixes two major issues of the previous algorithm: * branch partially rewritten (eg: head but not root) are no longer detected as replaced, * Prune are now properly handled. (This implementation was introduction in the evolve extension, version 6.0.0.) This new algorithm has an extended number of tests, a basic one is provided in this patch. The others will be introduced in their own changeset for clarity. In addition, we stop trying to process heads unknown locally, we do not have enough data to take an informed decision so we should stop pretending we do. This reflect a test that is now update.

test-http: add a (glob) for Windows I'm not sure why the glob was ineffective with $TESTTMP here, but replacing it with '*' works.

testlib: move the prune alias into a shell script for Windows Cramming all of this directly into an alias doesn't play nicely on Windows. Various test-exchange-obsmarkers-case-XX.t were failing with: abort: cannot select revision when creating marker It turned out that inside debugcommands.debugobsolete(), the following differed from Linux (where they were empty, at least in the case I debugged): 'rev': ['.`;'], 'template': "'{node}\\n'"

hidden: extract the code generating "filtered rev" error for wrapping The goal is to help experimentation in extensions (ie: evolve) around more advance messages.

serve: add support for Mercurial subrepositories I've been using `hg serve --web-conf ...` with a simple '/=projects/**' [paths] configuration for awhile without issue. Let's ditch the need for the manual configuration in this case, and limit the repos served to the actual subrepos. This doesn't attempt to handle the case where a new subrepo appears while the server is running. That could probably be handled with a hook if somebody wants it. But it's such a rare case, it probably doesn't matter for the temporary serves. The main repo is served at '/', just like a repository without subrepos. I'm not sure why the duplicate 'adding ...' lines appear on Linux. They don't appear on Windows (see 594dd384803c), so they are optional. Subrepositories that are configured with '../path' or absolute paths are not cloneable from the server. (They aren't cloneable locally either, unless they also exist at their configured source, perhaps via the share extension.) They are still served, so that they can be browsed, or cloned individually. If we care about that cloning someday, we can probably just add the extra entries to the webconf dictionary. Even if the entries use '../' to escape the root, only the related subrepositories would end up in the dictionary.

hgwebdir: allow a repository to be hosted at "/" This can be useful in general, but will also be useful for hosting subrepos, with the main repo at /.

httppeer: eliminate decompressresponse() proxy Now that the response instance itself is wrapped with error handling, we no longer need this code. This code became dead with the previous patch because the added code catches HTTPException and re-raises as something else.

httppeer: wrap HTTPResponse.read() globally There were a handful of places in the code where HTTPResponse.read() was called with no explicit error handling or with inconsistent error handling. In order to eliminate this class of bug, we globally swap out HTTPResponse.read() with a unified error handler. I initially attempted to fix all call sites. However, after going down that rabbit hole, I figured it was best to just change read() to do what we want. This appears to be a worthwhile change, as the tests demonstrate many of our uncaught exceptions go away. To better represent this class of failure, we introduce a new error type. The main benefit over IOError is it can hold a hint. I'm receptive to tweaking its name or inheritance.

tests: add tests for poorly behaving HTTP server I've spent several hours over the past few weeks investigating networking failures involving hg.mozilla.org. As part of this, it has become clear that the Mercurial client's error handling when it encounters network failures is far from robust. To prove this is true, I've devised a battery of tests simulating various network failures, notably premature connection closes. To achieve this, I've implemented an extension that monkeypatches the built-in HTTP server and hooks in at the socket level and allows various events to occur based on config options. For example, you can refuse to accept() a client socket or you can close() the socket after N bytes have been sent or received. The latter effectively simulates an unexpected connection drop (and these occur all the time in the real world). The new test file launches servers exhibiting various "bad" behaviors and points a client at them. As the many TODO comments in the test call attention to, Mercurial often displays unhelpful errors when network-related failures occur. This makes it difficult for users to understand what's going on and difficult for server administrators to pinpoint root causes without packet tracing. Upcoming patches will attempt to fix these error handling deficiencies.

phases: emit phases to pushkey protocol in deterministic order An upcoming test will report exact bytes sent over the wire protocol. Without this change, the ordering of phases listkey data is non-deterministic.

keepalive: send HTTP request headers in a deterministic order An upcoming patch will add low-level testing of the bytes being sent over the wire. As part of developing that test, I discovered that the order of headers in HTTP requests wasn't deterministic. This patch makes the order deterministic to make things easier to test.

revset: properly parse "descend" argument of followlines() We parse "descend" symbol as a Boolean using getboolean (prior extraction by getargsdict already checked that it is a symbol). In tests, check for error cases and vary Boolean values here and there.

revsetlang: add a getboolean helper function This will be used to parse followlines's "descend" argument.

track-tags: write all tag changes to a file The tag changes information we compute is now written to disk. This gives hooks full access to that data. The format picked for that file uses a 2 characters prefix for the action: -R: tag removed +A: tag added -M: tag moved (old value) +M: tag moved (new value) This format allows hooks to easily select the line that matters to them without having to post process the file too much. Here is a couple of examples: * to select all newly tagged changeset, match "^+", * to detect tag move, match "^.M", * to detect tag deletion, match "-R". Once again we rely on the fact the tag tests run through all possible situations to test this change.

track-tags: compute the actual differences between tags pre/post transaction We now compute the proper actuall differences between tags before and after the transaction. This catch a couple of false positives in the tests. The compute the full difference since we are about to make this data available to hooks in the next changeset.

track-tags: introduce first bits of tags tracking during transaction This changeset introduces detection of tags changes during transaction. When this happens a 'tag_moved=1' argument is set for hooks, similar to what we do for bookmarks and phases. This code is disabled by default as there are still various performance concerns. Some require a smarter use of our existing tag caches and some other require rework around the transaction logic to skip execution when unneeded. These performance improvements have been delayed, I would like to be able to experiment and stabilize the feature behavior first. Later changesets will push the concept further and provide a way for hooks to know what are the actual changes introduced by the transaction. Similar work is needed for the other families of changes (bookmark, phase, obsolescence, etc). Upgrade of the transaction logic will likely be performed at the same time. The current code can report some false positive when .hgtags file changes but resulting tags are unchanged. This will be fixed in the next changeset. For testing, we simply globally enable a hook in the tag test as all the possible tag update cases should exist there. A couple of them show the false positive mentioned above. See in code documentation for more details.

tags: introduce a function to return a valid fnodes list from revs This will get used to compare tags between two set of revisions during a transaction (pre and post heads). The end goal is to be able to track tags movement in transaction hooks.