rebase: inline _makextrafn() now that we have only one caller Also avoid even creating a function since we just end up calling it right away. Differential Revision: https://phab.mercurial-scm.org/D2918

rebase: pass in "extra" itself into conclude[memory]node() We were passing in a function instead for no clear reason (probably historical, but I haven't bothered looking). Differential Revision: https://phab.mercurial-scm.org/D2917

rebase: look up commit message to reuse outside of conclude[memory]node() This was done the same way in both functions, so let's let the single caller do it. Differential Revision: https://phab.mercurial-scm.org/D2916

rebase: pass in ctx, not rev, to conclude[memory]node() They both need it and there's no locking that might make the results different, so let's do it in one place. This also lets us move out more common code in the following patches. Differential Revision: https://phab.mercurial-scm.org/D2915

rebase: extract common _concludenode() _rebasenode() and _finishrebase() had a lot of code in common. This commit extracts some of that. This means we will also have a single caller of concludenode() and concludememorynode(), which gives us a place to put code that is common between those two functions (which is quite a bit). Differential Revision: https://phab.mercurial-scm.org/D2914

rebase: store rebase state after each commit Before this patch, we stored the rebase state early in the processing of a node, before we updated the rebase state to indicate that the node was processed. This meant that we could redo the working copy merge and run into conflicts. However, this only happened in the --collapse case if the rebase was interrupted while editing the final commit message; in the case earlier interruptions, we would instead detect the in-process revision by finding two dirstate parents. This patch moves the writing of the rebase state to after we have completed the revision completely, and, importantly, after we have updated the rebase state to mark it done. This means we'll realize that all nodes have been rebased in the case mentioned above of editing the final commit message of a --collapse. See change to test case. I also moved the writing outside of the large if/elif block in _rebasenode(). This shouldn't matter much, but seems cleaner. One observable effect is if rebase was interrupted just after ignoring an obsolete node ("not rebasing ####, already in destination"), we used to come up with the same decision after --continue too, but after this patch we'll instead say "already rebased ###". This seems more consistent, since that's what we would do with obsolete nodes that had been marked done earlier in the process (not only just before the interruption). Differential Revision: https://phab.mercurial-scm.org/D2913

rebase: register status file generator only once when using single transaction rebase.storestatus() behaved differently depending on whether a transaction is passed to it. If a transaction is passed, it registers a "file generator" that runs when the transaction commits. If no transaction was passed, it writes the rebase state immediately. This imprecise timing of the writing makes it hard to reason about, so let's make it more explicit which behavior we're getting by checking if we have a transaction before calling it. For the single-transaction case, move the call to storestatus(tr) early and do it only once since it's only going to write the file (at most) once anyway. Differential Revision: https://phab.mercurial-scm.org/D2912

tests: add some more tests to test-rebase-collapse.t Some of these are currently broken. Differential Revision: https://phab.mercurial-scm.org/D2911

tests: use drawdag in test-rebase-collapse.t This makes the tests shorter (410 lines -> 240 lines), faster (17s -> 12s), and easier to read (the setup for each test case is just before the test, and we can refer to commits by name). Differential Revision: https://phab.mercurial-scm.org/D2910

rebase: remove unused default argument values from conclude[memory]node() It's only "commitmsg" that we don't always pass. Differential Revision: https://phab.mercurial-scm.org/D2909

rebase: get "inmemory" state directly from rebase runtime As far as I can tell, rbsrt.inmemory is equivalent to rbsrt.wctx.isinmemory(), so let's use the shorter form. Differential Revision: https://phab.mercurial-scm.org/D2908

commands: use keyword arguments in update function This will help us in having a dictionary with the values of all the arguments and we can add more flags without adding an argument to the function. Differential Revision: https://phab.mercurial-scm.org/D2896

rebase: pass "inmemory" directly to _definedestmap() We no longer reassign rbsrt.inmemory in _definedestmap(), so we don't need to pass the whole rebase runtime instance anymore, thus making it clear that it won't be updated. Differential Revision: https://phab.mercurial-scm.org/D2905

hgweb: explain instabilities of unstable changesets (the rest of the themes)

test-merge-tools: stabilize for Windows See fe5c4b795999.

test-http-protocol: drop an extraneous glob for Windows

merge with stable

hgweb: evaluate the "default" value as template Strictly speaking, everything in the map file is a template. So let's not take out an unparsed template string.

templater: use named function to expand template against mapping dict (API) And replace __call__(t, **mapping) in favor of generate(t, mapping). I prefer a named function here since the templater isn't a simple function-like object. .. api:: The templater is no longer callable. Use ``templater.generate(t, mapping)`` instead of ``templater(t, **pycompat.strkwargs(mapping))``.

highlight: peek Content-Type header set by hgweb There should be no need to re-render the mimetype template since it's set before dispatching webcommands.

obsolete: add a note that explains creating aliases for marker flags

repair: rename _backup to backupbundle A future commit will introduce a caller from outside the module. The function should have already been public because histedit was calling it. Differential Revision: https://phab.mercurial-scm.org/D2666

templatefuncs: do not stringify result of if*() expression Returning a generator means that the result is a byte string. I can't find any reason to make the "if" condition lazy since it is evaluated anyway when {if()} has to be evaluated. So let's simply make if*() return an input expression unmodified.

templater: add brief doc about internal data types Hopefully this will help reviewing upcoming my series. Start small.

debugcommands: support sending HTTP requests with debugwireproto We implement an action that can issue an HTTP request. We can define headers via arguments and specify a file to use for the HTTP request body. The request uses the HTTP peer's opener, which is already configured for auth, etc. This is both good and bad. Good in that we get some nice behavior out of the box. Bad in that some HTTP request headers are added automatically. Differential Revision: https://phab.mercurial-scm.org/D2841

debugcommands: support connecting to HTTP peers Now that we have the plumbing for logging socket activity, let's hook it up to `hg debugwireproto` so we can collect low-level activity on sockets. The new code is a bit incomplete. But it is better than nothing: `hg debugwireproto` is still heavily evolving. The added test demonstrates some interesting behavior. For example, we're calling socket.makefile() and doing I/O on that. TIL. We're also sending an "Accept-Encoding: identity" request header. Differential Revision: https://phab.mercurial-scm.org/D2726

url: add HTTP handler that uses a proxied socket Now that we have a socket proxy that can log I/O, we need to teach mechanisms that open URLs how to use it. We invent a custom HTTP handler class that knows how to proxy sockets as soon as they are opened. We teach the high-level opener() to accept logging arguments so a logging HTTP handler can be constructed. We don't yet support intercepting HTTPS sockets because I don't want to go down that rabbit hole. For the record, the urllib API is crazy and it took way too long to figure out at what levels I needed to plug in to modify the socket. # no-check-commit because we must name http_open that way Differential Revision: https://phab.mercurial-scm.org/D2722

util: observable proxy objects for sockets We previously introduced proxy objects and observers for file objects to help implement low-level tests for the SSH wire protocol. In this commit, we do the same for sockets in order to help test the HTTP server. We only proxy/observe some socket methods. I didn't feel like implementing all the methods because there are so many of them and implementing them will provide no short term value. We can always implement them later. # no-check-commit because we implement foo_bar methods on stdlib types Differential Revision: https://phab.mercurial-scm.org/D2721

hgweb: allow defining Server response header for HTTP server By default, the string contains the Python version. Server operators may want to change this. Because we like deterministic tests, we change the test harness to always set this value to a known string. Various globs over the server header have now been removed. # no-check-commit because we add version_string() Differential Revision: https://phab.mercurial-scm.org/D2840

tests: use $HTTP_DATE$ for Date header Support for the $HTTP_DATE$ substitution was recently added. Let's adopt it more widely. We had to tweak the substitution to be case insensitive, since HTTP headers are case insensitive. I also found a minor test issue not globbing over the length of the Server response header. Differential Revision: https://phab.mercurial-scm.org/D2839

debugcommands: introduce actions to perform deterministic reads "readavailable" is useful as a debugging device to see what data is available on a pipe. But the mechanism isn't deterministic because what's available on a pipe is highly conditional on timing, system load, OS behavior, etc. This makes it not suitable for tests. We introduce "ereadline," "read," and "eread" for performing deterministic I/O operations (at least on blocking file descriptors). We stop short of converting existing consumers of "readavailable" in tests because we're working out race conditions and deadlocks on Windows. But the goal is to eventually move tests away from "readavailable" to these new APIs. Differential Revision: https://phab.mercurial-scm.org/D2720

httppeer: refactor how httppeer is created (API) Previously, we passed a bunch of arguments to httppeer.__init__, validated them, then possibly constructed a valid instance. A short while ago, we refactored sshpeer so all the validation and setup work occurs before the constructor. We introduced a makepeer() to hold most of this logic. This commit gives httppeer the same treatment. As a sign that the previous design was poor, __del__ no longer conditionally checks for the presence of an attribute that may not be defined (it is always defined in the new code). .. api:: httppeer.httppeer.__init__ now takes additional arguments. Instances should be obtained by calling httppeer.instance() or httppeer.makepeer() instead. Differential Revision: https://phab.mercurial-scm.org/D2725

bundle: condition the changegroup part when creating a new bundle We will generate stream bundle in the next changesets which doesn't need the changegroup part. Differential Revision: https://phab.mercurial-scm.org/D1951

stack: return a sorted smartrev by default Most stack customers will display a list of revisions, sort it by default. Differential Revision: https://phab.mercurial-scm.org/D2399

histedit: use the new stack definition for histedit Now that we have a common stack definition, use it in the hg histedit command. Differential Revision: https://phab.mercurial-scm.org/D2398

show: use the new stack definition for show stack Now that we have a common stack definition, use it in the hg show stack command. Differential Revision: https://phab.mercurial-scm.org/D2397

stack: import Evolve stack test file Import Evolve stack test file for starting to have good coverage on stack definition. Differential Revision: https://phab.mercurial-scm.org/D2396

stack: add a new module for stack-related commands Bootstrap the module with a function that returns the stack for the current revision. The definition of the stack is ihnerited from histedit default revset and used by hg show stack. Differential Revision: https://phab.mercurial-scm.org/D2395

filemerge: use a single temp dir instead of temp files This can help to remove the clutter from UIs that display just the filenames; instead of seeing foo~local.C9ru9r.txt and foo~base.2DMV22.txt (in the /tmp directory on most platforms), we create a single new directory and use that, producing filenames like /tmp/hgmerge.C9ru9r/foo~local.txt. Differential Revision: https://phab.mercurial-scm.org/D2888

filemerge: move temp file unlinks to _maketempfiles Differential Revision: https://phab.mercurial-scm.org/D2887

hgweb: refactor multirequest to be a dict of lists ... instead of a list of 2-tuples. This makes key lookups faster. The only downside is we lose total ordering of all entries. But we weren't relying on that before, so it's no loss. Differential Revision: https://phab.mercurial-scm.org/D2881

py3: use "%d" % int instead of str(int) We need to use bytes internally. Differential Revision: https://phab.mercurial-scm.org/D2895

py3: use pycompat.bytestr() in dirstate.py This prevents extra b'' prefixes in output. Differential Revision: https://phab.mercurial-scm.org/D2894

py3: make tests/test-log-linerange.t work on Python 3 Made sure we write bytes to file and we suppress the return value of open(). Differential Revision: https://phab.mercurial-scm.org/D2893

py3: use print as a function in tests/test-narrow-clone-non-narrow-server.t Differential Revision: https://phab.mercurial-scm.org/D2892

tests: stop over-specifying tempfile name Python 3 has more random characters in the default template, which is fine. Differential Revision: https://phab.mercurial-scm.org/D2676

wireproto: explicitly flush stdio to prevent stalls on Windows This is the key to fixing the hangs on Windows in D2720[1]. I put flushes in a bunch of other places that didn't help, but I suspect that's more a lack of test coverage than anything else. Chasing down stuff like this is pretty painful. I'm wondering if we can put a proxy around sys.stderr (and sys.stdout?) on Windows (only when daemonized?) that will flush on every write (or at least every write with a '\n'). [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-March/113352.html

test-lfs: glob over some output changes These changes started with 2090044a288d. They aren't important to LFS testing.

templater: factor out helper that renders named template as string This is quite common in non-web templating, and **kwargs expansion is annoying because of the unicode-ness of Python3.

templater: rename .render(mapping) to .renderdefault(mapping) (API) I'm going to add templ.render(t, mapping) and templ.generate(t, mapping) in place of stringify(templ(t, **mapping)) and templ(t, **mapping) respectively. .. api:: The ``render(mapping)`` method of the templater has been renamed to ``renderdefault(mapping)``.

templater: fix invalid reference of runsymbol in doctest Oversight at da710c3df4dc. I was just lucky that this lambda function is never evaluated.

cmdutil: drop redundant import of merge module

annotate: add support for template keywords and functions depending on ctx

formatter: unblock storing fctx as a template resource To keep templateformatter._renderitem() simple, a repo instance is looked through ctx if available. This is probably good for future subrepo support where ctx.repo() may be different from the global repo.

templater: process mapping dict by resource callables A resource item is a callable, so let's make it look up a resource object by itself.

templater: convert resources to a table of callables for future extension I'm going to add a full templating support to the annotate command. As the annotate is a filectx-oriented command, we'll need a way to look up a ctx from a bounded fctx only when necessary. This is the minimal change to support that. I'm thinking of defining a proper interface to look up template resources to fix other issues, but that isn't ready yet. (Changes from V1: just updated tests and patch descriptions.)

rebase: on abort, don't strip commits that didn't need rebased (issue5822) I clearly missed adding this condition in 78496ac30025 (rebase: allow rebase even if some revisions need no rebase (BC) (issue5422), 2017-05-11). Perhaps I should have opted for the "revdone" solution I mentioned there... Differential Revision: https://phab.mercurial-scm.org/D2879

rebase: avoid defining two lists with the same contents In abort(), there's "dstates" and "rebased" that are identical, which they seem to have been since 0806823370d8 (rebase: properly calculate descendant set when aborting (issue3332), 2012-03-22). Let's de-duplicate. I don't know what "dstates" means, but "rebased" makes sense (it's the list of rebased revisions), so let's pick that. Differential Revision: https://phab.mercurial-scm.org/D2878

tests: demonstrate aborted rebase strips commits that didn't need rebasing I haven't verified, but this has probably been broken ever since I added the feature in 78496ac30025 (rebase: allow rebase even if some revisions need no rebase (BC) (issue5422), 2017-05-11). Differential Revision: https://phab.mercurial-scm.org/D2877

merge with stable

hgweb: convert an assert to a ProgrammingError Because assert may get optimized away. Differential Revision: https://phab.mercurial-scm.org/D2882

filemerge: give some variables in _xmerge more descriptive names Differential Revision: https://phab.mercurial-scm.org/D2886

rebase: allow in-memory merge of the working copy parent Before this patch and when the rebase involved the working copy parent (and thus the working copy too), we would not do in-memory rebase even if requested to. The in-code comment explains that the reason had something to do with avoiding an extra update, but I don't know which update that refers to. Perhaps an earlier version of the code used to update to the destination before rebasing even if in-memory rebase was requested? That seems to not be done at least since aa660c1203a9 (rebase: do not bail on uncomitted changes if rebasing in-memory, 2017-12-07). To see if this still made it slower, I create a single tiny commit on top of one branch of the mozilla-unified repo (commit a1098c82 to be exact) and rebased it to another branch (commit d4e9a7be). Before this patch that took 11.8s and after this patch it took 8.6s (I only did two runs each, but the timings were very consistent). Differential Revision: https://phab.mercurial-scm.org/D2876

uncommit: simplify condition for keeping commit We used to keep the commit if it would have become empty and --keep was not passed. Since we just changed it so we also keep it if any patterns were passed on the command line, the only remaining case where we prune the commit is when no arguments were passed and --keep was not passed either, we can simplify the "not files and not allowempty" to just "not allowempty". Let's also rename "allowempty" to "keepcommit" since it's no longer about allowing an empty commit. Differential Revision: https://phab.mercurial-scm.org/D2891

uncommit: document when the commit will be pruned Differential Revision: https://phab.mercurial-scm.org/D2890

uncommit: leave empty commit if all files are uncommitted We had instructions for our internal users to add copy information after commit like this: hg uncommit <srcfile> <dstfile> hg cp --after <srcfile> <dstfile> hg amend That usually works, but if the rename was the only change in that commit, then the commit would get pruned. It's easy to fix the recipe: just pass the --keep option. However, it seems too subtle, so I think this is an indication that the commit should not be pruned if any patterns were given. Differential Revision: https://phab.mercurial-scm.org/D2862

test-template-engine: do not evaluate unused keywords by custom engine If the custom engine, "mytemplater", were installed as the default, it would enter to an infinite recursion at stringify(v) because template keywords may generate a nested mapping containing the same keywords. Spotted by a future patch which will replace context.resource('templ')(...) with context.process(...).

py3: make test-template-engine.t bytes-safe

test-template-engine: deduplicate methods of custom template engine

revbranchcache: advertise and use 'rbc' exchange capability The feature is now advertised and use. Updating the branchmap cache can be very expensive (up to minutes on large repository) and fetching revision branch data is about 80% of that. Exchanging the rev branch cache over the wire really help to recover from branch map invalidation. (There is a good chance other in flight chance would conflict on test-http-bad-server.t and other. So here is a small note to help update that test again: capabilities=19bytes, part-107bytes)

revbranchcache: disable the new part for narrow hg bundle The lack of some revisions confuses the new cache part. To simplify things, we disable it for now.

revbranchcache: add the necessary bit to send 'rbc' data over bundle2 Getbundle is now capable of sending rev-branch-cache information for the changesets it bundle. The data sent are mostly nodes so it is quite compact. The goal of the rev-branch-cache is to speed up branch map computation, especially when the branchmap gets invalidated so we send data for all exchanged changesets. In addition, computing the relevant heads to send in case of partial pulling would be challenging. The feature is still inactive since the capability is not advertised yet.

bundle: include advisory rev branch cache part in bundle2 bundle `hg bundle` command producing bundle2 will now include an optional part containing the revision-branch cache data. The data sent are mostly nodes so it is quite compact. The goal of the rev-branch-cache is to speed up branch map computation, especially when the branchmap gets invalidated so we send data for all exchanged changesets. In addition, computing the relevant heads to send in case of partial pulling would be challenging. As a reminder, the rev branch cache data significantly speed up branch computation. Having it around provides a small speedup to pull/clone and much higher tolerance to branch map cache invalidation that might happens from later commands. On the Mercurial repository, computing the visible branchmap from scratch move from 2.00 seconds to 0.34s (a -83% speedup). Using this new part, Unbundling the full Mercurial repository moves from 25.736 seconds to 24.030 seconds (around -7% speedup). The bundle size increase is around 3% (from 22.43 MB to 23.13MB) On an half a million revision repository with twenty thousand branches, computing the branchmap moves from 75 seconds to 45 second (-40%) if the caches is used. A bundle containing 50 000 changesets in such repository get a 0.5% size increase from such part for a -3% unbundling time speedup.

rev-branch-cache: add a function to generate a part The function is able to produce a rbc part consumed by the function introduced into previous changesets. More details on usage and impact in the next changesets.

revbranchcache: add a bundle2 handler for a rbc part We add the necessary bit to process a part containing rev-branch-cache information. The local rev branch cache is then updated with the received information. Computing branch cache can become a significant part of the time spent pulling.

revbranchcache: add a public function to update the data We want to exchange more cached data over the wire. To do so, we need a clean way to update the cache on the receiving ends.

httppeer: consolidate _requestbuilder assignments and document I collapsed multiple assignments because they don't appear to be necessary. We don't invoke the requestbuilder in anything called during __init__. So there's no reason to not define it earlier AFAICT. This seemingly useless attribute is actually an extension point. Document it as such. (A previous version of this patch removed the attribute because it appeared to just be an alias.) Differential Revision: https://phab.mercurial-scm.org/D2723

commands: don't check for merge.update() truthiness AFAICT ``stats`` is always a tuple in these cases. We don't need to check if the variable has a truthy value. Differential Revision: https://phab.mercurial-scm.org/D2691

httppeer: alias url as urlmod "url" is a common variable name. We do this aliasing elsewhere to avoid shadowing. Differential Revision: https://phab.mercurial-scm.org/D2724

util: prefer "bytesio" to "stringio" The io.BytesIO and io.StringIO types enforce the type of data being operated on. On Python 2, we use cStringIO.StringIO(), which is lax about mixing types. On Python 3, we actually use io.BytesIO. Ideally, we'd use io.BytesIO on Python 2. But I believe its performance is poor compared to cString.StringIO(). Anyway, we canonically define our pycompat type as "stringio." That name is misleading, especially on Python 3. This commit renames the canonical symbols to "bytesio." "stringio" is preserved as an alias for API compatibility. There are a lot of callers in the repo and I hesitate to take away the old name. I also don't feel like changing everything at this time. But at least new callers can use a "proper" name. Differential Revision: https://phab.mercurial-scm.org/D2868

contrib: fix a subtle bug in check-code's regex rewriting We rewrite `\s` to `[ \t]` when preparing our regular expressions, but we previously weren't working to avoid having nested sets. Previously, Python let this slide without incident, but in Python 3.7 wants to make sure you meant an actual [ in a set, and so this warns. This appears to be fortunate for us, because `[\s(]` was getting rewritten to be `[[ \t](]` which doesn't actually match what we expected. See preceding changes that were revealed to be necessary after implementing this fix. Differential Revision: https://phab.mercurial-scm.org/D2866

remotenames: work around move of ABCs in collections This starts warning in Python 3.7, and will break in 3.8. The import is performed in a curious place because `import collections.abc` explodes badly on Python 2.7, and `from collections import abc` flunks our import checker. Differential Revision: https://phab.mercurial-scm.org/D2847

hgweb: explain instabilities of unstable changesets

debug: add debugwhyunstable that explains instabilities This is a port of evolve's feature of listing all unstable changesets in detail (`hg evolve --list`).

obsolete: move marker flags to obsutil Evolve extension expects to have obsolete.bumpedfix (imported in hgext3rd/evolve/evolvecmd.py), so we provide it.

debugupdatecache: also warm rev branch cache We add basic code to have `debugupdatecache` ensure that the rev branch cache is fully warmed. This only affects the `debugupdatecache` command, not normal transaction operation.

debugbundle: do not display detailed part data in --quiet mode This changeset makes `hg debugbundle` respect --quiet and only display data about part headers.

tests: clean up two wayward `python` invocations These are also fallout from the check-code regex rewriting bug. Differential Revision: https://phab.mercurial-scm.org/D2865

contrib: open a hole in the open().read() ban for open().close() It turns out open().close() is both fine and something we occasionally do to verify something can be written. The few cases in the codebase were getting missed due to a regular expression bug (which I discussed in my previous change), but since I'm about to fix the bug, I need to fix the patterns. Differential Revision: https://phab.mercurial-scm.org/D2864

cleanup: fix some latent open(path).read() et al calls we previously missed This pattern was banned by check-code way back in 1b4b82063ce2 (may of 2011), but due to a regular expression rewriting bug in check-code these particular callsites were never detected. Python 3.7 caught the bug, which then exposed these errors. Differential Revision: https://phab.mercurial-scm.org/D2863

uncommit: fix unaligned indentation Differential Revision: https://phab.mercurial-scm.org/D2861

util: clear debugstacktrace call During the renaming of datestr, it seems that I have forget a debugstacktrace in util.py. Remove it. Differential Revision: https://phab.mercurial-scm.org/D2838

peer-request: include more details about batch commands A batch commands encapsulate multiple other commands. We display a bit more details about what is actually batched if peer request tracking is set.

tests: fix test-wireproto.py to work around serverrepo() not having a ui This started failing recently, but it doesn't look important as no actual ui objects have been in play. Differential Revision: https://phab.mercurial-scm.org/D2867

tests: add a cat of `error.log` in subrepo test This made some debugging a lot less painful when something was broken, and it costs us almost nothing, so I figure we may as well leave it in. Differential Revision: https://phab.mercurial-scm.org/D2853

hghave: fix hardlink-whitelisted check on Python 3 Differential Revision: https://phab.mercurial-scm.org/D2846