tests: stop over-specifying tempfile name Python 3 has more random characters in the default template, which is fine. Differential Revision: https://phab.mercurial-scm.org/D2676

wireproto: explicitly flush stdio to prevent stalls on Windows This is the key to fixing the hangs on Windows in D2720[1]. I put flushes in a bunch of other places that didn't help, but I suspect that's more a lack of test coverage than anything else. Chasing down stuff like this is pretty painful. I'm wondering if we can put a proxy around sys.stderr (and sys.stdout?) on Windows (only when daemonized?) that will flush on every write (or at least every write with a '\n'). [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-March/113352.html

test-lfs: glob over some output changes These changes started with 2090044a288d. They aren't important to LFS testing.

templater: factor out helper that renders named template as string This is quite common in non-web templating, and **kwargs expansion is annoying because of the unicode-ness of Python3.

templater: rename .render(mapping) to .renderdefault(mapping) (API) I'm going to add templ.render(t, mapping) and templ.generate(t, mapping) in place of stringify(templ(t, **mapping)) and templ(t, **mapping) respectively. .. api:: The ``render(mapping)`` method of the templater has been renamed to ``renderdefault(mapping)``.

templater: fix invalid reference of runsymbol in doctest Oversight at da710c3df4dc. I was just lucky that this lambda function is never evaluated.

cmdutil: drop redundant import of merge module

annotate: add support for template keywords and functions depending on ctx

formatter: unblock storing fctx as a template resource To keep templateformatter._renderitem() simple, a repo instance is looked through ctx if available. This is probably good for future subrepo support where ctx.repo() may be different from the global repo.

templater: process mapping dict by resource callables A resource item is a callable, so let's make it look up a resource object by itself.

templater: convert resources to a table of callables for future extension I'm going to add a full templating support to the annotate command. As the annotate is a filectx-oriented command, we'll need a way to look up a ctx from a bounded fctx only when necessary. This is the minimal change to support that. I'm thinking of defining a proper interface to look up template resources to fix other issues, but that isn't ready yet. (Changes from V1: just updated tests and patch descriptions.)

rebase: on abort, don't strip commits that didn't need rebased (issue5822) I clearly missed adding this condition in 78496ac30025 (rebase: allow rebase even if some revisions need no rebase (BC) (issue5422), 2017-05-11). Perhaps I should have opted for the "revdone" solution I mentioned there... Differential Revision: https://phab.mercurial-scm.org/D2879

rebase: avoid defining two lists with the same contents In abort(), there's "dstates" and "rebased" that are identical, which they seem to have been since 0806823370d8 (rebase: properly calculate descendant set when aborting (issue3332), 2012-03-22). Let's de-duplicate. I don't know what "dstates" means, but "rebased" makes sense (it's the list of rebased revisions), so let's pick that. Differential Revision: https://phab.mercurial-scm.org/D2878

tests: demonstrate aborted rebase strips commits that didn't need rebasing I haven't verified, but this has probably been broken ever since I added the feature in 78496ac30025 (rebase: allow rebase even if some revisions need no rebase (BC) (issue5422), 2017-05-11). Differential Revision: https://phab.mercurial-scm.org/D2877

merge with stable

hgweb: convert an assert to a ProgrammingError Because assert may get optimized away. Differential Revision: https://phab.mercurial-scm.org/D2882

filemerge: give some variables in _xmerge more descriptive names Differential Revision: https://phab.mercurial-scm.org/D2886

rebase: allow in-memory merge of the working copy parent Before this patch and when the rebase involved the working copy parent (and thus the working copy too), we would not do in-memory rebase even if requested to. The in-code comment explains that the reason had something to do with avoiding an extra update, but I don't know which update that refers to. Perhaps an earlier version of the code used to update to the destination before rebasing even if in-memory rebase was requested? That seems to not be done at least since aa660c1203a9 (rebase: do not bail on uncomitted changes if rebasing in-memory, 2017-12-07). To see if this still made it slower, I create a single tiny commit on top of one branch of the mozilla-unified repo (commit a1098c82 to be exact) and rebased it to another branch (commit d4e9a7be). Before this patch that took 11.8s and after this patch it took 8.6s (I only did two runs each, but the timings were very consistent). Differential Revision: https://phab.mercurial-scm.org/D2876

uncommit: simplify condition for keeping commit We used to keep the commit if it would have become empty and --keep was not passed. Since we just changed it so we also keep it if any patterns were passed on the command line, the only remaining case where we prune the commit is when no arguments were passed and --keep was not passed either, we can simplify the "not files and not allowempty" to just "not allowempty". Let's also rename "allowempty" to "keepcommit" since it's no longer about allowing an empty commit. Differential Revision: https://phab.mercurial-scm.org/D2891

uncommit: document when the commit will be pruned Differential Revision: https://phab.mercurial-scm.org/D2890

uncommit: leave empty commit if all files are uncommitted We had instructions for our internal users to add copy information after commit like this: hg uncommit <srcfile> <dstfile> hg cp --after <srcfile> <dstfile> hg amend That usually works, but if the rename was the only change in that commit, then the commit would get pruned. It's easy to fix the recipe: just pass the --keep option. However, it seems too subtle, so I think this is an indication that the commit should not be pruned if any patterns were given. Differential Revision: https://phab.mercurial-scm.org/D2862

test-template-engine: do not evaluate unused keywords by custom engine If the custom engine, "mytemplater", were installed as the default, it would enter to an infinite recursion at stringify(v) because template keywords may generate a nested mapping containing the same keywords. Spotted by a future patch which will replace context.resource('templ')(...) with context.process(...).

py3: make test-template-engine.t bytes-safe

test-template-engine: deduplicate methods of custom template engine

revbranchcache: advertise and use 'rbc' exchange capability The feature is now advertised and use. Updating the branchmap cache can be very expensive (up to minutes on large repository) and fetching revision branch data is about 80% of that. Exchanging the rev branch cache over the wire really help to recover from branch map invalidation. (There is a good chance other in flight chance would conflict on test-http-bad-server.t and other. So here is a small note to help update that test again: capabilities=19bytes, part-107bytes)

revbranchcache: disable the new part for narrow hg bundle The lack of some revisions confuses the new cache part. To simplify things, we disable it for now.

revbranchcache: add the necessary bit to send 'rbc' data over bundle2 Getbundle is now capable of sending rev-branch-cache information for the changesets it bundle. The data sent are mostly nodes so it is quite compact. The goal of the rev-branch-cache is to speed up branch map computation, especially when the branchmap gets invalidated so we send data for all exchanged changesets. In addition, computing the relevant heads to send in case of partial pulling would be challenging. The feature is still inactive since the capability is not advertised yet.

bundle: include advisory rev branch cache part in bundle2 bundle `hg bundle` command producing bundle2 will now include an optional part containing the revision-branch cache data. The data sent are mostly nodes so it is quite compact. The goal of the rev-branch-cache is to speed up branch map computation, especially when the branchmap gets invalidated so we send data for all exchanged changesets. In addition, computing the relevant heads to send in case of partial pulling would be challenging. As a reminder, the rev branch cache data significantly speed up branch computation. Having it around provides a small speedup to pull/clone and much higher tolerance to branch map cache invalidation that might happens from later commands. On the Mercurial repository, computing the visible branchmap from scratch move from 2.00 seconds to 0.34s (a -83% speedup). Using this new part, Unbundling the full Mercurial repository moves from 25.736 seconds to 24.030 seconds (around -7% speedup). The bundle size increase is around 3% (from 22.43 MB to 23.13MB) On an half a million revision repository with twenty thousand branches, computing the branchmap moves from 75 seconds to 45 second (-40%) if the caches is used. A bundle containing 50 000 changesets in such repository get a 0.5% size increase from such part for a -3% unbundling time speedup.

rev-branch-cache: add a function to generate a part The function is able to produce a rbc part consumed by the function introduced into previous changesets. More details on usage and impact in the next changesets.

revbranchcache: add a bundle2 handler for a rbc part We add the necessary bit to process a part containing rev-branch-cache information. The local rev branch cache is then updated with the received information. Computing branch cache can become a significant part of the time spent pulling.

revbranchcache: add a public function to update the data We want to exchange more cached data over the wire. To do so, we need a clean way to update the cache on the receiving ends.

httppeer: consolidate _requestbuilder assignments and document I collapsed multiple assignments because they don't appear to be necessary. We don't invoke the requestbuilder in anything called during __init__. So there's no reason to not define it earlier AFAICT. This seemingly useless attribute is actually an extension point. Document it as such. (A previous version of this patch removed the attribute because it appeared to just be an alias.) Differential Revision: https://phab.mercurial-scm.org/D2723

commands: don't check for merge.update() truthiness AFAICT ``stats`` is always a tuple in these cases. We don't need to check if the variable has a truthy value. Differential Revision: https://phab.mercurial-scm.org/D2691

httppeer: alias url as urlmod "url" is a common variable name. We do this aliasing elsewhere to avoid shadowing. Differential Revision: https://phab.mercurial-scm.org/D2724

util: prefer "bytesio" to "stringio" The io.BytesIO and io.StringIO types enforce the type of data being operated on. On Python 2, we use cStringIO.StringIO(), which is lax about mixing types. On Python 3, we actually use io.BytesIO. Ideally, we'd use io.BytesIO on Python 2. But I believe its performance is poor compared to cString.StringIO(). Anyway, we canonically define our pycompat type as "stringio." That name is misleading, especially on Python 3. This commit renames the canonical symbols to "bytesio." "stringio" is preserved as an alias for API compatibility. There are a lot of callers in the repo and I hesitate to take away the old name. I also don't feel like changing everything at this time. But at least new callers can use a "proper" name. Differential Revision: https://phab.mercurial-scm.org/D2868

contrib: fix a subtle bug in check-code's regex rewriting We rewrite `\s` to `[ \t]` when preparing our regular expressions, but we previously weren't working to avoid having nested sets. Previously, Python let this slide without incident, but in Python 3.7 wants to make sure you meant an actual [ in a set, and so this warns. This appears to be fortunate for us, because `[\s(]` was getting rewritten to be `[[ \t](]` which doesn't actually match what we expected. See preceding changes that were revealed to be necessary after implementing this fix. Differential Revision: https://phab.mercurial-scm.org/D2866

remotenames: work around move of ABCs in collections This starts warning in Python 3.7, and will break in 3.8. The import is performed in a curious place because `import collections.abc` explodes badly on Python 2.7, and `from collections import abc` flunks our import checker. Differential Revision: https://phab.mercurial-scm.org/D2847

hgweb: explain instabilities of unstable changesets

debug: add debugwhyunstable that explains instabilities This is a port of evolve's feature of listing all unstable changesets in detail (`hg evolve --list`).

obsolete: move marker flags to obsutil Evolve extension expects to have obsolete.bumpedfix (imported in hgext3rd/evolve/evolvecmd.py), so we provide it.

debugupdatecache: also warm rev branch cache We add basic code to have `debugupdatecache` ensure that the rev branch cache is fully warmed. This only affects the `debugupdatecache` command, not normal transaction operation.

debugbundle: do not display detailed part data in --quiet mode This changeset makes `hg debugbundle` respect --quiet and only display data about part headers.

tests: clean up two wayward `python` invocations These are also fallout from the check-code regex rewriting bug. Differential Revision: https://phab.mercurial-scm.org/D2865

contrib: open a hole in the open().read() ban for open().close() It turns out open().close() is both fine and something we occasionally do to verify something can be written. The few cases in the codebase were getting missed due to a regular expression bug (which I discussed in my previous change), but since I'm about to fix the bug, I need to fix the patterns. Differential Revision: https://phab.mercurial-scm.org/D2864

cleanup: fix some latent open(path).read() et al calls we previously missed This pattern was banned by check-code way back in 1b4b82063ce2 (may of 2011), but due to a regular expression rewriting bug in check-code these particular callsites were never detected. Python 3.7 caught the bug, which then exposed these errors. Differential Revision: https://phab.mercurial-scm.org/D2863

uncommit: fix unaligned indentation Differential Revision: https://phab.mercurial-scm.org/D2861

util: clear debugstacktrace call During the renaming of datestr, it seems that I have forget a debugstacktrace in util.py. Remove it. Differential Revision: https://phab.mercurial-scm.org/D2838

peer-request: include more details about batch commands A batch commands encapsulate multiple other commands. We display a bit more details about what is actually batched if peer request tracking is set.

tests: fix test-wireproto.py to work around serverrepo() not having a ui This started failing recently, but it doesn't look important as no actual ui objects have been in play. Differential Revision: https://phab.mercurial-scm.org/D2867

tests: add a cat of `error.log` in subrepo test This made some debugging a lot less painful when something was broken, and it costs us almost nothing, so I figure we may as well leave it in. Differential Revision: https://phab.mercurial-scm.org/D2853

hghave: fix hardlink-whitelisted check on Python 3 Differential Revision: https://phab.mercurial-scm.org/D2846

hghave: fix xdiff check on Python 3 Differential Revision: https://phab.mercurial-scm.org/D2845

commandserver: prefer first-party selectors module from Python 3 to backport Caught by some deprecation warnings on Python 3.7. Differential Revision: https://phab.mercurial-scm.org/D2844

forget: add --dry-run mode

push-discovery: extract the bookmark comparison logic in its own function This will help extensions to alter the behavior as they see fit.

push-discovery: don't turn use generator when comparing bookmarks We want extensions to be able to implement their own logic. Generators can be consume only once, impractical for this purpose.

xdiff: move stdint.h to xdiff.h It's more correct to put it in xdiff.h since that file actually uses int64_t etc and xdiff.h is included by xinclude.h. This should fix the oss-fuzz build. Thanks durin42 for discovering the issue. Test Plan: `make local` and xdiff related tests still work. Differential Revision: https://phab.mercurial-scm.org/D2848

annotate: correct parameter name of decorate() function

pycompat: name maplist() and ziplist() for better traceback message

rebase: move constant expressions out of inner loop in _performrebase() Differential Revision: https://phab.mercurial-scm.org/D2811

rebase: inline _performrebasesubset() Now that most of _performrebasesubset() has been moved into _rebasenode(), it's simple enough that we can inline it. Differential Revision: https://phab.mercurial-scm.org/D2810

rebase: extract function for rebasing a single node We currently have _performrebase() and _performrebasesubset(), but we don't have a method for rebasing a single node (that's inside a loop in _performrebasesubset()). I think it makes sense to have such a method, so that's what this patch does. I think it may simplify future patches I'm working on that have to do with transactions, but I think this patch makes sense on its own whether or not that future work happens. Differential Revision: https://phab.mercurial-scm.org/D2809

rebase: use configoverride context manager for ui.forcemerge Differential Revision: https://phab.mercurial-scm.org/D2761

rebase: also restore "ui.allowemptycommit" value It looks like this was lost when the code was converted to the ui.configoverride() context manager in f255b1811f5e (rebase: get rid of ui.backupconfig, 2017-03-16). (And then the bad example was duplicated in 228916ca12b5 (rebase: add concludememorynode(), and call it when rebasing in-memory, 2017-12-07).) Differential Revision: https://phab.mercurial-scm.org/D2760

rebase: fix issue 5494 also with --collapse Differential Revision: https://phab.mercurial-scm.org/D2759

rebase: also include commit of collapsed commits in single transaction When rebase.singletransaction is set, we still used to create a second transaction when committing with --collapse. It's simpler to create a single transaction. Note that in the affected .t file, the test that uses --collapse still appears to create two transactions (it prints "rebase status stored" twice). However, only a single transaction is actually created and the second printout comes from cmdutil.commitforceeditor() that explicitly calls tr.writepending(). Also note the that we now roll back any commits if the user closes the commit message editor with an error code (or leaves the message empty). That might be unfortunate, but it's consistent with how we behave in the --no-collapse case (if the user passed --edit). If we want to change that, I think it should be done consistently in a separate patch. Differential Revision: https://phab.mercurial-scm.org/D2728

lfs: debug print HTTP headers and JSON payload received from the server This has been extremely valuable to show divergences between `hg serve` and `lfs-test-server`. Once the `hg serve` code lands, there will be a certain amount of conditionalizing that needs to be done, because `lfs-test-server` doesn't always follow its spec. The $ISO_8601_DATE_TIME$ pattern masks the fact that `lfs-test-serve` is sending out an expires_at value of "0001-01-01T00:00:00Z". `hg serve` will (probably) use current time + 10 minutes or similar. The $HTTP_DATE$ is the current time.

test-lfs: dial up the debugging on commands that interact with the server This will be useful to let the client print out the HTTP headers and JSON in a future patch, so we can compare native LFS serving against test-lfs-server behavior. There tends to be a lot of debug stuff that we don't care about here in a push, so I was tempted to print this output with a [devel] config. But this will be useful for field debugging too, so just put up with the extra output here. It would have been nice to be able to set ui.debug once, but issue5815 prevents that.

tests: add a substitution pattern for dates in HTTP headers and LFS payload This will be useful when printing HTTP headers and JSON payload received from an LFS server. The RFC 1123 date masking has uses elsewhere too.

xdiff: fix a hard crash on Windows The xdiff case of test-diff-antipatience.t started crashing in the C extension with 882657a9f768 (with 6a71a5ba666b backported so it compiles). There are a few more instances of 'long', but this resolves the crashing.

templater: split template functions to new module It has grown enough to be a dedicated module.

templater: move hybrid class and functions to templateutil module And make _hybrid and _mappable classes public. _showlist() is still marked as private since it's weird and third-party codes shouldn't depend on it.

templater: move stringify() to templateutil module As we have a util module, it doesn't make sense to import stringify() from templatefilters.py.

dagop: move lines() out of annotate()

dagop: extract core algorithm of annotate() from context.py See the previous patch for why.

dagop: move annotateline and _annotatepair from context.py The annotate logic is large. Let's move it out of the context module, which is basically an abstraction layer of repository operations.

bdiff: convert more longs to int64_t MSVC previously flagged these where the function is stored in a pointer: bdiff.c(284) : warning C4028: formal parameter 1 different from declaration bdiff.c(284) : warning C4028: formal parameter 2 different from declaration bdiff.c(284) : warning C4028: formal parameter 3 different from declaration bdiff.c(284) : warning C4028: formal parameter 4 different from declaration

xdiff: silence a 32-bit shift warning on Windows It's probably harmless, but: warning C4334: '<<' : result of 32-bit shift implicitly converted to 64 bits (was 64-bit shift intended?) Adding a 'ULL' suffix to 1 also works, but I doubt that's portable.

xdiff: backport int64_t and uint64_t types to Windows Sadly, MSVC 2008 lacks stdint.h. These are the only two definitions needed right now.

templater: extract template evaluation utility to new module Prepares for splitting template functions to new module. All eval* functions were moved to templateutil.py, and run* functions had to be moved as well due to the dependency from eval*s. eval*s were aliased as they are commonly used in codebase. _getdictitem() had to be made public.

templater: move function table to the "context" object Prepares for splitting template functions from templater.py.

hgweb: remove wsgirequest (API) Good riddance. .. api:: The old ``wsgirequest`` class for handling everything WSGI in hgweb has been replaced by separate request and response types. Various high-level functions in the hgweb WSGI applications now receive these new types as arguments instead of the old ``wsgirequest`` type. Differential Revision: https://phab.mercurial-scm.org/D2832

tweakdefaults: add commands.status.verbose to tweakefaults commands.status,verbose if set to True, shows conflict information in `hg status`. It shows which unresolved state you are in, which are the unresolved files and how to continue the unresolved state. That sounds like a very good candidate for tweakdefaults. bisect is added to commands.status.skipstates because people generally leave unresolved bisect state and we should skip that in morestatus output. Differential Revision: https://phab.mercurial-scm.org/D2806

hgweb: store the raw WSGI environment dict We need this so we can construct a new request instance from the original dict. Differential Revision: https://phab.mercurial-scm.org/D2831

hgweb: remove dead wsgirequest code All responses now go through our modern response type. All code related to response handling can be deleted. Differential Revision: https://phab.mercurial-scm.org/D2830

hgweb: port to new response API These were the last consumers of wsgirequest.respond() \o/ Differential Revision: https://phab.mercurial-scm.org/D2829

hgweb: pass modern request type into templater() Only a handful of consumers of wsgirequest remaining in this file... Differential Revision: https://phab.mercurial-scm.org/D2828

hgweb: use modern response type for index generation Differential Revision: https://phab.mercurial-scm.org/D2827

hgweb: don't pass wsgireq to makeindex and other functions We only ever access attributes that are available on our newer request type. So we no longer need this argument. Differential Revision: https://phab.mercurial-scm.org/D2826

hgweb: replace PATH_INFO with dispatchpath This was the last consumer of wsgireq.env from our WSGI applications! (Although indirect consumers of this attribute exist in wsgirequest.respond().) Differential Revision: https://phab.mercurial-scm.org/D2825

hgweb: rewrite path generation for index entries I think this code is easier to read. But the real reason to do this is to eliminate a consumer of wsgirequest. Differential Revision: https://phab.mercurial-scm.org/D2824

hgweb: construct {url} with req.apppath This is how the hgweb WSGI application does it. Let's make the behavior consistent. Differential Revision: https://phab.mercurial-scm.org/D2823

hgweb: support constructing URLs from an alternate base URL The web.baseurl config option allows server operators to define a custom URL for hosted content. The way it works today is that hgwebdir parses this config option into URL components then updates the appropriate WSGI environment variables so the request "lies" about its details. For example, SERVER_NAME is updated to reflect the alternate base URL's hostname. The WSGI environment should not be modified because WSGI applications may want to know the original request details (for debugging, etc). This commit teaches our request parser about the existence of an alternate base URL. If defined, the advertised URL and other self-reflected paths will take the alternate base URL into account. The hgweb WSGI application didn't use web.baseurl. But hgwebdir did. We update hgwebdir to alter the environment parsing accordingly. The old code around environment manipulation has been removed. With this change, parserequestfromenv() has grown to a bit unwieldy. Now that practically everyone is using it, it is obvious that there is some unused features that can be trimmed. So look for this in follow-up commits. Differential Revision: https://phab.mercurial-scm.org/D2822

hgweb: clarify that apppath begins with a forward slash Differential Revision: https://phab.mercurial-scm.org/D2821

hgweb: change how dispatch path is reported When I implemented the new request object, I carried forward some ugly hacks until I could figure out what was happening. One of those was the handling of PATH_INFO to determine how to route hgweb requests. Essentially, if we have PATH_INFO data, we route according to that. But if we don't, we route by the query string. I question if we still need to support query string routing. But that's for another day, I suppose. In this commit, we clean up the ugly "havepathinfo" hack and replace it with a "dispatchpath" attribute that can hold None or empty string to differentiate between the presence of PATH_INFO. This is still a bit hacky. But at least the request parsing and routing code is explicit about the meaning now. Differential Revision: https://phab.mercurial-scm.org/D2820

hgweb: refactor repository name URL parsing The hgwebdir WSGI application detects when a requested URL is for a known repository and it effectively forwards the request to the hgweb WSGI application. The hgweb WSGI application needs to route the request based on the base URL for the repository. The way this normally works is SCRIPT_NAME is used to resolve the base URL and PATH_INFO contains the path after the script. But with hgwebdir, SCRIPT_NAME refers to hgwebdir, not the base URL for the repository. So, there was a hacky REPO_NAME environment variable being set to convey the part of the URL that represented the repository so hgweb could ignore this path component for routing purposes. The use of the environment variable for passing internal state is pretty hacky. Plus, it wasn't clear from the perspective of the URL parsing code what was going on. This commit improves matters by making the repository name an explicit argument to the request parser. The logic around handling of this value has been shored up. We add various checks that the argument is used properly - that the repository name does represent the prefix of the PATH_INFO. Differential Revision: https://phab.mercurial-scm.org/D2819