templater: make it clearer that _flatten() omits None

revset: support "follow(renamed.py, e22f4f3f06c3)" (issue5334) v2: fixes from review

coal: use inheritance to derive from paper This illustrates how much simpler this approach is, in particular the effect of map-relative paths.

templater: add inheritance support to style maps We can now specify a base map file: __base__ = path/to/map/file That map file will be read and used to populate unset elements of the current map. Unlike using %include, elements in the inherited class will be read relative to that path. This makes it much easier to make custom local tweaks to a style.

killdaemons: prevent killing all user processes by bad pid file When I was fixing the test-gpg issue, I noticed gpg-connect-agent could print "-1" as a server pid if command was wrong. I'm not pretty sure but nobody would want to kill their running applications by mistake.

py3: automatically glob out line numbers from check-py3-compat output It was boring task to update the result manually.

py3: have check-py3-compat require pygments to get stable result

computeoutgoing: move the function from 'changegroup' to 'exchange' Now that all users are in exchange, we can safely move the code in the 'exchange' module. This function is really about processing the argument of a 'getbundle' call, so it even makes senses to do so.

getchangegroup: take an 'outgoing' object as argument (API) There is various version of this function that differ mostly by the way they define the bundled set. The flexibility is now available in the outgoing object itself so we move the complexity into the caller themself. This will allow use to remove a good share of the similar function to obtains a changegroup in the 'changegroup.py' module. An important side effect is that we stop calling 'computeoutgoing' in 'getchangegroup'. This is fine as code that needs such argument processing is actually going through the 'exchange' module which already all this function itself.

outgoing: add a 'missingroots' argument This argument can be used instead of 'commonheads' to determine the 'outgoing' set. We remove the outgoingbetween function as its role can now be handled by 'outgoing' itself. I've thought of using an external function instead of making the constructor more complicated. However, there is low hanging fruit to improve the current code flow by storing some side products of the processing of 'missingroots'. So in my opinion it make senses to add all this to the class.

outgoing: adds some default value for argument We are about to introduce a third option to create an outgoing object: 'missingroots'. This argument will be mutually exclusive with 'commonheads' so we implement some default value handling in preparation. This will also help use to make more use of outgoing creation around the code base.

outgoing: pass a repo object to the constructor We are to introduce more code constructing such object in the code base. It will be more convenient to pass a repository object, all current users already operate at the repository level anyway. More changes to the contructor argument are coming in later changeset.

match: remove matchessubrepo method (API) Since it is no more used in cmdutil.{files,remove} and scmutil.addremove we remove this method.

subrepo: cleanup of subrepo filematcher logic Previously in the worst case we iterated the files in matcher twice and had a method only for this, which reimplemented logic in subdirmatchers constructor. So we replaced the method with a subdirmatcher.files() call.

pycompat: delay loading modules registered to stub Replacement _pycompatstub designed to be compatible with our demandimporter. try-except is replaced by version comparison because ImportError will no longer be raised immediately.

py3: import builtin wrappers automagically by code transformer This should be less invasive than mucking builtins. Since tokenize.untokenize() looks start/end positions of tokens, we calculates them from the NEWLINE token of the future import.

py3: provide (del|get|has|set)attr wrappers that accepts bytes These functions will be imported automagically by our code transformer. getattr() and setattr() are widely used in our code. We wouldn't probably want to rewrite every single call of getattr/setattr. delattr() and hasattr() aren't that important, but they are functions of the same kind.

py3: check python version to enable builtins hack Future patches will add (del|get|has|set)attr wrappers.

py3: move xrange alias next to import lines Builtin functions should be available in compatibility code.

check-code: allow assignment to hasattr variable

debugobsolete: add formatter support (issue5134) It appears that computing index isn't cheap if --rev is specified. That's why "index" field is available only if --index is specified. I've named marker.flags() as "flag" because "flags" implies a list or dict in template world. Thanks to Piotr Listkiewicz for the initial implementation of this patch.

formatter: add function to convert dict to appropriate format This will be used to process key-value pairs by formatter. The default field names and format are derived from the {extras} template keyword. Tests will be added later.

check-code: make dict() pattern less invasive 'foodict(x=y)' should be allowed.

hgweb: tweak zlib chunking behavior When doing streaming compression with zlib, zlib appears to emit chunks with data after ~20-30kb on average is available. In other words, most calls to compress() return an empty string. On the mozilla-unified repo, only 48,433 of 921,167 (5.26%) of calls to compress() returned data. In other words, we were sending hundreds of thousands of empty chunks via a generator where they touched who knows how many frames (my guess is millions). Filtering out the empty chunks from the generator cuts down on overhead. In addition, we were previously feeding 8kb chunks into zlib compression. Since this function tends to emit *compressed* data after 20-30kb is available, it would take several calls before data was produced. We increase the amount of data fed in at a time to 32kb. This reduces the number of calls to compress() from 921,167 to 115,146. It also reduces the number of output chunks from 48,433 to 31,377. This does increase the average output chunk size by a little. But I don't think this will matter in most scenarios. The combination of these 2 changes appears to shave ~6s CPU time or ~3% from a server serving the mozilla-unified repo.

test-gpg: run migration of v1 secret keys beforehand This suppresses unwanted output at "hg sign".

test-gpg: start gpg-agent under control of the test runner GnuPG v2 automatically starts gpg-agent. We should kill the daemon process.

test-gpg: make temporary copy of GNUPGHOME GnuPG v2 will convert v1 secret keys and create a socket under $GNUPGHOME. This patch makes sure no state would persist. We no longer need to verify trustdb.gpg, which was added by aae219a99a6e.

hgweb: document why we don't allow untrusted settings to control zlib Added comment per discussion on mercurial-devel.

hgweb: profile HTTP requests Currently, running `hg serve --profile` doesn't yield anything useful: when the process is terminated the profiling output displays results from the main thread, which typically spends most of its time in select.select(). Furthermore, it has no meaningful results from mercurial.* modules because the threads serving HTTP requests don't actually get profiled. This patch teaches the hgweb wsgi applications to profile individual requests. If profiling is enabled, the profiler kicks in after HTTP/WSGI environment processing but before Mercurial's main request processing. The profile results are printed to the configured profiling output. If running `hg serve` from a shell, they will be printed to stderr, just before the HTTP request line is logged. If profiling to a file, we only write a single profile to the file because the file is not opened in append mode. We could add support for appending to files in a future patch if someone wants it. Per request profiling doesn't work with the statprof profiler because internally that profiler collects samples from the thread that *initially* requested profiling be enabled. I have plans to address this by vendoring Facebook's customized statprof and then improving it.

hgweb: abstract call to hgwebdir wsgi function The function names and behavior now matches hgweb. The reason for this will be obvious in the next patch.

profiling: don't error with statprof when profiling has already started statprof.reset() asserts if profiling has already started. So don't call if it profiling is already running.

profiling: add a context manager that no-ops if profiling isn't enabled And refactor dispatch.py to use it. As you can see, the resulting code is much simpler. I was tempted to inline _runcommand as part of writing this series. However, a number of extensions wrap _runcommand. So keeping it around is necessary (extensions can't easily wrap runcommand because it calls hooks before and after command execution).

profiling: make profiling functions context managers (API) This makes profiling more flexible since we can now call multiple functions when a profiler is active. But the real reason for this is to enable a future consumer to profile a function that returns a generator. We can't do this from the profiling function itself because functions can either be generators or have return values: they can't be both. So therefore it isn't possible to have a generic profiling function that can both consume and re-emit a generator and return a value.

dispatch: set profiling.enabled when profiling is enabled We do this for other global command arguments. We don't for --profile for reasons that are unknown to me. Probably because nobody has needed it. An upcoming patch will introduce a new consumer of the profiling code. It doesn't have access to command line arguments. So let's set the config option during argument processing. We also remove a check for "options['profile']" because it is now redundant.

profiling: move profiling code from dispatch.py (API) Currently, profiling code lives in dispatch.py, which is a low-level module centered around command dispatch. Furthermore, dispatch.py imports a lot of other modules, meaning that importing dispatch.py to get at profiling functionality would often result in a module import cycle. Profiling is a generic activity. It shouldn't be limited to command dispatch. This patch moves profiling code from dispatch.py to the new profiling.py. The low-level "run a profiler against a function" functions have been moved verbatim. The code for determining how to invoke the profiler has been extracted to its own function. I decided to create a new module rather than stick this code elsewhere (such as util.py) because util.py is already quite large. And, I foresee this file growing larger once Facebook's profiling enhancements get added to it.

merge with stable

pycompat: avoid using an extra function We have a single line function which just lowercase the letters and replaces "_" with "". Its better to avoid that function call. Moreover we calling this function around 33 times.

pycompat: remove multiple occurences of urlencode By mistake we had two occurences of urlencode.

osx: stamp the hg version into the version field in the pkg This is required for tools like https://github.com/munki/munki, and is also more semantically correct.

performance: disable workaround for an old bug of Python gc Since disabling the gc does things worse for pypy and the bug was fixed in 2.7, let's only enable it in <2.7

merge: always use other, not remote, in user prompts Now that we store and display merge labels in user prompts (not just conflict markets), we should rely on labels to clarify the two sides of a merge operation (hg merge, hg update, hg rebase etc). "remote" is not a great name here, as it conflates "remote" as in "remote server" with "remote" as in "the side of the merge that's further away". In cases where you're merging the "wrong way" around, remote can even be the "local" commit that you're merging with something pulled from the remote server.

merge: use labels in prompts to the user Now that we persist the labels, we can consistently use the labels in prompts for the user without risk of confusion. This changes a huge amount of command output: This means that merge prompts like: no tool found to merge a keep (l)ocal, take (o)ther, or leave (u)nresolved? u and remote changed a which local deleted use (c)hanged version, leave (d)eleted, or leave (u)nresolved? c become: no tool found to merge a keep (l)ocal [working copy], take (o)ther [destination], or leave (u)nresolved? u and remote [source] changed a which local [dest] deleted use (c)hanged version, leave (d)eleted, or leave (u)nresolved? c where "working copy" and "destination" were supplied by the command that requested the merge as labels for conflict markers, and thus should be human-friendly.

journal: use the dirstate parentchange callbacks Instead of hacking into dirstate internals let's use the callbacks to be notified about wd parent change.

dirstate: add callback to notify extensions about wd parent change The journal extension had to touch the dirstate internals to be notified about wd parent change. To make that detection cleaner and reusable let's move it core. Now the extension can register to be notified about parent changes.

revpair: do not optimize tree to check for odd-range spec At cc3a30ff9490, we had to optimize a parsed tree to resolve x^:y ambiguity. Since we've moved the resolution of x^:y to parse(), we no longer have to call optimize(). Therefore, (x:y) can be taken as a single expression, not an odd range expression x:y.

revset: also parse x^: as (x^): Given x^:y is (x^):y, this seems sensible.

revset: resolve ambiguity of x^:y before alias expansion This is purely a parsing problem, which should be resolved before alias expansion.

revset: add test for resolution of infix/suffix ambiguity of x^:y This is the test for 805651777188, and I'm going to fix the failure of 'x^A' where 'revsetalias.A=:y'.

parser: remove unused binding parameter from suffix action Because a suffix action never takes subsequent tokens, it should have no binding strength nor closing character. I've tried if this value could be used to resolve infix/suffix ambiguity of x^:y, but it appears not. So I decided to resend this patch.

revset: fix keyword arguments to go through optimization process Before, a keyvalue node was processed by the last catch-all condition of _optimize(). Therefore, topo.firstbranch=expr would bypass tree rewriting and would crash if an expr wasn't trivial.

extensions: add unwrapfunction to undo wrapfunction Before this patch, we don't have a safe way to undo a wrapfunction because other extensions may wrap the same function and calling setattr will undo them accidentally. This patch adds an "unwrapfunction" to address the issue. It removes the wrapper from the wrapper chain, and re-wraps everything, which is not the most efficient but short and easy to understand. We can revisit the code if we have perf issues with long chains. The "undo" feature is useful in cases like wrapping a function just in a scope. Like, having a "select" command to interactively (using arrow keys) select content from some output (ex. smartlog). It could wrap "ui.label" to extract interesting texts just in the "select" command.

extensions: add getwrapperchain to get a list of wrappers The getwrapperchain returns a list of wrappers + the original function, making it easier to understand what has been wrapped by whom. For example: In : mercurial.extensions.getwrapperchain(mercurial.dispatch, '_runcommand') Out: [<function hgext.pager.pagecmd>, <function hgext.color.colorcmd>, <function hgext.zeroconf.cleanupafterdispatch>, <function mercurial.dispatch._runcommand>] It will also be useful to safely unwrap a function. See the next patch.

extensions: set attributes to wrappers so we can trace them back This patch adds two attributes about the original function and the unbound wrapper. It allows us to get a chain of wrappers. See the next patch.

ui: drop values returned by inspect.*frame*() to avoid cycles "f = inspect.currentframe()" instantly creates a cycle because "f.f_locals['f']" is "f" itself. This patch explicitly sets those frame objects to None to avoid cycles.

dispatch: split global error handling out so it can be reused We may want a similar error handling at worker.py. This patch extracts the error handling logic to "callcatch" so it can be reused.

py3: conditionalize _winreg import _winreg module is renamed to winreg in python 3. Added the conditionalize statements in the respective file because adding this in pycompat will result in pycompat throwing error as this is a windows registry module and we have buildbots and most of the contributors on linux.

py3: conditionalize the raise statement raise E,V,T is not acceptable in Python 3, thats is conditionalized. Moreover this will result in syntax error so we have to use exec() to execute this. Related PEP- https://www.python.org/dev/peps/pep-3109/#id14 My implementation is motivated from the six implementation except they are defining a new function exec_() to prevent adding an extra frame AFAIK :) https://bitbucket.org/gutworth/six/src/ca4580a5a648/six.py#six.py-680

match: added matchessubrepo method to matcher Previously there were three local implementations of this function in cmdutil.files, cmdutil.remove and scmutil.addremove.

changegroup: move branch cache debug message to proper location Before, we logged about performing a branch cache update when we weren't actually doing it. Fix that.

journal: take wlock for writting the 'shared' file As we did for the shared extension itself, we add some locking around the write of the 'shared' file.

debugbuilddag: take wlock to cover '.hg/localtags' This debug command can write local tags. local tags are in the .hg directory and should be covered by the 'wlock'. This is now covered.

fakemergerecord: take wlock to write the merge state The merge state is supposed to be covered by the wlock. We fix the test extensions to comply to that.

shared: take wlock for writting the 'shared' file I do not see a reason why this should not be covered by the wlock.

mq: take wlock when 'qqueue' is doing write operations Apparently when this command was added, the locking was forgotten. No code changes beside the indentation from the locking context.

py3: make check-py3-compat.py use correct module name at loading pure modules Before this patch, check-py3-compat.py implies unintentional ".pure" sub-package name at loading pure modules, because module name is composed by just replacing "/" in the path to actual ".py" file by ".". This makes pure modules belong to "mercurial.pure" package, and prevents them from importing a module belonging to "mercurial" package relatively by "from . import foo" or so. This is reason why pure modules fail to import another module relatively only at examination by check-py3-compat.py.

py3: update output of check-py3-compat.py with python3 Recent work on mercurial/pure/mpatch.py made it import mercurial.policy, and changed output of check-py3-compat.py with python3 in test-check-py3-compat.t. But test-check-py3-compat.t isn't yet updated.

mpatch: raise MemoryError instead of mpatchError if lalloc() failed MemoryError is handled differently in dispatch._runcatch(). Since mpatch_errors[] isn't that useful now, I've changed it to a simple switch statement.

hgweb: config option to control zlib compression level Before this patch, the HTTP transport protocol would always zlib compress certain responses (notably "getbundle" wire protocol commands) at zlib compression level 6. zlib can be a massive CPU resource sink for servers. Some server operators may wish to reduce server-side CPU requirements while requiring more bandwidth. This is common on corporate intranets, for example. Others may wish to use more CPU but reduce bandwidth. This patch introduces a config option to allow server operators to control the zlib compression level. On the "mozilla-unified" generaldelta repository, setting this value to "0" (disable compression) results in server-side CPU utilization for a `hg clone` going from ~180s to ~124s CPU time on my i7-6700K. A level of "1" (which increases the transfer size from ~1,074 MB at level 6 to ~1,222 MB) utilizes ~132s CPU time.

help: don't try to render a section on sub-topics This patch subtly changes the behavior of the parsing of "X.Y" values to not set the "section" variable when rendering a known sub-topic. Previously, "section" would be the same as the sub-topic name. This required the sub-topic RST to have a section named the same as the sub-topic name. When I made this change, the descriptions from help.internalstable started being rendered in command line output. This didn't look correct to me, as it didn't match the formatting of main help pages. I corrected this by moving the top section to help.internalstable and changing the section levels of all the "internals" topics. The end result is that "internals" topics now match the rendering of main topics on both the CLI and HTML. And, "internals" topics no longer require a main section matching the name of the topic.

branchmap: remove extra indent This clean up the rest of the previous changeset.

branchmap: simplify error handlind when writing rev branch cache Now that we have a general try except, we can move the error handling from the individual writes in it. Code will be reindented in the next changeset to help this on readability.

branchmap: acquires lock before writting the rev branch cache We now attempt to acquire a lock and write the branch cache within that lock. This would prevent cache corruption when multiple processes try to write the cache at the same time.

branchmap: preparatory indent of indent the branch rev writing code The rev branch cache is written without a lock, we are going to fix this but we indent the code beforehand to make the next changeset clearer.

mpatch: silence warning about maybe-uninitialized variable It's false positive, but it wouldn't be possible for gcc to know PyBytes_FromStringAndSize() sets PyErr_Occurred(). mercurial/mpatch_module.c:105:47: warning: 'r' may be used uninitialized in this function [-Wmaybe-uninitialized] PyErr_SetString(mpatch_Error, mpatch_errors[-r]);

mpatch: change lalloc() to local function It was mistakenly made public at b9b9f9a92481.

mpatch: remove superfluous whitespaces

cmdutil: remove duplicated badmatch call in cat() Subrepo logic is handled in ctx.walk().

statichttprepo: do not try to write caches The static http repository are read only, there is no hope in any cache writing attempt.

demandimport: omit default value of "level" at construction of _demandmod This makes construction of _demandmod require explicit level value, and helps to avoid issues like issue5208 in the future.

demandimport: import sub-module relatively as expected (issue5208) Before this patch, importing sub-module might (1) fail or (2) success but import incorrect module, because demandimport tries to import sub-module with level=-1 (on Python 2.x) or level=0 (on Python 3.x), which is default value of "level" argument at construction of "_demandmod" proxy object. (1) on Python 3.x, importing sub-module always fails to import existing sub-module (2) both on Python 2.x and 3.x, importing sub-module might import same name module on root level unintentionally On Python 2.x, existing sub-module is prior to this unexpected module. Therefore, this problem hasn't appeared. To import sub-module relatively as expected, this patch specifies "1" as import level explicitly at construction of "_demandmod" proxy object for sub-module.

wireproto: remove gboptslist (API) This variable has been unused since ce25f465e572, which was over 2 years ago. gboptsmap should be used instead. Marking as API because this could break extensions.

wireproto: unescape argument names in batch command (BC) Clients escape both argument names and values when using the batch command. Yet the server was only unescaping argument values. Fortunately we don't have any argument names that need escaped. But that isn't an excuse to lack symmetry in the code. Since the server wasn't properly unescaping argument names, this means we can never introduce an argument to a batchable command that needs escaped because an old server wouldn't properly decode its name. So we've introduced an assertion to detect the accidental introduction of this in the future. Of course, we could introduce a server capability that says the server knows how to decode argument names and allow special argument names to go through. But until there is a need for it (which I doubt there will be), we shouldn't bother with adding an unused capability.

wireproto: consolidate code for obtaining "cmds" argument value Both wireproto.py and sshpeer.py had code for producing the value to the "cmds" argument used by the "batch" command. This patch extracts that code to a standalone function and uses it.

revlog: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

md5sum: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

util: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

url: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

sshserver: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

sshpeer: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

patch: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

commands: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

changegroup: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

extdiff: isolate path variable of saved command to independent paragraph Otherwise, the whole paragraph wouldn't be translated.

extdiff: export __doc__ of saved command for translation

extdiff: refactor closure of saved diff command as a top-level class This allows us to collect __doc__ for translation.

i18n: use inspect.getsourcelines() to obtain lineno from func or class Before, func must be a function object. I want to make it parse docstring of classes, too.

chg: just take it as EOF if recv() returns 0 hgc->sockfd is a blocking stream socket. recv() should never return 0 other than EOF. See 4fc4b8cc9957 for the original problem.

vfs: use propertycache for open The current open method is currently behaving like a property cache. We use our utility decorator to make this explicit.

help: add example of '[templates]' usage V2: - move from shortest() with minlength 8 to minlength 4 - mention [templates] in config.txt - better describe the difference between [templatealias] and [templates] V3: - choose a better example template

check-commit: allow underbars in cffi_-prefix function names It seems reasonable to give cffi functions slightly more verbose names in some circumstances, given the way they interface with C.

bundlerepo: add support for treemanifests in cg3 bundles This is a little messier than I'd like, and I'll probably come back and do some more refactoring later, but as it is this unblocks narrowhg. An alternative approach (which I may do as part of the mentioned refactoring) would be to construct *all* dirlog instances up front, so that we don't have to keep track of the linkmapper method. This would avoid a reference cycle between the bundlemanifest and the bundlerepository, but I was hesitant to do all the work up front like that. With this change, it's possible to do 'hg incoming' and 'hg pull' from bundles in .hg/strip-backup in a treemanifest repository. Sadly, this doesn't make it possible to 'hg clone' one of those (if you do 'hg strip 0'), because the cg3 in the bundle gets written without a treemanifest flag. Since that's going to be an involved refactor in a different part of the code (which I *suspect* won't touch any of the code I've just written here), let's leave it as an idea for Later.

auditvfs: forward options property from nested vfs This was breaking my ability to use treemanifests in bundlerepos, and was deeply mysterious. We should probably just make the options property a formal part of the vfs API, and make it a required construction parameter. Sadly, I don't have time to dive into that refactor right now.

bundlerepo: use supportedincomingversions instead of allsupportedversions Since bundlerepo is really a pull-like operation, this is the correct method to use here.

bundlerepo: introduce method to find file starts and use it This moves us to the modern iter() technique instead of the `while True` pattern since it's easy. Factored out as a function because I'm about to need this in a second place.

bundlerevlog: use for loop over iterator instead of while True The iter() builtin has a neat pattern where you give it a callable of no arguments and a sentinel value, and you can then loop over the function calls like a normal iterator. This cleans up the code a little.

bundlerepo: use for loop over iterator instead of while True The iter() builtin has a neat pattern where you give it a callable of no arguments and a sentinel value, and you can then loop over the function calls like a normal iterator. This cleans up the code a little.

localrepo: jettison now-unused dirlog() method from localrepo

repair: build dirlogs using manifest, rather than repo shortcut method As before, this was rarely used, so let's get rid of the convenience method.

cmdutil: open dirlogs via manifest property, not via repo This was a convenience method that was rarely used, so let's get rid of it.

wirepeer: rename confusing `source` parameter It's named "url" everyplace else this method is defined, so let's be consistent.

develwarn: use the lock helper in local repo We have a helper function to know if a lock is taken. When checking lock usage we now use it instead of manually accessing the locks.

exchange: correctly specify url to unbundle (issue5145) This parameter is slightly confusingly named in wireproto, so it got mis-specified from the start as 'push' instead of the URL to which we are pushing. Sigh. I've got a patch for that which I'll mail separately since it's not really appropriate for stable. Fixes a regression in bundle2 from bundle1.

debugextension: change "testedwith" to a list (BC) It wasn't a list since the formatter couldn't process a list. We have no such problem now and the -T option is experimental, so we can change it.

debugextensions: unindent nested if

debugextensions: give short name to util.version()

debugextensions: simply keep testedwith variable as a list There should be no need to distinguish [] and None.

test-treemanifest: ensure manifest command isn't broken I realized we weren't testing this while hunting a broken manifest command bug that ended up being narrowhg's fault.

py3: use unicode literals in pure/osutil.py The first element of _fields_ tuples must be a str in Python 3. Also fix some function calls that were also expecting str.

py3: use unicode literals in crecord.py

py3: use unicode literals in changelog.py collections.namedtuple type and field names must be str in Python 3. Our custom module importer was rewriting them to bytes literals, making this fail. In addition, __slots__ values must also be unicode.

mpatch: write a cffi version of mpatch.patches

mpatch: remove dependency on Python.h in mpatch.c Now all the CPython-related stuff are referenced only from mpatch_module.c with mpatch.c being freely usable from a future cffi module

mpatch: split mpatch into two files

mpatch: provide things that will be exported later with a prefixed name For cffi a bunch of mpatch functions need to be visible through a .h file. This change renames them so it won't create potential c namespace conflicts.

mpatch: change Py_ssize_t to ssize_t in places that will be later copied

discovery: move code to create outgoing from roots and heads changegroup.changegroupsubset() contained somewhat low-level code for constructing an "outgoing" instance from a list of roots and heads nodes. It feels like discovery.py is a more appropriate location for this code. This code can definitely be optimized, as outgoing.missing will recompute the set of changesets we've already discovered from cl.between(). But code shouldn't be refactored during a move, so I've simply inserted a TODO calling attention to that.

bundle2: remove 'experimental.bundle2-exp' boolean config (BC) All users are migrated to 'devel.legacy.exchange', we can clean up the experimental namespace. Marking as (BC) because I know some large installation have bundle2 off and I want to make sure they notice the change.

tests: remove all remaining usage of experimental.bundle2-exp The only remaining usage of the experimental config were enforcing bundle2 on. These are very old remains of when bundle2 was off by default. This was also useful to highlight the fact that this was a bundle2 run and that a bundle1 one was nearby. However, we want a future developer working on bundle3 to notice possible output/behavior change on these tests and take them in account. So we do not enforce bundle2 for these runs. We leave a comment around to make sure dev still notice the bundle1 version.

tests: use 'legacy.exchange' option in various mixed tests The new option will stay around. The experimental option was only meant to be temporary. We update various tests that validate both bundle1 and bundle2 version side by side. This changeset only takes care of enforcing bundle1. The other use of 'experimental.bundle2-exp=True' will be taken care of in the next changeset.

tests: use 'legacy.exchange' option in various bundle1 tests The new option will stay around. The experimental option was only meant to be temporary.

tests: use 'legacy.exchange' option in 'test-bundle2-exchange.t' The new option will stay around. The experimental option was only meant to be temporary.

bundlerepo: also read the 'devel.legacy.exchange' config Bundlerepo does its own bundle2 related logic.

bundle2: add a devel option controling bundle version used for exchange We need an official way to force bundle1 to be used in test. We introduce a new option 'devel.legacy.exchange' to control this. When specified, this option will control the list of bundle version Mercurial consider when exchanging with a peer. Current valid value are 'bundle1' and 'bundle2'. Using this option in all tests will allow us to remove the 'experimental.bundle2-exp' option. We will simplify the code once the experimental option is dropped.

bundle2: rename the _canusebundle2 method to _forcebundle1 We rename and invert the logic of the _canusebundle2 utility. The idea here is that we need to have a way to enforce the use of bundle1 in the tests. The Mercurial philosophy is to try to use the best method available. Currently that best method is bundle2, but this might change in the future. Therefore expressing "do not use bundle2" is a loosy way to say "use bundle 1" and will likely create issue in the future. As the config option will be explicitly about bundle1, we rename the function beforehand to align with this. This will make the life of a future developer working on bundle3 easier.

tests: remove bundle2 activation from test-bookmark-pushpull.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: update bugzilla link in test-issue1175.t

journal: use fm.formatdate() to pass date tuple in appropriate type (BC)

formatter: add function to convert date tuple to appropriate format

journal: use fm.formatlist() to pass hashes in appropriate type (BC)

formatter: add function to convert list to appropriate format (issue5217) Before, it wasn't possible for formatter to handle array structure other than date tuple. We've discussed that at the last sprint, which ended we would probably want to allow only templatable data structure, i.e. a list of dicts: data(tags=[{'tag': a}, {'tag': b}, ...]) Unfortunately, it turned out not working well with template functions: "{ifcontains(a, tags, ...)}" ^^^^^^^^^^^^^^^^^^ "a in tags", where tags should be a plain list/set of tags So the formatter must at least know if the type [{}] was constructed from a plain list or was actually a list of dicts. This patch introduces new explicit interface to convert an array structure to an appropriate data type for the current formatter, which can be used as follows: fm.write('tags', _('tags: %s\n'), fm.formatlist(tags, name='tag')) No separate fm.data() call should be necessary.

journal: use fm.hexfunc() to get full hash in JSON/template output (BC) We generally do that.

py3: update test-check-py3-compat.t output The lower part of test-check-py3-compat.t runs only on py3 and hence its remain unchanged. Hence this patch updates the output so that change in output in the next patches will be only related to the change in the patch.

deprecation: enforce thew 'tr' argument of 'dirstate.write' (API) Compatibility was meant to be drop after 3.9 is released.

tests: remove initial bundle2 enabling in various bundle2 tests Bundle 2 is enable by default since 3.6 so we don't need this anymore. The remaining use of 'experimental.bundle2-exp' are there to test legacy behavior with bundle 1. Future patches will introduce a way outside of experimental to test that.

tests: remove bundle2 activation from test-largefiles.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-tags.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

templatekw: change joinfmt to a mandatory argument of _hybrid object We've fixed several bugs caused by the misuse of the default joinfmt. Make it more explicit to prevent future bugs. dict.values()[0] is replaced by dict[element] as showlist() knows what the key is.

convert: move svn config initializer out of the module level The svn_config_get_config config call was being called at the module level, but had the potential to throw permission denied errors if ~/.subversion/servers was not readable. This could happen in certain test environments where the user permissions were very particular. This prevented the remotenames extension from loading, since it imports convert's hg module, which imports convert's subversion module, which calls this. The config is only ever used from this one constructor, so let's just move it in to there.

osx: update bugzilla link in readme

win32: update link to mailing list in readme

win32: update wiki link in mercurial.ini

debian: update source URL in copyright file

help: update link to wiki/CommandServer

url: drop compatibility wrapper of socket.create_connection() It should be available on Python 2.6+.

doc: remove double imports of inspect from hgmanpage.py

merge: concatenate default conflict marker at parsing phase of .py "+" operations are unnecessary.

tests: remove bundle2 activation from test-push-warn.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-phases-exchanges.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-obsolete.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-http-proxy.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-hook.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-acl.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

merge with stable

Added signature for changeset 299546f84e68

Added tag 3.9 for changeset 299546f84e68

merge with i18n

doc: make previous line of certificate example end with "::" Before this patch, certificate example is formatted just as normal text.

doc: fix incorrect use of rst hg role in help text

doc: use field rst syntax to show keywords in debugdeltachain help correctly List of available keywords is well formatted as a list of fields in doc string, but is formatted as just normal text in online help output.

revset: refactor to make xgettext put i18n comments into hg.pot file xgettext expects both "_()" and (a part of) text to be placed at just next line of "i18n:" comment.

doc: omit useless _() invocation In this case, column positioning isn't needed for i18n, too. Maybe, check-code warning "missing _() in ui message" caused this useless _() invocation in 92d37fb3f1aa.

doc: trim newline at the end of exception message

i18n-ja: synchronized with 6fd751fa58d3

demandimport: avoid infinite recursion at actual module importing (issue5304) Before this patch, importing C module on Windows environment causes infinite recursion call, if py2exe is used with -b2 option. At importing C module "a.b", extra hooking by zipextimporter of py2exe causes: 0. assumption before accessing "b" of "a": - built-in module object is created for "a", (= "a" is actually imported) - _demandmod is created for "a.b" as a proxy object, and (= "a.b" is not yet imported) - an attribute "b" of "a" is initialized by the latter 1. invocation of __import__ via _hgextimport() in _demandmod._load() for "a.b" implies _demandimport() for "a.b" This is unintentional, because _demandmod might be returned by _hgextimport() instead of built-in module object. 2. _demandimport() at (1) is invoked with not context of "a", but context of zipextimporter Just after invocation of _hgextimport() in _demandimport(), an attribute "b" of the built-in module object for "a" is still bound to the proxy object for "a.b", because context of "a" isn't updated by actual importing "a.b". even though the built-in module object for "a.b" already appears in sys.modules. Therefore, chainmodules() returns _demandmod for "a.b", which is gotten from the attribute "b" of "a". 3. processfromitem() on "a.b" causes _demandmod._load() for "a.b" again _demandimport() takes context of "a" in this case. Therefore, attributes below are bound to built-in module object for "a.b", as expected: - "b" of built-in module object for "a" - _module of _demandmod for "a.b" 4. but _demandimport() invoked at (1) returns _demandmod object because _demandimport() just returns the object returned by chainmodules() at (3) above. 5. then, _demandmod._load() causes infinite recursion call _demandimport() returns _demandmod for "a.b", and it is "self" at _demandmod._load(). To avoid infinite recursion at actual module importing, this patch uses self._module, if _hgextimport() returns _demandmod itself. If _demandmod._module isn't yet bound at this point, execution should be aborted, because actual importing failed. In this patch, _demandmod._module is examined not on _demandimport() side, but on _demandmod._load() side, because: - the former has some exit points - only the latter uses _hgextimport(), except for _demandimport() BTW, this issue occurs only in the code path for non .py/.pyc files in zipextimporter (strictly speaking, in _memimporter) of py2exe. Even if zipextimporter is enabled, .py/.pyc files are handled by zipimporter, and it doesn't imply unintentional _demandimport() at invocation of __import__ via _hgextimport().

packagelib: do not remove packages directory in hggetversion (issue5262) People running packages related code probably do care about the content of this directory. In particular this shound fix the rpm builder process.

make: introduce a target to clean everything but packages Removing the 'packages' directory makes nightly builder life much harder.

url: avoid re-issuing incorrect password (issue3210) Some draconian IT setups lock accounts after a small number of incorrect password attempts. Mercurial's implementation of the urllib2 authentication was causing 5 retry attempts with the same credentials, without prompting the user. The code was attempting to check whether the authorization token had changed, but unfortunately was reading the misleading 'headers' member of the request instead of using the 'get_header' accessor. Modelled on fix for Python issue 8797: https://bugs.python.org/issue8797 https://hg.python.org/cpython/rev/30e8a8f22a2a

date: accept broader range of ISO 8601 time specs The "normal" ISO date/time includes a T between date and time. It also allows dropping the colons and seconds from the timespec. Add new patterns for these forms as well as tests.

date: parse ISO-style Z and +hh:mm timezone specs

date: refactor timezone parsing We want to be able to accept ISO 8601 style timezones that don't include a space separator, so we change the timezone parsing function to accept a full date string and return both the offset and the non-timezone portion.

tests: glob over ssl error We do this in the rest of the file. This bad line was introduced in 3fde328d0913.

keyword: avoid traceback when kwdemo is run outside a repo f0564402d059 causes a fatal AttributeError if kwdemo is run outside a repo because in the temporary repo creation repo is None and therefore cannot have a baseui attribute. In this case fall back to using ui. Add test case.

cmdutil: warnings not issued in cat if subrepopath overlaps Previously a subrepository "sub" would cause no warnings to be issued for a file "subnot/a", if it's not present in the corresponding changeset when calling: hg cat subnot/a

graft: use opts.get() consistently Make life easier for extension writers.

sslutil: work around SSLContext.get_ca_certs bug on Windows (issue5313) SSLContext.get_ca_certs() can raise "ssl.SSLError: unknown error (_ssl.c:636)" on Windows. See https://bugs.python.org/issue20916 for more info. We add a try..except that swallows the exception to work around this bug. If we encounter the bug, we won't print a warning message about attempting to load CA certificates. This is unfortunate. But there appears to be little we can do :/

extdiff: escape path for docstring (issue5301) The existing code (a) assumed path would be specified in encoding.encoding and (b) assumed unicode() objects wouldn't cause other parts of Mercurial to blow up. Both are dangerous assumptions. Since we don't know the encoding of path and can't pass non-ASCII through docstrings, just escape the path and drop the early _(). Will have to suffice until we can teach docstrings to handle UTF-8b escaping. This has the side-effect that the line containing the path is now variable by the time it reaches _() and thus can't be translated.

update: fix bug when update tries to modify folder symlink In 1e4512eac59e0114bc60ecbcdc4157fc0fa0439d, I introduced a new bug: when a symlink points to a folder in commit A and to another folder in commit B, while updating from A to B, Mercurial will try to use removedir on this symlink, which will fail. This is a very bad bug, since it basically renders symlinks to folders unusable in repos. Added test case fails without a fix and passes with it.

spartan: make annotate popup use theme colors

monoblue: make annotate popup use theme colors

gitweb: make annotate popup use theme colors

paper: make annotate popup use theme colors

templatekw: fix join format of parents keyword (issue5292) Since the default joinfmt() can't process a dict of multiple keywords, we need a dedicated joinfmt for showparents(). Unlike revset(), parents are formatted as '{rev}:{node|formatnode}' by default. We copy the default formatting just like showextras() and showfilecopies() do.

templatekw: fix join format of revset() function It's been broken since e4609ec959f8, which made makemap() return a dict of multiple keywords. Because the default joinfmt() randomly picks one item from a dict, we have to make revset() select d[name] explicitly.

cmdutil: warnings not issued in remove if subrepopath overlaps Previously a subrepository "sub" would cause no warnings to be issued for a file "subnot/a" if it is not removed when calling: hg remove -S "subnot/a"

merge with i18n

i18n-pt_BR: synchronized with 519bb4f9d3a4

sslutil: improve messaging around unsupported protocols (issue5303) There are various causes for the inability to negotiate common SSL/TLS protocol between client and server. Previously, we had a single, not very actionable warning message for all of them. As people encountered TLS 1.0 servers in real life, it was quickly obvious that the existing messaging was inadequate to help users rectify the situation. This patch makes the warning messages much more verbose in hopes of making them more actionable while simultaneously encouraging users and servers to adopt better security practices. This messaging flirts with the anti-pattern of "never blame the user" by signaling out poorly-configured servers. But if we're going to disallow TLS 1.0 by default, I think we need to say *something* or people are just going to blame Mercurial for not being able to connect. The messaging tries to exonerate Mercurial from being the at fault party by pointing out the server is the entity that doesn't support proper security (when appropriate, of course).

sslutil: capture string string representation of protocol This will be used in a subsequent patch to improve messaging.

sslutil: allow TLS 1.0 when --insecure is used --insecure is our psuedo-supported footgun for disabling connection security. The flag already disables CA verification. I think allowing the use of TLS 1.0 when specified is appropriate.

hg: copy [hostsecurity] options to remote ui instances (issue5305) TIL that ui instances for remote/peer repos don't automagically inherit config options from .hg/hgrc files. This patch makes remote ui instances inherit options from the [hostsecurity] section. We were already inheriting options from [hostfingerprints] and [auth]. So adding [hostsecurity] to the list seems appropriate.

rbc: fix superfluous rebuilding from scratch - don't abuse self._rbcnamescount The code used self._rbcnamescount as if it was the length of self._names ... but actually it is just the number of good entries on disk. This caused the cache to be populated inefficiently. In some cases very inefficiently. Instead of checking the length before lookup, just try a lookup in self._names - that is also in most cases faster. Comments and debug messages are tweaked to help understanding the issue and the fix.

rbc: test case for incorrect and too aggressive invalidation of invalid caches

util: better handle '-' in version string (issue5302) versiontuple() was previously only splitting on '+' and strings like "3.9-rc" were causing it to misreport the version as (3, None). By splitting on either '+' or '-' we can handle our version strings with "-rc" in them.

convert: update use of deprecated bzrlib property The inventory property was deprecated in favor of root_inventory in bzr 2.5.0. Current version is 2.7.0. I noticed this when testing locally on Python 2.6.9, which has warnings turned on by default. The failure that occurs without this patch can be seen on Python 2.7 by running with warnings enabled: $ PYTHONWARNINGS=::DeprecationWarning make 'test-convert-bzr*'

hghave: fix typo of sslutil.supportedprotocols

rebase: turn rebase revs into set before filtering obsolete When the inhibit extension from mutable-history is enabled, it attempts to iterate over the rebaseset to prevent the nodes being rebased from being marked obsolete. This happens at the same time as rebase's _filterobsoleterevs function trying to iterate over the rebaseset to figure out which ones are obsolete. The two of these iterating over the same revset generatorset cause a 'generator already executing' exception. This is probably a flaw in the revset implementation, since iterating over the same set twice should be supported. This regression was introduced in 5d16ebe7b14, since it changed _filterobsoleterevs to be called before the rebaseset was turned into a set(). For now let’s just make the rebaseset an actual set again before calling that function. This was caught by the inhibit tests. The relevant call stack from test-inhibit.t: File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 285, in _preparenewrebase obsrevs = _filterobsoleterevs(self.repo, rebaseset) File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/inhibit.py", line 197, in _filterobsoleterevswrap r = orig(repo, rebasesetrevs, *args, **kwargs) File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 1380, in _filterobsoleterevs return set(r for r in revs if repo[r].obsolete()) File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 1380, in <genexpr> return set(r for r in revs if repo[r].obsolete()) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3079, in _iterordered val2 = next(iter2) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3417, in gen yield nextrev() File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3424, in _consumegen for item in self._gen: File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 71, in iterate cl = repo.changelog File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 319, in changelog revs = filterrevs(unfi, self.filtername) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 261, in filterrevs repo.filteredrevcache[filtername] = func(repo.unfiltered()) File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/directaccess.py", line 65, in _computehidden hidden = repoview.filterrevs(repo, 'visible') File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 261, in filterrevs repo.filteredrevcache[filtername] = func(repo.unfiltered()) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 175, in computehidden hideable = hideablerevs(repo) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 33, in hideablerevs return obsolete.getrevs(repo, 'obsolete') File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/obsolete.py", line 1097, in getrevs repo.obsstore.caches[name] = cachefuncs[name](repo) File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/inhibit.py", line 255, in _computeobsoleteset if getrev(n) not in blacklist: File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3264, in __contains__ return x in self._r1 or x in self._r2 File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3348, in __contains__ for l in self._consumegen(): File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3424, in _consumegen for item in self._gen: ValueError: generator already executing

commandserver: update comment about setpgid Now setpgid has 2 main purposes: better handling for terminal-generated SIGTSTP, SIGINT, and process-exit-generated SIGHUP. Update the comment to explain things more clearly.

chg: forward SIGINT, SIGHUP to process group These signals are meant to send to a process group, instead of a single process: SIGINT is usually emitted by the terminal and sent to the process group. SIGHUP usually happens to a process group if termination of a process causes that process group to become orphaned. Before this patch, chg will only forward these signals to the single server process. This patch changes it to the server process group. This will allow us to properly kill processes started by the forked server process, like a ssh process. The behavior difference can be observed by setting SSH_ASKPASS to a dummy script doing "sleep 100" and then run "chg push ssh://dest-need-password-auth". Before this patch, the first Ctrl+C will kill the hg process while ssh-askpass and ssh will remain alive. This patch will make sure they are killed properly.

Added signature for changeset 519bb4f9d3a4

Added tag 3.9-rc for changeset 519bb4f9d3a4

merge default into stable for 3.9 code freeze

rbc: fix invalid rbc-revs entries caused by missing cache growth It was in some cases possible to end up writing to the cache file without growing it first. The range assignment in _setcachedata would append instead of writing at the requested position and thus write the new record in the wrong place. To fix this, we avoid looking up in too small caches, and when growing the cache, do it right before writing the new record to it so we know it has been done correctly.

rbc: test case for cache file not growing correctly, causing bad new entries

chg: handle EOF reading data block We recently discovered a case in production that chg uses 100% CPU and is trying to read data forever: recvfrom(4, "", 1814012019, 0, NULL, NULL) = 0 Using gdb, apparently readchannel() got wrong data. It was reading in an infinite loop because rsize == 0 does not exit the loop, while the server process had ended. (gdb) bt #0 ... in recv () at /lib64/libc.so.6 #1 ... in readchannel (...) at /usr/include/bits/socket2.h:45 #2 ... in readchannel (hgc=...) at hgclient.c:129 #3 ... in handleresponse (hgc=...) at hgclient.c:255 #4 ... in hgc_runcommand (hgc=..., args=<optimized>, argsize=<optimized>) #5 ... in main (argc=...486922636, argv=..., envp=...) at chg.c:661 (gdb) frame 2 (gdb) p *hgc $1 = {sockfd = 4, pid = 381152, ctx = {ch = 108 'l', data = 0x7fb05164f010 "st):\nTraceback (most recent call last):\n" "Traceback (most recent call last):\ne", maxdatasize = 1814065152," " datasize = 1814064225}, capflags = 16131} This patch addresses the infinite loop issue by detecting continuously empty responses and abort in that case. Note that datasize can be translated to ['l', ' ', 'l', 'a']. Concatenate datasize and data, it forms part of "Traceback (most recent call last):". This may indicate a server-side channeledoutput issue. If it is a race condition, we may want to use flock to protect the channels.

sslutil: more robustly detect protocol support The Python ssl module conditionally sets the TLS 1.1 and TLS 1.2 constants depending on whether HAVE_TLSv1_2 is defined. Yes, these are both tied to the same constant (I would think there would be separate constants for each version). Perhaps support for TLS 1.1 and 1.2 were added at the same time and the assumption is that OpenSSL either has neither or both. I don't know. As part of developing this patch, it was discovered that Apple's /usr/bin/python2.7 does not support TLS 1.1 and 1.2 (only TLS 1.0)! On OS X 10.11, Apple Python has the modern ssl module including SSLContext, but it doesn't appear to negotiate TLS 1.1+ nor does it expose the constants related to TLS 1.1+. Since this code is doing more robust feature detection (and not assuming modern ssl implies TLS 1.1+ support), we now get TLS 1.0 warnings when running on Apple Python. Hence the test changes. I'm not super thrilled about shipping a Mercurial that always whines about TLS 1.0 on OS X. We may want a follow-up patch to suppress this warning.

osutil: add darwin-only version of os.listdir using cffi

url: drop support for proxying HTTP (not HTTPS) over CONNECT tunneling It's been broken since cca59ef27e60, which made ui argument mandatory. I've tried several combinations of HTTP/HTTPS proxying on old/new Python versions, but I couldn't figure out how to reach this code path. Also, wrapping HTTP connection by SSLSocket seems wrong. My understanding is that self.realhostport is set by _generic_start_transaction() if HTTPS connection is tunneled. This patch removes proxy tunneling from httpconnection.connect() assuming that it was dead code from the beginning. Note that HTTPS over tunneling should be handled by httpsconnection class.

chgserver: rename private functions and variables of chgunixservicehandler self.address has been reanmed to self._realaddress to clarify that it can be different from the address argument.

chgserver: refactor initialization of real/base addresses Instead of overwriting self.address, calculate it from the address argument, which is the base address.

chgserver: reorder functions in chgunixservicehandler This should make it slightly easier to follow the call path.

chgserver: use ui.debug() to print server debug messages commandserver.log() is noop at this time because no client connection is established.

ssl: remove special case of web.cacerts=! from remoteui() It was introduced by b76d8c641746, which is no longer necessary thanks to recent refactoring of sslutil including ef316c653b7f.

bundle2: store changeset count when creating file bundles The bundle2 changegroup part has an advisory param saying how many changesets are in the part. Before this patch, we were setting this part when generating bundle2 parts via the wire protocol but not when generating local bundle2 files. A side effect of not setting the changeset count part is that progress bars don't work when applying changesets. As the tests show, this impacted clone bundles, shelve, backup bundles, `hg unbundle`, and anything touching bundle2 files. This patch adds a backdoor to allow us to pass state from changegroup generation into the unbundler. We store the number of changesets in the changegroup in this state and use it to populate the aforementioned advisory part parameter when generating the bundle2 bundle. I concede that I'm not thrilled by how state is being passed in changegroup.py (it feels a bit hacky). I would love to overhaul the rather confusing set of functions in changegroup.py with something that passes rich objects around instead of e.g. low-level generators. However, given the code freeze for 3.9 is imminent, I'd rather not undertake this endeavor right now. This feels like the easiest way to get the parameter added to the changegroup part.

util: implement a deterministic __repr__ on sortdict `hg debugbundle` is calling repr() on bundle2 part params, which are now util.sortdict instances. Unfortunately, repr() doesn't appear to be deterministic for util.sortdict. So, we implement one. We include the type name because that's the common convention for __repr__ implementations. Having the type name in `hg debugbundle` is a bit ugly. But it's a debug command and I don't care enough to fix it.

bundle2: use a sorted dict for holding parameters An upcoming change that introduces a 2nd part parameter to a part reveals that `hg debugbundle` isn't deterministic because parameters are stored on n plain, unsorted dict. While we could change that command to sort before output, I think the more important underlying issue is that bundle2 reading is taking an ordered data structure and converting it to an unordered one. Plugging in util.sortdict() fixes that problem while preserving API compatibility. This patch also appears to shine light on the fact that we don't have tests verifying parts with multiple parameters roundtrip correctly. That would be a good thing to test (and fuzz)... someday.

wireproto: extract repo filtering to standalone function As part of teaching Mozilla's replication extension to better handle repositories with obsolescence data, I encountered a few scenarios where I wanted built-in wire protocol commands from replication clients to operate on unfiltered repositories so they could have access to obsolete changesets. While the undocumented "web.view" config option provides a mechanism to choose what filter/view hgweb operates on, this doesn't apply to wire protocol commands because wireproto.dispatch() is always operating on the "served" repo. This patch extracts the line for obtaining the repo that wireproto commands operate on to its own function so extensions can monkeypatch it to e.g. return an unfiltered repo. I stopped short of exposing a config option because I view the use case for changing this as a niche feature, best left to the domain of extensions.

url: add distribution and version to user-agent request header (BC) As a server operator, I've always wanted to know what Mercurial version clients are running so I can track version adoption and make informed decisions about which versions of Mercurial to support in extensions. Unfortunately, there is no easy way to discern this today: the best you can do is look for high-level feature usage (e.g. bundle2) or sniff capabilities from bundle2 commands. And these things aren't changed frequently enough to tell you anything that interesting. Nearly every piece of software talking HTTP sends its version in the user agent. This includes web browsers, curl, and even Git. This patch adds the distribution name and version to the user-agent HTTP request header. We choose "Mercurial" for the distribution name because that seems appropriate. The version string comes from __version__. The value is inside parenthesis for a few reasons: * The version *may* contain spaces * Alternate forms like "Mercurial/<version>" imply structure and since the user agent should not be used by servers for protocol or feature negotiation/detection, we don't want to even give the illusion that the value should be parsed. A free form field is the most hostile to parsing. Flagging the patch as BC so it shows up in release notes. This change should be backwards compatible. But I wouldn't be surprised if a server somewhere is filtering on the exact old user agent string. So I want to make noise about this change.

commandserver: use SOMAXCONN as queue size of pending connections The old value 5 was arbitrary chosen. Since there's no practical reason to limit the backlog, this patch simply uses SOMAXCONN as a value large enough.

commandserver: rename _serveworker() to _runworker() "run" sounds more natural as the function does never listen for new connection.

commandserver: separate initialization and cleanup of forked process Separated _initworkerprocess() and _serverequest() can be reused when implementing a prefork service.

commandserver: unindent superfluous "if True" blocks

pycompat: make pycompat demandimport friendly pycompat.py includes hack to import modules whose names are changed in Python 3. We use try-except to load module according to the version of python. But this method forces us to import the modules to raise an ImportError and hence making it demandimport unfriendly. This patch changes the try-except blocks to a single if-else block. To avoid test-check-pyflakes.t complain about unused imports, pycompat.py is excluded from the test.

run-tests: make --local set --with-chg if --chg is used --local should work with chg as well.

run-tests: allow --local to set multiple attributes This is to make the next patch easier to review. It does not change logic.

chg: add pgid to hgclient struct The previous patch makes the server tell the client its pgid. This patch stores it in hgclient_t and adds a function to get it.

commandserver: send pgid in hello message See the next patches for why we need it.

tests: update test certificate generation instructions Suggestions from Anton Shestakov and Julien Cristau to use -subj and faketime, respectively.

sslutil: move comment about protocol constants protocolsettings() is the appropriate place for this comment.

sslutil: support defining cipher list Python 2.7 supports specifying a custom cipher list to TLS sockets. Advanced users may wish to specify a custom cipher list to increase security. Or in some cases they may wish to prefer weaker ciphers in order to increase performance (e.g. when doing stream clones of very large repositories). This patch introduces a [hostsecurity] config option for defining the cipher list. The help documentation states that it is for advanced users only. Honestly, I'm a bit on the fence about providing this because it is a footgun and can be used to decrease security. However, there are legitimate use cases for it, so I think support should be provided.

hghave: add test for Python 2.7+ Setting ciphers in the ssl module requires Python 2.7. Surprisingly, we didn't have a test for running on Python 2.7.