bundlerepo: use for loop over iterator instead of while True The iter() builtin has a neat pattern where you give it a callable of no arguments and a sentinel value, and you can then loop over the function calls like a normal iterator. This cleans up the code a little.

localrepo: jettison now-unused dirlog() method from localrepo

repair: build dirlogs using manifest, rather than repo shortcut method As before, this was rarely used, so let's get rid of the convenience method.

cmdutil: open dirlogs via manifest property, not via repo This was a convenience method that was rarely used, so let's get rid of it.

wirepeer: rename confusing `source` parameter It's named "url" everyplace else this method is defined, so let's be consistent.

develwarn: use the lock helper in local repo We have a helper function to know if a lock is taken. When checking lock usage we now use it instead of manually accessing the locks.

debugextension: change "testedwith" to a list (BC) It wasn't a list since the formatter couldn't process a list. We have no such problem now and the -T option is experimental, so we can change it.

debugextensions: unindent nested if

debugextensions: give short name to util.version()

debugextensions: simply keep testedwith variable as a list There should be no need to distinguish [] and None.

test-treemanifest: ensure manifest command isn't broken I realized we weren't testing this while hunting a broken manifest command bug that ended up being narrowhg's fault.

py3: use unicode literals in pure/osutil.py The first element of _fields_ tuples must be a str in Python 3. Also fix some function calls that were also expecting str.

py3: use unicode literals in crecord.py

py3: use unicode literals in changelog.py collections.namedtuple type and field names must be str in Python 3. Our custom module importer was rewriting them to bytes literals, making this fail. In addition, __slots__ values must also be unicode.

mpatch: write a cffi version of mpatch.patches

mpatch: remove dependency on Python.h in mpatch.c Now all the CPython-related stuff are referenced only from mpatch_module.c with mpatch.c being freely usable from a future cffi module

mpatch: split mpatch into two files

mpatch: provide things that will be exported later with a prefixed name For cffi a bunch of mpatch functions need to be visible through a .h file. This change renames them so it won't create potential c namespace conflicts.

mpatch: change Py_ssize_t to ssize_t in places that will be later copied

discovery: move code to create outgoing from roots and heads changegroup.changegroupsubset() contained somewhat low-level code for constructing an "outgoing" instance from a list of roots and heads nodes. It feels like discovery.py is a more appropriate location for this code. This code can definitely be optimized, as outgoing.missing will recompute the set of changesets we've already discovered from cl.between(). But code shouldn't be refactored during a move, so I've simply inserted a TODO calling attention to that.

bundle2: remove 'experimental.bundle2-exp' boolean config (BC) All users are migrated to 'devel.legacy.exchange', we can clean up the experimental namespace. Marking as (BC) because I know some large installation have bundle2 off and I want to make sure they notice the change.

tests: remove all remaining usage of experimental.bundle2-exp The only remaining usage of the experimental config were enforcing bundle2 on. These are very old remains of when bundle2 was off by default. This was also useful to highlight the fact that this was a bundle2 run and that a bundle1 one was nearby. However, we want a future developer working on bundle3 to notice possible output/behavior change on these tests and take them in account. So we do not enforce bundle2 for these runs. We leave a comment around to make sure dev still notice the bundle1 version.

tests: use 'legacy.exchange' option in various mixed tests The new option will stay around. The experimental option was only meant to be temporary. We update various tests that validate both bundle1 and bundle2 version side by side. This changeset only takes care of enforcing bundle1. The other use of 'experimental.bundle2-exp=True' will be taken care of in the next changeset.

tests: use 'legacy.exchange' option in various bundle1 tests The new option will stay around. The experimental option was only meant to be temporary.

tests: use 'legacy.exchange' option in 'test-bundle2-exchange.t' The new option will stay around. The experimental option was only meant to be temporary.

bundlerepo: also read the 'devel.legacy.exchange' config Bundlerepo does its own bundle2 related logic.

bundle2: add a devel option controling bundle version used for exchange We need an official way to force bundle1 to be used in test. We introduce a new option 'devel.legacy.exchange' to control this. When specified, this option will control the list of bundle version Mercurial consider when exchanging with a peer. Current valid value are 'bundle1' and 'bundle2'. Using this option in all tests will allow us to remove the 'experimental.bundle2-exp' option. We will simplify the code once the experimental option is dropped.

bundle2: rename the _canusebundle2 method to _forcebundle1 We rename and invert the logic of the _canusebundle2 utility. The idea here is that we need to have a way to enforce the use of bundle1 in the tests. The Mercurial philosophy is to try to use the best method available. Currently that best method is bundle2, but this might change in the future. Therefore expressing "do not use bundle2" is a loosy way to say "use bundle 1" and will likely create issue in the future. As the config option will be explicitly about bundle1, we rename the function beforehand to align with this. This will make the life of a future developer working on bundle3 easier.

tests: remove bundle2 activation from test-bookmark-pushpull.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: update bugzilla link in test-issue1175.t

journal: use fm.formatdate() to pass date tuple in appropriate type (BC)

formatter: add function to convert date tuple to appropriate format

journal: use fm.formatlist() to pass hashes in appropriate type (BC)

formatter: add function to convert list to appropriate format (issue5217) Before, it wasn't possible for formatter to handle array structure other than date tuple. We've discussed that at the last sprint, which ended we would probably want to allow only templatable data structure, i.e. a list of dicts: data(tags=[{'tag': a}, {'tag': b}, ...]) Unfortunately, it turned out not working well with template functions: "{ifcontains(a, tags, ...)}" ^^^^^^^^^^^^^^^^^^ "a in tags", where tags should be a plain list/set of tags So the formatter must at least know if the type [{}] was constructed from a plain list or was actually a list of dicts. This patch introduces new explicit interface to convert an array structure to an appropriate data type for the current formatter, which can be used as follows: fm.write('tags', _('tags: %s\n'), fm.formatlist(tags, name='tag')) No separate fm.data() call should be necessary.

journal: use fm.hexfunc() to get full hash in JSON/template output (BC) We generally do that.

py3: update test-check-py3-compat.t output The lower part of test-check-py3-compat.t runs only on py3 and hence its remain unchanged. Hence this patch updates the output so that change in output in the next patches will be only related to the change in the patch.

deprecation: enforce thew 'tr' argument of 'dirstate.write' (API) Compatibility was meant to be drop after 3.9 is released.

tests: remove initial bundle2 enabling in various bundle2 tests Bundle 2 is enable by default since 3.6 so we don't need this anymore. The remaining use of 'experimental.bundle2-exp' are there to test legacy behavior with bundle 1. Future patches will introduce a way outside of experimental to test that.

tests: remove bundle2 activation from test-largefiles.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-tags.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

templatekw: change joinfmt to a mandatory argument of _hybrid object We've fixed several bugs caused by the misuse of the default joinfmt. Make it more explicit to prevent future bugs. dict.values()[0] is replaced by dict[element] as showlist() knows what the key is.

url: drop compatibility wrapper of socket.create_connection() It should be available on Python 2.6+.

doc: remove double imports of inspect from hgmanpage.py

merge: concatenate default conflict marker at parsing phase of .py "+" operations are unnecessary.

tests: remove bundle2 activation from test-push-warn.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-phases-exchanges.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-obsolete.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-http-proxy.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-hook.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

tests: remove bundle2 activation from test-acl.t This is an old config that predate bundle2 on by default. This should have been remove after Mercurail 3.6 got released.

merge with stable

Added signature for changeset 149433e68974

Added tag 3.9.2 for changeset 149433e68974

py3: update test expectation on stable

grep: rewrite help to better document current (confusing) behavior

crecord: properly handle files with No newline at eof (issue5268) Yes, this bug was a single character with the wrong case...

annotate: pre-calculate the "needed" dictionary (issue5360) The "needed" dict is used as a reference counter to free items in the giant "hist" dict. However, currently it is not very accurate and can lead to dropping "hist" items unnecessarily, for example, with the following DAG, -3- / \ 0--1--2--4-- The current algorithm will visit and calculate rev 1 twice, undesired. And it tries to be smart by clearing rev 1's parents: "pcache[1] = []" at the time hist[1] being accessed (note: hist[1] needs to be used twice, by rev 2 and rev 3). It can result in incorrect results if p1 of rev 4 deletes chunks belonging to rev 0. However, simply removing "needed" is not okay, because it will consume 10x memory: # without any change % HGRCPATH= lrun ./hg annotate mercurial/commands.py -r d130a38 3>&2 [1] MEMORY 49074176 CPUTIME 9.213 REALTIME 9.270 # with "needed" removed MEMORY 637673472 CPUTIME 8.164 REALTIME 8.249 This patch moves "needed" (and "pcache") calculation to a separate DFS to address the issue. It improves perf and fixes issue5360 by correctly reusing hist, while maintaining low memory usage. Some additional attempt has been made to further reduce memory usage, like changing "pcache[f] = []" to "del pcache[f]". Therefore the result can be both faster and lower memory usage: # with this patch applied MEMORY 47575040 CPUTIME 7.870 REALTIME 7.926 [1]: lrun is a lightweight sandbox built on Linux cgroup and namespace. It's used to measure CPU and memory usage here. Source code is available at github.com/quark-zju/lrun.

Added signature for changeset ccd436f7db6d

Added tag 3.9.1 for changeset ccd436f7db6d

bundle2: localize handleoutput remote prompts Code archaeology suggests that there was no good reason for this not to be localized. 'remote: ' is already localized elsewhere.

i18n-pt_BR: synchronized with a12d13eac513

bundle2: fail faster when interrupted Before this patch, bundle2 application attempted to consume remaining bundle2 part data when the process is interrupted (SIGINT) or when sys.exit is called (translated into a SystemExit exception). This meant that if one of these occurred when applying a say 1 GB changegroup bundle2 part being downloaded over a network, it may take Mercurial *several minutes* to terminate after a SIGINT because the process is waiting on the network to stream megabytes of data. This is not a great user experience and a regression from bundle1. Furthermore, many process supervisors tend to only give processes a finite amount of time to exit after delivering SIGINT: if processes take too long to self-terminate, a SIGKILL is issued and Mercurial has no opportunity to clean up. This would mean orphaned locks and transactions. Not good. This patch changes the bundle2 application behavior to fail faster when an interrupt or system exit is requested. It does so by not catching BaseException (which includes KeyboardInterrupt and SystemExit) and by explicitly checking for these conditions in yet another handler which would also seek to the end of the current bundle2 part on failure. The end result of this patch is that SIGINT is now reacted to significantly faster: the active transaction is rolled back immediately without waiting for incoming bundle2 data to be consumed. This restores the pre-bundle2 behavior and makes Mercurial treat signals with the urgency they deserve.

revset: fix keyword arguments to go through optimization process Before, a keyvalue node was processed by the last catch-all condition of _optimize(). Therefore, topo.firstbranch=expr would bypass tree rewriting and would crash if an expr wasn't trivial.

help: add example of '[templates]' usage V2: - move from shortest() with minlength 8 to minlength 4 - mention [templates] in config.txt - better describe the difference between [templatealias] and [templates] V3: - choose a better example template

exchange: correctly specify url to unbundle (issue5145) This parameter is slightly confusingly named in wireproto, so it got mis-specified from the start as 'push' instead of the URL to which we are pushing. Sigh. I've got a patch for that which I'll mail separately since it's not really appropriate for stable. Fixes a regression in bundle2 from bundle1.

convert: move svn config initializer out of the module level The svn_config_get_config config call was being called at the module level, but had the potential to throw permission denied errors if ~/.subversion/servers was not readable. This could happen in certain test environments where the user permissions were very particular. This prevented the remotenames extension from loading, since it imports convert's hg module, which imports convert's subversion module, which calls this. The config is only ever used from this one constructor, so let's just move it in to there.

osx: update bugzilla link in readme

win32: update link to mailing list in readme

win32: update wiki link in mercurial.ini

debian: update source URL in copyright file

help: update link to wiki/CommandServer

Added signature for changeset 299546f84e68

Added tag 3.9 for changeset 299546f84e68

merge with i18n

doc: make previous line of certificate example end with "::" Before this patch, certificate example is formatted just as normal text.

doc: fix incorrect use of rst hg role in help text

doc: use field rst syntax to show keywords in debugdeltachain help correctly List of available keywords is well formatted as a list of fields in doc string, but is formatted as just normal text in online help output.

revset: refactor to make xgettext put i18n comments into hg.pot file xgettext expects both "_()" and (a part of) text to be placed at just next line of "i18n:" comment.

doc: omit useless _() invocation In this case, column positioning isn't needed for i18n, too. Maybe, check-code warning "missing _() in ui message" caused this useless _() invocation in 92d37fb3f1aa.

doc: trim newline at the end of exception message

i18n-ja: synchronized with 6fd751fa58d3

demandimport: avoid infinite recursion at actual module importing (issue5304) Before this patch, importing C module on Windows environment causes infinite recursion call, if py2exe is used with -b2 option. At importing C module "a.b", extra hooking by zipextimporter of py2exe causes: 0. assumption before accessing "b" of "a": - built-in module object is created for "a", (= "a" is actually imported) - _demandmod is created for "a.b" as a proxy object, and (= "a.b" is not yet imported) - an attribute "b" of "a" is initialized by the latter 1. invocation of __import__ via _hgextimport() in _demandmod._load() for "a.b" implies _demandimport() for "a.b" This is unintentional, because _demandmod might be returned by _hgextimport() instead of built-in module object. 2. _demandimport() at (1) is invoked with not context of "a", but context of zipextimporter Just after invocation of _hgextimport() in _demandimport(), an attribute "b" of the built-in module object for "a" is still bound to the proxy object for "a.b", because context of "a" isn't updated by actual importing "a.b". even though the built-in module object for "a.b" already appears in sys.modules. Therefore, chainmodules() returns _demandmod for "a.b", which is gotten from the attribute "b" of "a". 3. processfromitem() on "a.b" causes _demandmod._load() for "a.b" again _demandimport() takes context of "a" in this case. Therefore, attributes below are bound to built-in module object for "a.b", as expected: - "b" of built-in module object for "a" - _module of _demandmod for "a.b" 4. but _demandimport() invoked at (1) returns _demandmod object because _demandimport() just returns the object returned by chainmodules() at (3) above. 5. then, _demandmod._load() causes infinite recursion call _demandimport() returns _demandmod for "a.b", and it is "self" at _demandmod._load(). To avoid infinite recursion at actual module importing, this patch uses self._module, if _hgextimport() returns _demandmod itself. If _demandmod._module isn't yet bound at this point, execution should be aborted, because actual importing failed. In this patch, _demandmod._module is examined not on _demandimport() side, but on _demandmod._load() side, because: - the former has some exit points - only the latter uses _hgextimport(), except for _demandimport() BTW, this issue occurs only in the code path for non .py/.pyc files in zipextimporter (strictly speaking, in _memimporter) of py2exe. Even if zipextimporter is enabled, .py/.pyc files are handled by zipimporter, and it doesn't imply unintentional _demandimport() at invocation of __import__ via _hgextimport().

packagelib: do not remove packages directory in hggetversion (issue5262) People running packages related code probably do care about the content of this directory. In particular this shound fix the rpm builder process.

make: introduce a target to clean everything but packages Removing the 'packages' directory makes nightly builder life much harder.

url: avoid re-issuing incorrect password (issue3210) Some draconian IT setups lock accounts after a small number of incorrect password attempts. Mercurial's implementation of the urllib2 authentication was causing 5 retry attempts with the same credentials, without prompting the user. The code was attempting to check whether the authorization token had changed, but unfortunately was reading the misleading 'headers' member of the request instead of using the 'get_header' accessor. Modelled on fix for Python issue 8797: https://bugs.python.org/issue8797 https://hg.python.org/cpython/rev/30e8a8f22a2a

date: accept broader range of ISO 8601 time specs The "normal" ISO date/time includes a T between date and time. It also allows dropping the colons and seconds from the timespec. Add new patterns for these forms as well as tests.

date: parse ISO-style Z and +hh:mm timezone specs

date: refactor timezone parsing We want to be able to accept ISO 8601 style timezones that don't include a space separator, so we change the timezone parsing function to accept a full date string and return both the offset and the non-timezone portion.

tests: glob over ssl error We do this in the rest of the file. This bad line was introduced in 3fde328d0913.

keyword: avoid traceback when kwdemo is run outside a repo f0564402d059 causes a fatal AttributeError if kwdemo is run outside a repo because in the temporary repo creation repo is None and therefore cannot have a baseui attribute. In this case fall back to using ui. Add test case.

cmdutil: warnings not issued in cat if subrepopath overlaps Previously a subrepository "sub" would cause no warnings to be issued for a file "subnot/a", if it's not present in the corresponding changeset when calling: hg cat subnot/a

graft: use opts.get() consistently Make life easier for extension writers.

sslutil: work around SSLContext.get_ca_certs bug on Windows (issue5313) SSLContext.get_ca_certs() can raise "ssl.SSLError: unknown error (_ssl.c:636)" on Windows. See https://bugs.python.org/issue20916 for more info. We add a try..except that swallows the exception to work around this bug. If we encounter the bug, we won't print a warning message about attempting to load CA certificates. This is unfortunate. But there appears to be little we can do :/

extdiff: escape path for docstring (issue5301) The existing code (a) assumed path would be specified in encoding.encoding and (b) assumed unicode() objects wouldn't cause other parts of Mercurial to blow up. Both are dangerous assumptions. Since we don't know the encoding of path and can't pass non-ASCII through docstrings, just escape the path and drop the early _(). Will have to suffice until we can teach docstrings to handle UTF-8b escaping. This has the side-effect that the line containing the path is now variable by the time it reaches _() and thus can't be translated.

update: fix bug when update tries to modify folder symlink In 1e4512eac59e0114bc60ecbcdc4157fc0fa0439d, I introduced a new bug: when a symlink points to a folder in commit A and to another folder in commit B, while updating from A to B, Mercurial will try to use removedir on this symlink, which will fail. This is a very bad bug, since it basically renders symlinks to folders unusable in repos. Added test case fails without a fix and passes with it.

spartan: make annotate popup use theme colors

monoblue: make annotate popup use theme colors

gitweb: make annotate popup use theme colors

paper: make annotate popup use theme colors

templatekw: fix join format of parents keyword (issue5292) Since the default joinfmt() can't process a dict of multiple keywords, we need a dedicated joinfmt for showparents(). Unlike revset(), parents are formatted as '{rev}:{node|formatnode}' by default. We copy the default formatting just like showextras() and showfilecopies() do.

templatekw: fix join format of revset() function It's been broken since e4609ec959f8, which made makemap() return a dict of multiple keywords. Because the default joinfmt() randomly picks one item from a dict, we have to make revset() select d[name] explicitly.

cmdutil: warnings not issued in remove if subrepopath overlaps Previously a subrepository "sub" would cause no warnings to be issued for a file "subnot/a" if it is not removed when calling: hg remove -S "subnot/a"

merge with i18n

i18n-pt_BR: synchronized with 519bb4f9d3a4

sslutil: improve messaging around unsupported protocols (issue5303) There are various causes for the inability to negotiate common SSL/TLS protocol between client and server. Previously, we had a single, not very actionable warning message for all of them. As people encountered TLS 1.0 servers in real life, it was quickly obvious that the existing messaging was inadequate to help users rectify the situation. This patch makes the warning messages much more verbose in hopes of making them more actionable while simultaneously encouraging users and servers to adopt better security practices. This messaging flirts with the anti-pattern of "never blame the user" by signaling out poorly-configured servers. But if we're going to disallow TLS 1.0 by default, I think we need to say *something* or people are just going to blame Mercurial for not being able to connect. The messaging tries to exonerate Mercurial from being the at fault party by pointing out the server is the entity that doesn't support proper security (when appropriate, of course).

sslutil: capture string string representation of protocol This will be used in a subsequent patch to improve messaging.

sslutil: allow TLS 1.0 when --insecure is used --insecure is our psuedo-supported footgun for disabling connection security. The flag already disables CA verification. I think allowing the use of TLS 1.0 when specified is appropriate.

hg: copy [hostsecurity] options to remote ui instances (issue5305) TIL that ui instances for remote/peer repos don't automagically inherit config options from .hg/hgrc files. This patch makes remote ui instances inherit options from the [hostsecurity] section. We were already inheriting options from [hostfingerprints] and [auth]. So adding [hostsecurity] to the list seems appropriate.

rbc: fix superfluous rebuilding from scratch - don't abuse self._rbcnamescount The code used self._rbcnamescount as if it was the length of self._names ... but actually it is just the number of good entries on disk. This caused the cache to be populated inefficiently. In some cases very inefficiently. Instead of checking the length before lookup, just try a lookup in self._names - that is also in most cases faster. Comments and debug messages are tweaked to help understanding the issue and the fix.

rbc: test case for incorrect and too aggressive invalidation of invalid caches

util: better handle '-' in version string (issue5302) versiontuple() was previously only splitting on '+' and strings like "3.9-rc" were causing it to misreport the version as (3, None). By splitting on either '+' or '-' we can handle our version strings with "-rc" in them.

convert: update use of deprecated bzrlib property The inventory property was deprecated in favor of root_inventory in bzr 2.5.0. Current version is 2.7.0. I noticed this when testing locally on Python 2.6.9, which has warnings turned on by default. The failure that occurs without this patch can be seen on Python 2.7 by running with warnings enabled: $ PYTHONWARNINGS=::DeprecationWarning make 'test-convert-bzr*'

hghave: fix typo of sslutil.supportedprotocols

rebase: turn rebase revs into set before filtering obsolete When the inhibit extension from mutable-history is enabled, it attempts to iterate over the rebaseset to prevent the nodes being rebased from being marked obsolete. This happens at the same time as rebase's _filterobsoleterevs function trying to iterate over the rebaseset to figure out which ones are obsolete. The two of these iterating over the same revset generatorset cause a 'generator already executing' exception. This is probably a flaw in the revset implementation, since iterating over the same set twice should be supported. This regression was introduced in 5d16ebe7b14, since it changed _filterobsoleterevs to be called before the rebaseset was turned into a set(). For now let’s just make the rebaseset an actual set again before calling that function. This was caught by the inhibit tests. The relevant call stack from test-inhibit.t: File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 285, in _preparenewrebase obsrevs = _filterobsoleterevs(self.repo, rebaseset) File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/inhibit.py", line 197, in _filterobsoleterevswrap r = orig(repo, rebasesetrevs, *args, **kwargs) File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 1380, in _filterobsoleterevs return set(r for r in revs if repo[r].obsolete()) File "/tmp/hgtests.jgjrN5/install/lib/python/hgext/rebase.py", line 1380, in <genexpr> return set(r for r in revs if repo[r].obsolete()) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3079, in _iterordered val2 = next(iter2) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3417, in gen yield nextrev() File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3424, in _consumegen for item in self._gen: File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 71, in iterate cl = repo.changelog File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 319, in changelog revs = filterrevs(unfi, self.filtername) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 261, in filterrevs repo.filteredrevcache[filtername] = func(repo.unfiltered()) File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/directaccess.py", line 65, in _computehidden hidden = repoview.filterrevs(repo, 'visible') File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 261, in filterrevs repo.filteredrevcache[filtername] = func(repo.unfiltered()) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 175, in computehidden hideable = hideablerevs(repo) File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/repoview.py", line 33, in hideablerevs return obsolete.getrevs(repo, 'obsolete') File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/obsolete.py", line 1097, in getrevs repo.obsstore.caches[name] = cachefuncs[name](repo) File "/data/hgbuild/facebook-hg-rpms/mutable-history/hgext/inhibit.py", line 255, in _computeobsoleteset if getrev(n) not in blacklist: File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3264, in __contains__ return x in self._r1 or x in self._r2 File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3348, in __contains__ for l in self._consumegen(): File "/tmp/hgtests.jgjrN5/install/lib/python/mercurial/revset.py", line 3424, in _consumegen for item in self._gen: ValueError: generator already executing

commandserver: update comment about setpgid Now setpgid has 2 main purposes: better handling for terminal-generated SIGTSTP, SIGINT, and process-exit-generated SIGHUP. Update the comment to explain things more clearly.

chg: forward SIGINT, SIGHUP to process group These signals are meant to send to a process group, instead of a single process: SIGINT is usually emitted by the terminal and sent to the process group. SIGHUP usually happens to a process group if termination of a process causes that process group to become orphaned. Before this patch, chg will only forward these signals to the single server process. This patch changes it to the server process group. This will allow us to properly kill processes started by the forked server process, like a ssh process. The behavior difference can be observed by setting SSH_ASKPASS to a dummy script doing "sleep 100" and then run "chg push ssh://dest-need-password-auth". Before this patch, the first Ctrl+C will kill the hg process while ssh-askpass and ssh will remain alive. This patch will make sure they are killed properly.

Added signature for changeset 519bb4f9d3a4

Added tag 3.9-rc for changeset 519bb4f9d3a4

merge default into stable for 3.9 code freeze

rbc: fix invalid rbc-revs entries caused by missing cache growth It was in some cases possible to end up writing to the cache file without growing it first. The range assignment in _setcachedata would append instead of writing at the requested position and thus write the new record in the wrong place. To fix this, we avoid looking up in too small caches, and when growing the cache, do it right before writing the new record to it so we know it has been done correctly.

rbc: test case for cache file not growing correctly, causing bad new entries

chg: handle EOF reading data block We recently discovered a case in production that chg uses 100% CPU and is trying to read data forever: recvfrom(4, "", 1814012019, 0, NULL, NULL) = 0 Using gdb, apparently readchannel() got wrong data. It was reading in an infinite loop because rsize == 0 does not exit the loop, while the server process had ended. (gdb) bt #0 ... in recv () at /lib64/libc.so.6 #1 ... in readchannel (...) at /usr/include/bits/socket2.h:45 #2 ... in readchannel (hgc=...) at hgclient.c:129 #3 ... in handleresponse (hgc=...) at hgclient.c:255 #4 ... in hgc_runcommand (hgc=..., args=<optimized>, argsize=<optimized>) #5 ... in main (argc=...486922636, argv=..., envp=...) at chg.c:661 (gdb) frame 2 (gdb) p *hgc $1 = {sockfd = 4, pid = 381152, ctx = {ch = 108 'l', data = 0x7fb05164f010 "st):\nTraceback (most recent call last):\n" "Traceback (most recent call last):\ne", maxdatasize = 1814065152," " datasize = 1814064225}, capflags = 16131} This patch addresses the infinite loop issue by detecting continuously empty responses and abort in that case. Note that datasize can be translated to ['l', ' ', 'l', 'a']. Concatenate datasize and data, it forms part of "Traceback (most recent call last):". This may indicate a server-side channeledoutput issue. If it is a race condition, we may want to use flock to protect the channels.

sslutil: more robustly detect protocol support The Python ssl module conditionally sets the TLS 1.1 and TLS 1.2 constants depending on whether HAVE_TLSv1_2 is defined. Yes, these are both tied to the same constant (I would think there would be separate constants for each version). Perhaps support for TLS 1.1 and 1.2 were added at the same time and the assumption is that OpenSSL either has neither or both. I don't know. As part of developing this patch, it was discovered that Apple's /usr/bin/python2.7 does not support TLS 1.1 and 1.2 (only TLS 1.0)! On OS X 10.11, Apple Python has the modern ssl module including SSLContext, but it doesn't appear to negotiate TLS 1.1+ nor does it expose the constants related to TLS 1.1+. Since this code is doing more robust feature detection (and not assuming modern ssl implies TLS 1.1+ support), we now get TLS 1.0 warnings when running on Apple Python. Hence the test changes. I'm not super thrilled about shipping a Mercurial that always whines about TLS 1.0 on OS X. We may want a follow-up patch to suppress this warning.

osutil: add darwin-only version of os.listdir using cffi

url: drop support for proxying HTTP (not HTTPS) over CONNECT tunneling It's been broken since cca59ef27e60, which made ui argument mandatory. I've tried several combinations of HTTP/HTTPS proxying on old/new Python versions, but I couldn't figure out how to reach this code path. Also, wrapping HTTP connection by SSLSocket seems wrong. My understanding is that self.realhostport is set by _generic_start_transaction() if HTTPS connection is tunneled. This patch removes proxy tunneling from httpconnection.connect() assuming that it was dead code from the beginning. Note that HTTPS over tunneling should be handled by httpsconnection class.

chgserver: rename private functions and variables of chgunixservicehandler self.address has been reanmed to self._realaddress to clarify that it can be different from the address argument.

chgserver: refactor initialization of real/base addresses Instead of overwriting self.address, calculate it from the address argument, which is the base address.

chgserver: reorder functions in chgunixservicehandler This should make it slightly easier to follow the call path.

chgserver: use ui.debug() to print server debug messages commandserver.log() is noop at this time because no client connection is established.

ssl: remove special case of web.cacerts=! from remoteui() It was introduced by b76d8c641746, which is no longer necessary thanks to recent refactoring of sslutil including ef316c653b7f.

bundle2: store changeset count when creating file bundles The bundle2 changegroup part has an advisory param saying how many changesets are in the part. Before this patch, we were setting this part when generating bundle2 parts via the wire protocol but not when generating local bundle2 files. A side effect of not setting the changeset count part is that progress bars don't work when applying changesets. As the tests show, this impacted clone bundles, shelve, backup bundles, `hg unbundle`, and anything touching bundle2 files. This patch adds a backdoor to allow us to pass state from changegroup generation into the unbundler. We store the number of changesets in the changegroup in this state and use it to populate the aforementioned advisory part parameter when generating the bundle2 bundle. I concede that I'm not thrilled by how state is being passed in changegroup.py (it feels a bit hacky). I would love to overhaul the rather confusing set of functions in changegroup.py with something that passes rich objects around instead of e.g. low-level generators. However, given the code freeze for 3.9 is imminent, I'd rather not undertake this endeavor right now. This feels like the easiest way to get the parameter added to the changegroup part.

util: implement a deterministic __repr__ on sortdict `hg debugbundle` is calling repr() on bundle2 part params, which are now util.sortdict instances. Unfortunately, repr() doesn't appear to be deterministic for util.sortdict. So, we implement one. We include the type name because that's the common convention for __repr__ implementations. Having the type name in `hg debugbundle` is a bit ugly. But it's a debug command and I don't care enough to fix it.

bundle2: use a sorted dict for holding parameters An upcoming change that introduces a 2nd part parameter to a part reveals that `hg debugbundle` isn't deterministic because parameters are stored on n plain, unsorted dict. While we could change that command to sort before output, I think the more important underlying issue is that bundle2 reading is taking an ordered data structure and converting it to an unordered one. Plugging in util.sortdict() fixes that problem while preserving API compatibility. This patch also appears to shine light on the fact that we don't have tests verifying parts with multiple parameters roundtrip correctly. That would be a good thing to test (and fuzz)... someday.

wireproto: extract repo filtering to standalone function As part of teaching Mozilla's replication extension to better handle repositories with obsolescence data, I encountered a few scenarios where I wanted built-in wire protocol commands from replication clients to operate on unfiltered repositories so they could have access to obsolete changesets. While the undocumented "web.view" config option provides a mechanism to choose what filter/view hgweb operates on, this doesn't apply to wire protocol commands because wireproto.dispatch() is always operating on the "served" repo. This patch extracts the line for obtaining the repo that wireproto commands operate on to its own function so extensions can monkeypatch it to e.g. return an unfiltered repo. I stopped short of exposing a config option because I view the use case for changing this as a niche feature, best left to the domain of extensions.

url: add distribution and version to user-agent request header (BC) As a server operator, I've always wanted to know what Mercurial version clients are running so I can track version adoption and make informed decisions about which versions of Mercurial to support in extensions. Unfortunately, there is no easy way to discern this today: the best you can do is look for high-level feature usage (e.g. bundle2) or sniff capabilities from bundle2 commands. And these things aren't changed frequently enough to tell you anything that interesting. Nearly every piece of software talking HTTP sends its version in the user agent. This includes web browsers, curl, and even Git. This patch adds the distribution name and version to the user-agent HTTP request header. We choose "Mercurial" for the distribution name because that seems appropriate. The version string comes from __version__. The value is inside parenthesis for a few reasons: * The version *may* contain spaces * Alternate forms like "Mercurial/<version>" imply structure and since the user agent should not be used by servers for protocol or feature negotiation/detection, we don't want to even give the illusion that the value should be parsed. A free form field is the most hostile to parsing. Flagging the patch as BC so it shows up in release notes. This change should be backwards compatible. But I wouldn't be surprised if a server somewhere is filtering on the exact old user agent string. So I want to make noise about this change.

commandserver: use SOMAXCONN as queue size of pending connections The old value 5 was arbitrary chosen. Since there's no practical reason to limit the backlog, this patch simply uses SOMAXCONN as a value large enough.

commandserver: rename _serveworker() to _runworker() "run" sounds more natural as the function does never listen for new connection.

commandserver: separate initialization and cleanup of forked process Separated _initworkerprocess() and _serverequest() can be reused when implementing a prefork service.

commandserver: unindent superfluous "if True" blocks

pycompat: make pycompat demandimport friendly pycompat.py includes hack to import modules whose names are changed in Python 3. We use try-except to load module according to the version of python. But this method forces us to import the modules to raise an ImportError and hence making it demandimport unfriendly. This patch changes the try-except blocks to a single if-else block. To avoid test-check-pyflakes.t complain about unused imports, pycompat.py is excluded from the test.

run-tests: make --local set --with-chg if --chg is used --local should work with chg as well.

run-tests: allow --local to set multiple attributes This is to make the next patch easier to review. It does not change logic.

chg: add pgid to hgclient struct The previous patch makes the server tell the client its pgid. This patch stores it in hgclient_t and adds a function to get it.

commandserver: send pgid in hello message See the next patches for why we need it.

tests: update test certificate generation instructions Suggestions from Anton Shestakov and Julien Cristau to use -subj and faketime, respectively.

sslutil: move comment about protocol constants protocolsettings() is the appropriate place for this comment.

sslutil: support defining cipher list Python 2.7 supports specifying a custom cipher list to TLS sockets. Advanced users may wish to specify a custom cipher list to increase security. Or in some cases they may wish to prefer weaker ciphers in order to increase performance (e.g. when doing stream clones of very large repositories). This patch introduces a [hostsecurity] config option for defining the cipher list. The help documentation states that it is for advanced users only. Honestly, I'm a bit on the fence about providing this because it is a footgun and can be used to decrease security. However, there are legitimate use cases for it, so I think support should be provided.

hghave: add test for Python 2.7+ Setting ciphers in the ssl module requires Python 2.7. Surprisingly, we didn't have a test for running on Python 2.7.

spartan: make different blocks of annotated lines have different colors

monoblue: make different blocks of annotated lines have different colors

gitweb: make different blocks of annotated lines have different colors

paper: make different blocks of annotated lines have different colors

tests: check importing modules in perf.py for historical portability To check importing modules in perf.py for historical portability, this patch lists up files by "hg files" both for "1.2" and tip, and builds up "module whitelist" check from those files. This patch uses "1.2" as earlier side version of "module whitelist", because "mercurial.error" module is a blocker for loading perf.py with Mercurial earlier than 1.2, and just importing "mercurial.error" separately isn't enough.

tests: introduce check-perf-code.py to add extra checks on perf.py This patch introduces tests/check-perf-code.py as a preparation for adding extra checks on contrib/perf.py in subsequent patches (mainly, for historical portability). At this change, check-perf-code.py doesn't add any extra check, and is equal to check-code.py. This makes subsequent patch focus only on adding an extra check on perf.py check-perf-code.py. check-perf-code.py adds extra checks on perf.py by wrapping contrib/check-code.py, because "filtering" by check-code.py (e.g. normalize characters in string literal or comment line) is useful to simplify regexp for check, and avoid false positive matching.

check-code: move fixing up regexp into main procedure This patch makes an extra check pattern to be prepared by "_preparepats()" as similarly as existing patterns, if it is added to "checks" array before invocation of "main()" in check-code.py. This is a part of preparation for adding check-code.py extra checks by another python script in subsequent patch. This is also useful for SkeletonExtensionPlan. https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan

check-code: factor out boot procedure into main This is a part of preparation for adding check-code.py extra checks by another python script in subsequent patch. This is also useful for SkeletonExtensionPlan. https://www.mercurial-scm.org/wiki/SkeletonExtensionPlan

perf: import newer modules separately for earlier Mercurial demandimport of early Mercurial loads an imported module immediately, if a module is imported absolutely by "from a import b" style. Recent perf.py satisfies this condition, because it does: - have "from __future__ import absolute_import" line - use "from a import b" style for modules in "mercurial" package Before this patch, importing modules below prevents perf.py from being loaded by earlier Mercurial, because these aren't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9. - branchmap 2.5 (or bcee63733aad) - repoview 2.5 (or 3a6ddacb7198) - obsolete 2.3 (or ad0d6c2b3279) - scmutil 1.9 (or 8b252e826c68) For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with Mercurial earlier than 1.8 (or 61c9bc3da402). After this patch, "mercurial.error" is the only blocker in "from mercurial import" statement for loading perf.py with Mercurial earlier than 1.2. This patch ignores it, because just importing it separately isn't enough.

py3: conditionalize BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer import The BaseHTTPServer, SimpleHTTPServer and CGIHTTPServer has been merged into http.server in python 3. All of them has been merged as util.httpserver to use in both python 2 and 3. This patch adds a regex to check-code to warn against the use of BaseHTTPServer. Moreover this patch also includes updates to lower part of test-check-py3-compat.t which used to remain unchanged.

py3: re-implement the BaseHTTPServer.test() function The function is changed in python 3. So the latest version of function is re-implemented. One can look at https://hg.python.org/cpython/file/3.5/Lib/http/server.py#l1184 and https://hg.python.org/cpython/file/2.7/Lib/BaseHTTPServer.py#l590 to see the change

test-http: use sed instead of fixed-with cut for reading access.log Some systems (like FreeBSD jails) use something other than 127.0.0.1 for localhost, and it's not safe to assume it'll always be the same width. Using sed with a replacement like this sidesteps the problem.

test-serve: add missing globs check-code missed this because of the closing ) in the "bound to" message.

tests: glob whitespace between path and OK in unzip(1) output FreeBSD's unzip(1) uses tabs instead of a run of spaces.

sslutil: print a warning when using TLS 1.0 on legacy Python Mercurial now requires TLS 1.1+ when TLS 1.1+ is supported by the client. Since we made the decision to require TLS 1.1+ when running with modern Python versions, it makes sense to do something for legacy Python versions that only support TLS 1.0. Feature parity would be to prevent TLS 1.0 connections out of the box and require a config option to enable them. However, this is extremely user hostile since Mercurial wouldn't talk to https:// by default in these installations! I can easily see how someone would do something foolish like use "--insecure" instead - and that would be worse than allowing TLS 1.0! This patch takes the compromise position of printing a warning when performing TLS 1.0 connections when running on old Python versions. While this warning is no more annoying than the CA certificate / fingerprint warnings in Mercurial 3.8, we provide a config option to disable the warning because to many people upgrading Python to make the warning go away is not an available recourse (unlike pinning fingerprints is for the CA warning). The warning appears as optional output in a lot of tests.

sslutil: require TLS 1.1+ when supported Currently, Mercurial will use TLS 1.0 or newer when connecting to remote servers, selecting the highest TLS version supported by both peers. On older Pythons, only TLS 1.0 is available. On newer Pythons, TLS 1.1 and 1.2 should be available. Security professionals recommend avoiding TLS 1.0 if possible. PCI DSS 3.1 "strongly encourages" the use of TLS 1.2. Known attacks like BEAST and POODLE exist against TLS 1.0 (although mitigations are available and properly configured servers aren't vulnerable). I asked Eric Rescorla - Mozilla's resident crypto expert - whether Mercurial should drop support for TLS 1.0. His response was "if you can get away with it." Essentially, a number of servers on the Internet don't support TLS 1.1+. This is why web browsers continue to support TLS 1.0 despite desires from security experts. This patch changes Mercurial's default behavior on modern Python versions to require TLS 1.1+, thus avoiding known security issues with TLS 1.0 and making Mercurial more secure by default. Rather than drop TLS 1.0 support wholesale, we still allow TLS 1.0 to be used if configured. This is a compromise solution - ideally we'd disallow TLS 1.0. However, since we're not sure how many Mercurial servers don't support TLS 1.1+ and we're not sure how much user inconvenience this change will bring, I think it is prudent to ship an escape hatch that still allows usage of TLS 1.0. In the default case our users get better security. In the worst case, they are no worse off than before this patch. This patch has no effect when running on Python versions that don't support TLS 1.1+. As the added test shows, connecting to a server that doesn't support TLS 1.1+ will display a warning message with a link to our wiki, where we can guide people to configure their client to allow less secure connections.

sslutil: config option to specify TLS protocol version Currently, Mercurial will use TLS 1.0 or newer when connecting to remote servers, selecting the highest TLS version supported by both peers. On older Pythons, only TLS 1.0 is available. On newer Pythons, TLS 1.1 and 1.2 should be available. Security-minded people may want to not take any risks running TLS 1.0 (or even TLS 1.1). This patch gives those people a config option to explicitly control which TLS versions Mercurial should use. By providing this option, one can require newer TLS versions before they are formally deprecated by Mercurial/Python/OpenSSL/etc and lower their security exposure. This option also provides an easy mechanism to change protocol policies in Mercurial. If there is a 0-day and TLS 1.0 is completely broken, we can act quickly without changing much code. Because setting the minimum TLS protocol is something you'll likely want to do globally, this patch introduces a global config option under [hostsecurity] for that purpose. wrapserversocket() has been taught a hidden config option to define the explicit protocol to use. This is queried in this function and not passed as an argument because I don't want to expose this dangerous option as part of the Python API. There is a risk someone could footgun themselves. But the config option is a devel option, has a warning comment, and I doubt most people are using `hg serve` to run a production HTTPS server (I would have something not Mercurial/Python handle TLS). If this is problematic, we can go back to using a custom extension in tests to coerce the server into bad behavior.

sslutil: prevent CRIME ssl.create_default_context() disables compression on the TLS channel in order to prevent CRIME. I think we should follow CPython's lead and attempt to disable channel compression in order to help prevent information leakage. Sadly, I don't think there is anything we can do on Python versions that don't have an SSLContext, as there is no way to set channel options with the limited ssl API.

sslutil: update comment about create_default_context() While ssl.create_default_context() creates a SSLContext with reasonable default options, we can't use it because it conflicts with our CA loading controls. So replace the comment with reality. (FWIW the comment was written before the existing CA loading code was in place.)

tests: use sslutil.wrapserversocket() Like the built-in HTTPS server, this code was using the ssl module directly and only using TLS 1.0. Like the built-in HTTPS server, we switch it to use sslutil.wrapserversocket() so it can follow better practices.

hgweb: use sslutil.wrapserversocket() This patch transitions the built-in HTTPS server to use sslutil for creating the server socket. As part of this transition, we implement developer-only config options to control CA loading and whether to require client certificates. This eliminates the need for the custom extension in test-https.t to define these. There is a slight change in behavior with regards to protocol selection. Before, we would always use the TLS 1.0 constant to define the protocol version. This would *only* use TLS 1.0. sslutil defaults to TLS 1.0+. So this patch improves the security of `hg serve` out of the box by allowing it to use TLS 1.1 and 1.2 (if available).

sslutil: implement wrapserversocket() wrapsocket() is heavily tailored towards client use. In preparation for converting the built-in server to use sslutil (as opposed to the ssl module directly), we add wrapserversocket() for wrapping a socket to be used on servers.

hgweb: pass ui into preparehttpserver Upcoming patches will need the built-in HTTPS server to be more configurable.

rebase: remove sortedstate-related confusion The following rebase implementation details are frustrating: - storing a list of sorted revision numbers in a field named sortedstate - having sortedstate be a field of the rebaseruntime class - using sortedstate[-1] as opposed to a more intuitive max(self.state) to compute the latest revision in the state This commit fixes those imperfections.

rebase: replace extrafn field with _makeextrafn invocations As per Yuya's advice, we would like to slightly reduce the amount of state which is stored in rebaseruntime class. In this case, we don't need to store extrafn field, as we can produce the necessary value by calling _makeextrafn and the perf overhead is negligible.

mercurial: implement a source transforming module loader on Python 3 The most painful part of ensuring Python code runs on both Python 2 and 3 is string encoding. Making this difficult is that string literals in Python 2 are bytes and string literals in Python 3 are unicode. So, to ensure consistent types are used, you have to use "from __future__ import unicode_literals" and/or prefix literals with their type (e.g. b'foo' or u'foo'). Nearly every string in Mercurial is bytes. So, to use the same source code on both Python 2 and 3 would require prefixing nearly every string literal with "b" to make it a byte literal. This is ugly and not something mpm is willing to do at this point in time. This patch implements a custom module loader on Python 3 that performs source transformation to convert string literals (unicode in Python 3) to byte literals. In effect, it changes Python 3's string literals to behave like Python 2's. In addition, the module loader recognizes well-known built-in functions (getattr, setattr, hasattr) and methods (encode and decode) that barf when bytes are used and prevents these from being rewritten. This prevents excessive source changes to accommodate this change (we would have to rewrite every occurrence of these functions passing string literals otherwise). The module loader is only used on Python packages belonging to Mercurial. The loader works by tokenizing the loaded source and replacing "string" tokens if necessary. The modified token stream is untokenized back to source and loaded like normal. This does add some overhead. However, this all occurs before caching: .pyc files will cache the transformed version. This means the transformation penalty is only paid on first load. As the extensive inline comments explain, the presence of a custom source transformer invalidates assumptions made by Python's built-in bytecode caching mechanism. So, we have to wrap bytecode loading and writing and add an additional header to bytecode files to facilitate additional cache validation when the source transformations change in the future. There are still a few things this code doesn't handle well, namely support for zip files as module sources and for extensions. Since Mercurial doesn't officially support Python 3 yet, I'm inclined to leave these as to-do items: getting a basic module loading mechanism in place to unblock further Python 3 porting effort is more important than comprehensive module importing support. check-py3-compat.py has been updated to ignore frames. This is necessary because CPython has built-in code to strip frames from the built-in importer. When our custom code is present, this doesn't work and the frames get all messed up. The new code is not perfect. It works for now. But once you start chasing import failures you find some edge cases where the files aren't being printed properly. This only burdens people doing future Python 3 porting work so I'm inclined to punt on the issue: the most important thing is for the source transforming module loader to land. There was a bit of churn in test-check-py3-compat.t because we now trip up on str/unicode/bytes failures as a result of source transformation. This is unfortunate but what are you going to do. It's worth noting that other approaches were investigated. We considered using a custom file encoding whose decode() would apply source transformations. This was rejected because it would require each source file to declare its custom Mercurial encoding. Furthermore, when changing the source transformation we'd need to version bump the encoding name otherwise the module caching layer wouldn't know the .pyc file was invalidated. This would mean mass updating every file when the source transformation changes. Yuck. We also considered transforming at the AST layer. However, Python's ast module is quite gnarly and doing AST transforms is quite complicated, even for trivial rewrites. There are whole Python packages that exist to make AST transformations usable. AST transforms would still require import machinery, so the choice was basically to perform source-level, token-level, or ast-level transforms. Token-level rewriting delivers the metadata we need to rewrite intelligently while being relatively easy to understand. So it won. General consensus seems to be that this approach is the best available to avoid bulk rewriting of '' to b''. However, we aren't confident that this approach will never be a future maintenance burden. This approach does unblock serious Python 3 porting efforts. So we can re-evaulate once more work is done to support Python 3.

compat: define ssize_t as int on 32bit Windows, silences C4142 warning It appears Python.h provides ssize_t, which is aliased to int. https://hg.python.org/cpython/file/v2.7.11/PC/pyconfig.h#l205

commandserver: drop old unixservice implementation It's been superseded by unixforkingservice.

chgserver: switch to new forking service Threading and complex classes are no longer necessary. _autoexitloop() has been replaced by polling cycle in the main thread.

chgserver: extract stub factory of service object The class inheritance will be replaced by composition. See the next patch for details.

chgserver: reorder service classes to make future patches readable Includes no functional change.

commandserver: add new forking server implemented without using SocketServer SocketServer.ForkingMixIn of Python 2.x has a couple of issues, such as: - race condition that leads to 100% CPU usage (Python 2.6) https://bugs.python.org/issue21491 - can't wait for children belonging to different process groups (Python 2.6) - leaves at least one zombie process (Python 2.6, 2.7) https://bugs.python.org/issue11109 The first two are critical because we do setpgid(0, 0) in child process to isolate terminal signals. The last one isn't, but ForkingMixIn seems to be doing silly. So there are two choices: a) backport and maintain SocketServer until we can drop support for Python 2.x b) replace SocketServer by simpler one and eliminate glue codes I chose (b) because it's great time for getting rid of utterly complicated SocketServer stuff, and preparing for future move towards prefork service. New unixforkingservice is implemented loosely based on chg 531f8ef64be6. It is monolithic but much simpler than SocketServer. unixservicehandler provides customizing points for chg, and it will be shared with future prefork service. Old unixservice class is still used by chgserver. It will be removed later. Thanks to Jun Wu for investigating these issues.

commandserver: extract function that serves for the current connection This will be used by new server implementation.

commandserver: manually create file objects from socket Prepares for moving away from SocketServer. See the subsequent patches for why.

bdiff: split bdiff into cpy-aware and cpy-agnostic part

bdiff: rename functions and structs to be amenable for later exporting

bdiff: use ssize_t in favor of Py_ssize_t in cpython-unaware locations This function and struct will be exposed via cffi, so we need to remove the cpython API dependency they currently have.

hgweb: enumerate lines in loop header, not before Doing this will allow access to the lines in arbitrary order (because the result of enumerate() is an iterator), and that will help calculating rowspan for annotate blocks.

sslutil: add assertion to prevent accidental CA usage on Windows Yuya suggested we add this check to ensure we don't accidentally try to load user-writable paths on Windows if we change the control flow of this function later.

shelve: make unshelve be able to abort in any case

osx: explicitly build hg with /usr/bin/python2.7 This should help avoid creating a package that depends on a custom Python, as happened when I built a package for 3.8.

osx: correct comment about ordering of welcome page

osx: jettison outdated build instructions

commandserver: extract _cleanup() hook to clarify chg is doing differently This makes it clear that chg needs its own way to unlink closed socket file. I made a mistake in draft patches without noting the difference.

chgserver: drop repo at chgunixservice.__init__() Since it isn't expensive operation, we don't have to delay it to init().

chgserver: extract utility to bind unix domain socket to long path This is common problem of using sockaddr_un.

chgserver: narrow scope of chdir() to socket.bind() This helps extracting a utility function.

annotate: handle empty files earlier Rather than looping on funcmap and then checking for non-zero `l` continue if the result of fctx.annotate is empty.

context: eliminate handling of linenumber being None in annotate I could not find any use of this parameter value. And it arguably makes understanding of the function more difficult. Setting the parameter default value to False.

tests: regenerate x509 test certificates The old x509 test certificates were using cryptographic settings that are ancient by today's standards, namely 512 bit RSA keys. To put things in perspective, browsers have been dropping support for 1024 bit RSA keys. I think it is important that tests match the realities of the times. And 2048 bit RSA keys with SHA-2 hashing are what the world is moving to. This patch replaces all the x509 certificates with new versions using modern best practices. In addition, the docs for generating the keys have been updated, as the existing docs left out a few steps, namely how to generate certs that were not active yet or expired.

hgweb: add a link on node id in annotate hover-box The link pointing the annotate view at this revision, just like the one in the left-column but accessible from anywhere.

hgweb: move author information from left-column to hover-box in annotate view And display the full author information since there is enough space there.

hgweb: add links to diff and changeset in hover-box on annotate view

hgweb: add link to parents of annotated revision in annotate view The link is embedded into a div with class="annotate-info" that only shows up upon hover of the annotate column. To avoid duplicate hover-overs (this new one and the one coming from link's title), drop "title" attribute from a element and put it in the annotate-info element.

compat: provide a declaration of ssize_t, for MS windows

check-code: enforce (glob) on output lines containing 127.0.0.1

tests: add (glob) annotations to output lines with 127.0.0.1

run-tests: add support for using 127.0.0.1 as a glob Some systems don't have a 127/8 address for localhost (I noticed this on a FreeBSD jail). In order to work around this, use 127.0.0.1 as a glob pattern. A future commit will update needed output lines and add a requirement to check-code.py.

check-code: only treat a # as a line in a t-test if it has a space before it Prior to this, check-code wouldn't notice things like (glob) annotations or similar in a test if they were after a # anywhere in the line. This resolves a defect in a future change, and also exposed a couple of small spots that needed some attention.

test-export: be more aggressive about quoting ^ An upcoming change to check-code will notice this isn't quoted enough. Presumably it's been fine by luck all this time.

test-check-shbang: work around check-code not wanting hardcoded paths I'm about to fix a bug in check-code that a # anywhere on a line treated the rest of the line as a comment, even if it was meaningful. This test is the one place we explicitly *do* want hardcoded paths referenced, but we can work around that by specifying bin as a regular expression.

tests: relax "Connection refused" match We already had the match relaxed on Windows, but on Google Compute Engine VMs I'm seeing "Network is unreachable" instead of "Connection refused". At this point, just give up and make sure we get an error back.

commandserver: backport handling of forking server from chgserver This is common between chg and vanilla forking server, so move it to commandserver and unify handle(). It would be debatable whether we really need gc.collect() or not, but that is beyond the scope of this series. Maybe we can remove gc.collect() once all resource deallocations are switched to context manager.

commandserver: promote .cleanup() hook from chgserver This allows us to unify _requesthandler.handle().

commandserver: extract method to create commandserver instance per request This is a step toward merging chgserver._requesthandler with commandserver's.

error: make hintable exceptions reject unknown keyword arguments (API) Previously they would accept any typos of the hint keyword.

error: make HintException a mix-in class not derived from BaseException (API) HintException is unrelated to the hierarchy of errors. It is an implementation detail whether a class inherits from HintException or not, a sort of "private inheritance" in C++. New Hint isn't an exception class, which prevents catching error by its type: try: dosomething() except error.Hint: pass Unfortunately, this passes on PyPy 5.3.1, but not on Python 2, and raises more detailed TypeError on Python 3.

sslutil: move context options flags to _hostsettings Again, moving configuration determination to a single location.

sslutil: move protocol determination to _hostsettings Most of the logic for configuring TLS is now in this function. Let's move protocol determination code there as well.

share: don't recreate the source repo each time Previously, every time you asked for the source repo of a shared working copy it would recreate the repo object, which required calling reposetup. With certain extension enabled, this can be quite expensive, and it can happen many times (for instance, share attaches a post transaction hook to update bookmarks that triggers this). The fix is to just cache the repo object instead of constantly recreating it.

setup: prepare for future cffi modules by adding placeholder in setup

journal: add support for seaching by pattern If a pattern is used, include the entry name in the output, to make it clear what name was matched.

journal: add share extension support Rather than put everything into one journal file, split entries up in *shared* and *local* entries. Working copy changes are local to a specific working copy, so should remain local only. Other entries are shared with the source if so configured when the share was created. When unsharing, any shared journale entries are copied across.

journal: add dirstate tracking Note that now the default action for `hg journal` is to list the working copy history, not all bookmarks. In its place is the `--all` switch which lists all name changes recorded, including the name for which the change was recorded on each line. Locking is switched to using a dedicated lock to avoid issues with the dirstate being written during wlock unlocking (you can't re-lock during that process).

merge with stable

sslutil: try to find CA certficates in well-known locations Many Linux distros and other Nixen have CA certificates in well-defined locations. Rather than potentially fail to load any CA certificates at all (which will always result in a certificate verification failure), we scan for paths to known CA certificate files and load one if seen. Because a proper Mercurial install will have the path to the CA certificate file defined at install time, we print a warning that the install isn't proper and provide a URL with instructions to correct things. We only perform path-based fallback on Pythons that don't know how to call into OpenSSL to load the default verify locations. This is because we trust that Python/OpenSSL is properly configured and knows better than Mercurial. So this new code effectively only runs on Python <2.7.9 (technically Pythons without the modern ssl module).

sslutil: issue warning when unable to load certificates on OS X Previously, failure to load system certificates on OS X would lead to a certificate verify failure and that's it. We now print a warning message with a URL that will contain information on how to configure certificates on OS X. As the inline comment states, there is room to improve here. I think we could try harder to detect Homebrew and MacPorts installed certificate files, for example. It's worth noting that Homebrew's openssl package uses `security find-certificate -a -p` during package installation to export the system keychain root CAs to etc/openssl/cert.pem. This is something we could consider adding to setup.py. We could also encourage packagers to do this. For now, I'd just like to get this warning (which matches Windows behavior) landed. We should have time to improve things before release.

revert: don't backup if no files reverted in interactive mode (issue4793) When reverting interactively, we always backup files before prompting the user to find out if they actually want to revert them. This can create spurious *.orig files if a user enters an interactive revert session and then doesn't revert any files. Instead, we should only backup files that are actually being touched.

perf: define command annotation locally for Mercurial earlier than 3.1 Before this patch, using cmdutil.command() for "@command" annotation prevents perf.py from being loaded by Mercurial earlier than 1.9 (or 2daa5179e73f), because cmdutil.command() isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). In addition to it, "norepo" option of command annotation has been available since 3.1 (or 75a96326cecb), and this is another blocker for loading perf.py with earlier Mercurial. ============ ============ ====== command of hg version cmdutil norepo ============ ============ ====== 3.1 or later o o 1.9 or later o x earlier x x ============ ============ ====== This patch defines "command()" for annotation locally as below: - define wrapper of existing cmdutil.command(), if cmdutil.command() doesn't support "norepo" (for Mercurial earlier than 3.1) - define full command() locally with minimum function, if cmdutil.command() isn't available at runtime (for Mercurial earlier than 1.9) This patch also defines parsealiases() locally without examining whether it is available or not, because it is small enough to define locally.

perf: avoid using formatteropts for Mercurial earlier than 3.2 Before this patch, referring commands.formatteropts prevents perf.py from being loaded by Mercurial earlier than 3.2 (or 7a7eed5176a4), because it isn't available in such Mercurial, even though formatting itself has been available since 2.2 (or ae5f92e154d3). In addition to it, there are some code paths for Mercurial earlier than 3.2. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). This patch uses empty option list as formatteropts, if it isn't available in commands module at runtime. Disabling -T/--template option for earlier Mercurial should be reasonable, because: - since 427e80a18ef8, -T/--template for formatter has been available - since 7a7eed5176a4, commands.formatteropts has been available - the latter revision is direct child of the former

perf: use locally defined revlog option list for Mercurial earlier than 3.7 Before this patch, referring commands.debugrevlogopts prevents perf.py from being loaded by Mercurial earlier than 3.7 (or 5606f7d0d063), because it isn't available in such Mercurial, even though cmdutil.openrevlog(), a user of these options, has been available since 1.9 (or a79fea6b3e77). In addition to it, there are some code paths for Mercurial earlier than 3.7. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with hg earlier than 1.8 (or 61c9bc3da402). But just "using locally defined revlog option list" might cause unexpected behavior at runtime. If --dir option is specified to cmdutil.openrevlog() of Mercurial earlier than 3.5 (or 49c583ca48c4), it is silently ignored without any warning or so. ============ ============ ===== =============== debugrevlogopts hg version openrevlog() --dir of commands ============ ============ ===== =============== 3.7 or later o o o 3.5 or later o o x 1.9 or later o x x earlier x x x ============ ============ ===== =============== Therefore, this patch does: - use locally defined option list, if commands.debugrevlogopts isn't available (for Mercurial earlier than 3.7) - wrap cmdutil.openrevlog(), if it is ambiguous whether cmdutil.openrevlog() can recognize --dir option correctly (for Mercurial earlier than 3.5) This wrapper function aborts execution, if: - --dir option is specified, and - localrepository doesn't have "dirlog" attribute, which indicates that localrepository has a function for '--dir' BTW, extensions.wrapfunction() has been available since 1.1 (or 0ab5f21c390b), and this seems old enough for "historical portability" of perf.py, which has been available since 1.1 (or eb240755386d).

perf: define util.safehasattr forcibly for Mercurial earlier than 1.9.3 Before this patch, using util.safehasattr() prevents perf.py from being loaded by Mercurial earlier than 1.9.3 (or 94b200a11cf7), because util.safehasattr() isn't available in such Mercurial, even though there are some code paths for Mercurial earlier than 1.9.3. For example, setting "_prereadsize" attribute in perfindex() and perfnodelookup() is effective only with Mercurial earlier than 1.8 (or 61c9bc3da402). This patch is a preparation for using util.safehasattr() safely in subsequent patches. This patch defines util.safehasattr() forcibly without examining whether it is available or not, because: - examining existence of "safehasattr" safely itself needs similar logic - safehasattr() is small enough to define locally

perf: add historical portability policy for future reference

tests: check ETag format in test-hgweb-commands

hgweb: emit a valid, weak ETag Previously, ETag headers from hgweb weren't correctly formed, because rfc2616 (section 14, header definitions) requires double quotes around the content of the header. str(web.mtime) didn't do that. Additionally, strong ETags signify that the resource representations are byte-for-byte identical. That is, they can be reconstructed from byte ranges if client so wishes. Considering ETags for all hgweb pages is just mtime of 00changelog.i and doesn't consider of e.g. .hg/hgrc with description, contact and other fields, it's clearly shouldn't be strong. The W/ prefix marks it as weak, which still allows caching the whole served file/page, but doesn't allow byte-range requests.

policy: add cffi policy for PyPy This adds cffi policy in the case where we don't want to use C modules, but instead we're happy to rely on cffi (bundled with pypy)

sslutil: handle default CA certificate loading on Windows See the inline comment for what's going on here. There is magic built into the "ssl" module that ships with modern CPython that knows how to load the system CA certificates on Windows. Since we're not shipping a CA bundle with Mercurial, if we're running on legacy CPython there's nothing we can do to load CAs on Windows, so it makes sense to print a warning. I don't anticipate many people will see this warning because the official (presumed popular) Mercurial distributions on Windows bundle Python and should be distributing a modern Python capable of loading system CA certs.

sslutil: expand _defaultcacerts docstring to note calling assumptions We should document this so future message additions don't seem out of place.

sslutil: document the Apple OpenSSL cert trick This is sort of documented in _plainapplypython()'s docstring. But it helps to be explicit in security code.

sslutil: use certificates provided by certifi if available The "certifi" Python package provides a distribution of the Mozilla trusted CA certificates as a Python package. If it is present, we assume the user intends it to be used and we use it to provide the default CA certificates when certificates are otherwise not configured. It's worth noting that this behavior roughly matches the popular "requests" package, which also attempts to use "certifi" if present.

py3: make files use absolute_import and print_function This patch includes addition of absolute_import and print_function to the files where they are missing. The modern importing conventions are also followed.

sslutil: don't attempt to find default CA certs file when told not to Before, devel.disableloaddefaultcerts only impacted the loading of default certs via SSLContext. After this patch, the config option also prevents sslutil._defaultcacerts() from being called. This config option is meant to be used by tests to force no CA certs to be loaded. Future patches will enable _defaultcacerts() to have success more often. Without this change we can't reliably test the failure to load CA certs. (This patch also likely fixes test failures on some OS X configurations.)

sslutil: pass ui to _defaultcacerts We'll use this shortly.

sslutil: change comment and logged message for found ca cert file Future patches will change _defaultcacerts() to do something on platforms that aren't OS X. Change the comment and logged message to reflect the future.

tests: better testing of loaded certificates Tests were failing on systems like RHEL 7 where loading the system certificates results in CA certs being reported to Python. We add a feature that detects when we're able to load *and detect* the loading of system certificates. We update the tests to cover the 3 scenarios: 1) system CAs are loadable and detected 2) system CAs are loadable but not detected 3) system CAs aren't loadable

update: teach hg to override untracked dir with a tracked file on update This is a fix to an old problem when Mercurial got confused by an untracked folder with the same name as one of the files in a commit hg was trying to update to. It is pretty safe to remove this folder if it is empty. Backing up an empty folder seems to go against Mercurial's "don't track dirs" philosophy.

rebase: move handling of obsolete commits to be a separate RR class method

rebase: move rebase finish logic to be a method of the RR class Rebase finish logic includes collapsing working directorystate into a single commit, moving bookmarks, clearing status and collapsemsg files, reporting skipped commits to the user and obsoleting precursors of the newly created commits.

rebase: move core rebase logic to be a method of the RR class

rebase: move local variable 'extrafn' to the RR class

rebase: move local variable 'currentbookmarks' to the RR class

rebase: make collapsing use explicit logic to decide on the rev to reuse This code: for rev in sortedstate: ... ... newnode = concludenode(repo, rev, p1, rbsrt.external, commitmsg=commitmsg, extrafn=extrafn, editor=editor, keepbranches=rbsrt.keepbranchesf, date=rbsrt.date) uses 'rev' variable in 'concludenode' function invocation. It is not explicitly assigned before, but its value comes as last value or 'rev' in a for loop, e.g. last element in a 'sortedstate'. IMO this a bad style and it also makes it hard to refactor the function, so it is better to explicitly define the value passed to 'concludenode'.

rebase: move new rebase preparation to be a method of the RR class This commit moves logic that prepares the execution of a new rebase operation to be a method of the rebaseruntime class.

rebase: move abort/continue prep to be a method of the RR class This commit moves logic that prepares the execution of abort and continue phases or rebase operation to be a method of the rebaseruntime class.

hgweb: expose list of per-repo labels to templates hgweb currently offers limited functionality for "classifying" repositories. This patch aims to change that. The web.labels config option list is introduced. Its values are exposed to the "index" and "summary" templates. Custom templates can use template features like ifcontains() to e.g. look for the presence of a specific label and engage specific behavior. For example, a site operator may wish to assign a "defunct" label to a repository so the repository is prominently marked as dead in repository indexes.

histedit: move autoverb rule to the commit it matches Inspired by how 'git rebase -i' works, we move the autoverb to the commit line summary that it matches. We do this by iterating over all rules and inserting each non-autoverb line into a key in an ordered dictionary. If we find an autoverb line later, we then search for the matching key and append it to the list (which is the value of each key in the dictionary). If we can't find a previous line to move to, then we leave the rule in the same spot. Tests have been updated but the diff looks a little messy because we need to change one of the summary lines so that it will actually move to a new spot. On top of that, we added -q flags to future some of the output and needed to change the file it modified so that it wouldn't cause a conflict.

histedit: use _getsummary in ruleeditor This patch uses our common method instead of duplicating logic.

histedit: use _getsummary in torule This patch uses our common method instead of duplicating logic.