crecord: properly handle files with No newline at eof (issue5268) Yes, this bug was a single character with the wrong case...

annotate: pre-calculate the "needed" dictionary (issue5360) The "needed" dict is used as a reference counter to free items in the giant "hist" dict. However, currently it is not very accurate and can lead to dropping "hist" items unnecessarily, for example, with the following DAG, -3- / \ 0--1--2--4-- The current algorithm will visit and calculate rev 1 twice, undesired. And it tries to be smart by clearing rev 1's parents: "pcache[1] = []" at the time hist[1] being accessed (note: hist[1] needs to be used twice, by rev 2 and rev 3). It can result in incorrect results if p1 of rev 4 deletes chunks belonging to rev 0. However, simply removing "needed" is not okay, because it will consume 10x memory: # without any change % HGRCPATH= lrun ./hg annotate mercurial/commands.py -r d130a38 3>&2 [1] MEMORY 49074176 CPUTIME 9.213 REALTIME 9.270 # with "needed" removed MEMORY 637673472 CPUTIME 8.164 REALTIME 8.249 This patch moves "needed" (and "pcache") calculation to a separate DFS to address the issue. It improves perf and fixes issue5360 by correctly reusing hist, while maintaining low memory usage. Some additional attempt has been made to further reduce memory usage, like changing "pcache[f] = []" to "del pcache[f]". Therefore the result can be both faster and lower memory usage: # with this patch applied MEMORY 47575040 CPUTIME 7.870 REALTIME 7.926 [1]: lrun is a lightweight sandbox built on Linux cgroup and namespace. It's used to measure CPU and memory usage here. Source code is available at github.com/quark-zju/lrun.

help: document wire protocol transport protocols The HTTP and SSH transport protocols are documented. This includes how commands and arguments are serialized as well as response types.

help: internals topic for wire protocol The Mercurial wire protocol is under-documented. This includes a lack of source docstrings and comments as well as pages on the official wiki. This patch adds the beginnings of "internals" documentation on the wire protocol. The documentation should have nearly complete coverage on the lower-level parts of the protocol, such as the different transport mechanims, how commands and arguments are sent, capabilities, and, of course, the commands themselves. As part of writing this documentation, I discovered a number of deficiencies in the protocol and bugs in the implementation. I've started sending patches for some of the issues. I hope to send a lot more. This patch starts with the scaffolding for a new internals page.

grep: add formatter support Several fields are renamed to be consistent with the annotate command, which doesn't mean the last call for the name unification [1]. Actually, I'd rather rename line_number to linenumber, linenum, lineno or line, but I want to port the grep command to formatter first. [1]: https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary I don't have any better name for the list of matched/unmatched texts, so they are just called as "texts".

grep: build list of all columns regardless of display options These columns should always be available in JSON or template outputs. The "change" column is excluded because it has no useful data unless --all is specified.

grep: build list of columns without "grep." label prefix Prepares for formatter support. We need field names without "grep.".

grep: factor out function that prints matched line with labels Prepares for formatter support.

grep: refactor loop that yields matched text with label As preparation for formatter support, this and the next patch split linestate.__iter__() into two functions, line scanner and displayer. New code uses regexp.search(str, pos) in place of regexp.search(substr), which appears to fix a bug of highlighting.

Added signature for changeset ccd436f7db6d

Added tag 3.9.1 for changeset ccd436f7db6d

bundle2: localize handleoutput remote prompts Code archaeology suggests that there was no good reason for this not to be localized. 'remote: ' is already localized elsewhere.

i18n-pt_BR: synchronized with a12d13eac513

files: change documentation to match its behaviour (issue5276) Documentation gave the usage pattern as '[OPTION]... [PATTERN]...' when the command match given files as relpaths by default.

templater: add template path to __base__ search This does a fall-back check for style files or directories that are in Mercurial's template path for user convenience. We intentionally don't use this for the built-in coal style because we don't want the style to mysteriously break if the working directory just happens to have a file named "paper".

bundle2: fail faster when interrupted Before this patch, bundle2 application attempted to consume remaining bundle2 part data when the process is interrupted (SIGINT) or when sys.exit is called (translated into a SystemExit exception). This meant that if one of these occurred when applying a say 1 GB changegroup bundle2 part being downloaded over a network, it may take Mercurial *several minutes* to terminate after a SIGINT because the process is waiting on the network to stream megabytes of data. This is not a great user experience and a regression from bundle1. Furthermore, many process supervisors tend to only give processes a finite amount of time to exit after delivering SIGINT: if processes take too long to self-terminate, a SIGKILL is issued and Mercurial has no opportunity to clean up. This would mean orphaned locks and transactions. Not good. This patch changes the bundle2 application behavior to fail faster when an interrupt or system exit is requested. It does so by not catching BaseException (which includes KeyboardInterrupt and SystemExit) and by explicitly checking for these conditions in yet another handler which would also seek to the end of the current bundle2 part on failure. The end result of this patch is that SIGINT is now reacted to significantly faster: the active transaction is rolled back immediately without waiting for incoming bundle2 data to be consumed. This restores the pre-bundle2 behavior and makes Mercurial treat signals with the urgency they deserve.

blackbox: also log alias expansions This should be extremely useful for helping users debug without having to see their complete configuration. Shell aliases do not get their expansion logged, because we don't look and see if we're in a repo before we dive into the execution of a shell alias. As a result, the ui object doesn't know where to log.

test-clonebundles: accept another error string here This is what happens if you run the test in a FreeBSD Jail, rather than "connection refused".

tests: guard test-archive-symlink with unziplinks check This fixes the test on FreeBSD, where the stock unzip doesn't understand symlinks.

hghave: add a check for unzip(1) that understands symlinks unzip(1) from the FreeBSD base system does not understand symlinks, so test-archive-symlinks is busted.

test-https: drop two spurious --traceback flags These make test failures scarier than they are.

extensions: change magic "shipped with hg" string I've caught multiple extensions in the wild lying about being 'internal', so it's time to move the goalposts on people. Goalpost moving will continue until third party extensions stop trying to defeat the system.

version: add formatter support The license message isn't exported, which I don't think is useful and I couldn't find a way to restructure it for JSON or template outputs.

version: factor out mapping of internal/external labels Prepares for formatter support, where translation should be disabled conditionally.

version: always build list of extension versions This patch just moves "if ui.verbose:" to the inner loop, as preparation for formatter support.

formatter: add fm.nested(field) to either write or build sub items We sometimes need to build nested items by formatter, but there was no convenient way other than building and putting them manually by fm.data(): exts = [] for n, v in extensions: fm.plain('%s %s\n' % (n, v)) exts.append({'name': n, 'ver': v}) fm.data(extensions=exts) This should work for simple cases, but doing this would make it harder to change the underlying data type for better templating support. So this patch provides fm.nested(field), which returns new nested formatter (or self if items aren't structured and just written to ui.) A nested formatter stores items which will later be rendered by the parent formatter. fn = fm.nested('extensions') for n, v in extensions: fn.startitem() fn.write('name ver', '%s %s\n', n, v) fn.end() Nested items are directly exported to a template for now: {extensions % "{name} {ver}\n"} There's no {extensions} nor {join(extensions, sep)} yet. I have a plan for them by extending fm.nested() API, but I want to revisit it after trying out this API in the real world.

formatter: factor out format*() functions to separate classes New converter classes will be reused by a nested formatter. See the next patch for details. This change is also good in that the default values are defined uniquely by the baseformatter.

crecord: restore SIGWINCH handler before return Previously, the SIGWINCH handler does not get cleared and if the commit message editor also needs SIGWINCH handling (like vim), the two SIGWINCH handlers (the editor's, ours) will have a race. And we may erase the editor's screen content. This patch restores SIGWINCH handler to address the above issue.

bdiff: implement cffi version of bdiff

bdiff: implement cffi version of blocks

util: checknlink should remove file it creates if an exception occurs There's no reason to leave the file behind.

merge: remove files with extra actions from merge action list See the comment for a detailed explanation why. Even though this is a bug, I've sent it to 'default' rather than 'stable' because it isn't triggered in any code paths in stock Mercurial, just with the merge driver included. For the same reason I haven't included any tests here -- the merge driver is getting a new test.

revlog: use an LRU cache for delta chain bases Profiling using statprof revealed a hotspot during changegroup application calculating delta chain bases on generaldelta repos. Essentially, revlog._addrevision() was performing a lot of redundant work tracing the delta chain as part of determining when the chain distance was acceptable. This was most pronounced when adding revisions to manifests, which can have delta chains thousands of revisions long. There was a delta chain base cache on revlogs before, but it only captured a single revision. This was acceptable before generaldelta, when _addrevision would build deltas from the previous revision and thus we'd pretty much guarantee a cache hit when resolving the delta chain base on a subsequent _addrevision call. However, it isn't suitable for generaldelta because parent revisions aren't necessarily the last processed revision. This patch converts the delta chain base cache to an LRU dict cache. The cache can hold multiple entries, so generaldelta repos have a higher chance of getting a cache hit. The impact of this change when processing changegroup additions is significant. On a generaldelta conversion of the "mozilla-unified" repo (which contains heads of the main Firefox repositories in chronological order - this means there are lots of transitions between heads in revlog order), this change has the following impact when performing an `hg unbundle` of an uncompressed bundle of the repo: before: 5:42 CPU time after: 4:34 CPU time Most of this time is saved when applying the changelog and manifest revlogs: before: 2:30 CPU time after: 1:17 CPU time That nearly a 50% reduction in CPU time applying changesets and manifests! Applying a gzipped bundle of the same repo (effectively simulating a `hg clone` over HTTP) showed a similar speedup: before: 5:53 CPU time after: 4:46 CPU time Wall time improvements were basically the same as CPU time. I didn't measure explicitly, but it feels like most of the time is saved when processing manifests. This makes sense, as large manifests tend to have very long delta chains and thus benefit the most from this cache. So, this change effectively makes changegroup application (which is used by `hg unbundle`, `hg clone`, `hg pull`, `hg unshelve`, and various other commands) significantly faster when delta chains are long (which can happen on repos with large numbers of files and thus large manifests). In theory, this change can result in more memory utilization. However, we're caching a dict of ints. At most we have 200 ints + Python object overhead per revlog. And, the cache is really only populated when performing read-heavy operations, such as adding changegroups or scanning an individual revlog. For memory bloat to be an issue, we'd need to scan/read several revisions from several revlogs all while having active references to several revlogs. I don't think there are many operations that do this, so I don't think memory bloat from the cache will be an issue.

revlog: remove unused variables

util: properly implement lrucachedict.get() Before, it was returning the raw _lrucachenode instance instead of its value.

manifest: change changectx to access manifest via manifestlog This is the first place where we'll start using manifestctx instances instead of manifestdict. This will facilitate using different manifestctx implementations in the future.

manifest: use property instead of field for manifest revlog storage The file caches we're using to avoid reloading the manifest from disk everytime has an annoying bug that causes the in memory structure to not be reloaded if the mtime and the size haven't changed. This causes a breakage in the tests because the manifestlog is not being reloaded after a commit+strip operation in mq (the mtime is the same because it all happens in the same second, and the resulting size is the same because we add 1 and remove 1). The only reason this doesn't affect the manifest itself is because we touch it so often that we had already reloaded it after the commit, but before the strip. Once the entire manifest has migrated to manifestlog, we can get rid of these properties, since then the manifestlog will be touched after the commit, but before the strip, as well.

manifest: introduce manifestlog and manifestctx classes This is the start of a large refactoring of the manifest class. It introduces the new manifestlog and manifestctx classes which will represent the collection of all manifests and individual instances, respectively. Future patches will begin to convert usages of repo.manifest to repo.manifestlog, adding the necessary functionality to manifestlog and instance as they are needed.

manifest: make manifest derive from manifestrevlog As part of our refactoring to split the manifest concept from its storage, we need to start moving the revlog specific parts of the manifest implementation to a new class. This patch creates manifestrevlog and moves the fulltextcache onto the base class.

manifest: break mancache into two caches The old manifest cache cached both the inmemory representation and the raw text. As part of the manifest refactor we want to separate the storage format from the in memory representation, so let's split this cache into two caches. This will let other manifest implementations participate in the in memory cache, while allowing the revlog based implementations to still depend on the full text caching where necessary.

dispatch: explicitly pass fancyopts optional arg as a keyword I've been baffled by this a couple of times (mainly wondering if any callers of fancyopts.fancyopts that don't use gnu=True exist), so let's just specify this as a keyword argument to preserve sanity.

osutil: fix the bug on OS X when we return more in listdir The pointer arithmetic somehow got ommitted during the recent change to use a struct.

histedit: use samefile function from cmdutil Replace usage of inefficient samefile function in collapse with a call to cmdutil.samefile().

cmdutil: extract samefile function from amend()

templater: rename "right" argument of pad() function Before, right=True meant right justify, which I think is left padding.

templater: make pad() evaluate boolean argument (BC) Otherwise it would crash if template expression was passed. This patch unifies the way how boolean expression is evaluated, which involves BC. Before "if(true)" and "pad(..., 'false')" were False, which are now True since they are boolean literal and non-empty string respectively. "func is runsymbol" is the same hack as evalstringliteral(), which is needed for label() to take color literals.

templater: fix if() to not evaluate False as bool('False') Before, False was True. This patch fixes the issue by processing True/False transparently. The other values (including integer 0) are tested as strings for backward compatibility, which means "if(latesttagdistance)" never be False. Should we change the behavior of "if(0)" as well?

templater: make it clearer that _flatten() omits None

revset: support "follow(renamed.py, e22f4f3f06c3)" (issue5334) v2: fixes from review

coal: use inheritance to derive from paper This illustrates how much simpler this approach is, in particular the effect of map-relative paths.

templater: add inheritance support to style maps We can now specify a base map file: __base__ = path/to/map/file That map file will be read and used to populate unset elements of the current map. Unlike using %include, elements in the inherited class will be read relative to that path. This makes it much easier to make custom local tweaks to a style.

killdaemons: prevent killing all user processes by bad pid file When I was fixing the test-gpg issue, I noticed gpg-connect-agent could print "-1" as a server pid if command was wrong. I'm not pretty sure but nobody would want to kill their running applications by mistake.

py3: automatically glob out line numbers from check-py3-compat output It was boring task to update the result manually.

py3: have check-py3-compat require pygments to get stable result

computeoutgoing: move the function from 'changegroup' to 'exchange' Now that all users are in exchange, we can safely move the code in the 'exchange' module. This function is really about processing the argument of a 'getbundle' call, so it even makes senses to do so.

getchangegroup: take an 'outgoing' object as argument (API) There is various version of this function that differ mostly by the way they define the bundled set. The flexibility is now available in the outgoing object itself so we move the complexity into the caller themself. This will allow use to remove a good share of the similar function to obtains a changegroup in the 'changegroup.py' module. An important side effect is that we stop calling 'computeoutgoing' in 'getchangegroup'. This is fine as code that needs such argument processing is actually going through the 'exchange' module which already all this function itself.

outgoing: add a 'missingroots' argument This argument can be used instead of 'commonheads' to determine the 'outgoing' set. We remove the outgoingbetween function as its role can now be handled by 'outgoing' itself. I've thought of using an external function instead of making the constructor more complicated. However, there is low hanging fruit to improve the current code flow by storing some side products of the processing of 'missingroots'. So in my opinion it make senses to add all this to the class.

outgoing: adds some default value for argument We are about to introduce a third option to create an outgoing object: 'missingroots'. This argument will be mutually exclusive with 'commonheads' so we implement some default value handling in preparation. This will also help use to make more use of outgoing creation around the code base.

outgoing: pass a repo object to the constructor We are to introduce more code constructing such object in the code base. It will be more convenient to pass a repository object, all current users already operate at the repository level anyway. More changes to the contructor argument are coming in later changeset.

match: remove matchessubrepo method (API) Since it is no more used in cmdutil.{files,remove} and scmutil.addremove we remove this method.

subrepo: cleanup of subrepo filematcher logic Previously in the worst case we iterated the files in matcher twice and had a method only for this, which reimplemented logic in subdirmatchers constructor. So we replaced the method with a subdirmatcher.files() call.

pycompat: delay loading modules registered to stub Replacement _pycompatstub designed to be compatible with our demandimporter. try-except is replaced by version comparison because ImportError will no longer be raised immediately.

py3: import builtin wrappers automagically by code transformer This should be less invasive than mucking builtins. Since tokenize.untokenize() looks start/end positions of tokens, we calculates them from the NEWLINE token of the future import.

py3: provide (del|get|has|set)attr wrappers that accepts bytes These functions will be imported automagically by our code transformer. getattr() and setattr() are widely used in our code. We wouldn't probably want to rewrite every single call of getattr/setattr. delattr() and hasattr() aren't that important, but they are functions of the same kind.