manifest: make manifest derive from manifestrevlog As part of our refactoring to split the manifest concept from its storage, we need to start moving the revlog specific parts of the manifest implementation to a new class. This patch creates manifestrevlog and moves the fulltextcache onto the base class.

manifest: break mancache into two caches The old manifest cache cached both the inmemory representation and the raw text. As part of the manifest refactor we want to separate the storage format from the in memory representation, so let's split this cache into two caches. This will let other manifest implementations participate in the in memory cache, while allowing the revlog based implementations to still depend on the full text caching where necessary.

dispatch: explicitly pass fancyopts optional arg as a keyword I've been baffled by this a couple of times (mainly wondering if any callers of fancyopts.fancyopts that don't use gnu=True exist), so let's just specify this as a keyword argument to preserve sanity.

osutil: fix the bug on OS X when we return more in listdir The pointer arithmetic somehow got ommitted during the recent change to use a struct.

histedit: use samefile function from cmdutil Replace usage of inefficient samefile function in collapse with a call to cmdutil.samefile().

cmdutil: extract samefile function from amend()

templater: rename "right" argument of pad() function Before, right=True meant right justify, which I think is left padding.

templater: make pad() evaluate boolean argument (BC) Otherwise it would crash if template expression was passed. This patch unifies the way how boolean expression is evaluated, which involves BC. Before "if(true)" and "pad(..., 'false')" were False, which are now True since they are boolean literal and non-empty string respectively. "func is runsymbol" is the same hack as evalstringliteral(), which is needed for label() to take color literals.

templater: fix if() to not evaluate False as bool('False') Before, False was True. This patch fixes the issue by processing True/False transparently. The other values (including integer 0) are tested as strings for backward compatibility, which means "if(latesttagdistance)" never be False. Should we change the behavior of "if(0)" as well?

templater: make it clearer that _flatten() omits None

revset: support "follow(renamed.py, e22f4f3f06c3)" (issue5334) v2: fixes from review

coal: use inheritance to derive from paper This illustrates how much simpler this approach is, in particular the effect of map-relative paths.

templater: add inheritance support to style maps We can now specify a base map file: __base__ = path/to/map/file That map file will be read and used to populate unset elements of the current map. Unlike using %include, elements in the inherited class will be read relative to that path. This makes it much easier to make custom local tweaks to a style.

killdaemons: prevent killing all user processes by bad pid file When I was fixing the test-gpg issue, I noticed gpg-connect-agent could print "-1" as a server pid if command was wrong. I'm not pretty sure but nobody would want to kill their running applications by mistake.

py3: automatically glob out line numbers from check-py3-compat output It was boring task to update the result manually.

py3: have check-py3-compat require pygments to get stable result

computeoutgoing: move the function from 'changegroup' to 'exchange' Now that all users are in exchange, we can safely move the code in the 'exchange' module. This function is really about processing the argument of a 'getbundle' call, so it even makes senses to do so.

getchangegroup: take an 'outgoing' object as argument (API) There is various version of this function that differ mostly by the way they define the bundled set. The flexibility is now available in the outgoing object itself so we move the complexity into the caller themself. This will allow use to remove a good share of the similar function to obtains a changegroup in the 'changegroup.py' module. An important side effect is that we stop calling 'computeoutgoing' in 'getchangegroup'. This is fine as code that needs such argument processing is actually going through the 'exchange' module which already all this function itself.

outgoing: add a 'missingroots' argument This argument can be used instead of 'commonheads' to determine the 'outgoing' set. We remove the outgoingbetween function as its role can now be handled by 'outgoing' itself. I've thought of using an external function instead of making the constructor more complicated. However, there is low hanging fruit to improve the current code flow by storing some side products of the processing of 'missingroots'. So in my opinion it make senses to add all this to the class.

outgoing: adds some default value for argument We are about to introduce a third option to create an outgoing object: 'missingroots'. This argument will be mutually exclusive with 'commonheads' so we implement some default value handling in preparation. This will also help use to make more use of outgoing creation around the code base.

outgoing: pass a repo object to the constructor We are to introduce more code constructing such object in the code base. It will be more convenient to pass a repository object, all current users already operate at the repository level anyway. More changes to the contructor argument are coming in later changeset.

match: remove matchessubrepo method (API) Since it is no more used in cmdutil.{files,remove} and scmutil.addremove we remove this method.

subrepo: cleanup of subrepo filematcher logic Previously in the worst case we iterated the files in matcher twice and had a method only for this, which reimplemented logic in subdirmatchers constructor. So we replaced the method with a subdirmatcher.files() call.

pycompat: delay loading modules registered to stub Replacement _pycompatstub designed to be compatible with our demandimporter. try-except is replaced by version comparison because ImportError will no longer be raised immediately.

py3: import builtin wrappers automagically by code transformer This should be less invasive than mucking builtins. Since tokenize.untokenize() looks start/end positions of tokens, we calculates them from the NEWLINE token of the future import.

py3: provide (del|get|has|set)attr wrappers that accepts bytes These functions will be imported automagically by our code transformer. getattr() and setattr() are widely used in our code. We wouldn't probably want to rewrite every single call of getattr/setattr. delattr() and hasattr() aren't that important, but they are functions of the same kind.

py3: check python version to enable builtins hack Future patches will add (del|get|has|set)attr wrappers.

py3: move xrange alias next to import lines Builtin functions should be available in compatibility code.

check-code: allow assignment to hasattr variable

debugobsolete: add formatter support (issue5134) It appears that computing index isn't cheap if --rev is specified. That's why "index" field is available only if --index is specified. I've named marker.flags() as "flag" because "flags" implies a list or dict in template world. Thanks to Piotr Listkiewicz for the initial implementation of this patch.

formatter: add function to convert dict to appropriate format This will be used to process key-value pairs by formatter. The default field names and format are derived from the {extras} template keyword. Tests will be added later.

check-code: make dict() pattern less invasive 'foodict(x=y)' should be allowed.

hgweb: tweak zlib chunking behavior When doing streaming compression with zlib, zlib appears to emit chunks with data after ~20-30kb on average is available. In other words, most calls to compress() return an empty string. On the mozilla-unified repo, only 48,433 of 921,167 (5.26%) of calls to compress() returned data. In other words, we were sending hundreds of thousands of empty chunks via a generator where they touched who knows how many frames (my guess is millions). Filtering out the empty chunks from the generator cuts down on overhead. In addition, we were previously feeding 8kb chunks into zlib compression. Since this function tends to emit *compressed* data after 20-30kb is available, it would take several calls before data was produced. We increase the amount of data fed in at a time to 32kb. This reduces the number of calls to compress() from 921,167 to 115,146. It also reduces the number of output chunks from 48,433 to 31,377. This does increase the average output chunk size by a little. But I don't think this will matter in most scenarios. The combination of these 2 changes appears to shave ~6s CPU time or ~3% from a server serving the mozilla-unified repo.

test-gpg: run migration of v1 secret keys beforehand This suppresses unwanted output at "hg sign".

test-gpg: start gpg-agent under control of the test runner GnuPG v2 automatically starts gpg-agent. We should kill the daemon process.

test-gpg: make temporary copy of GNUPGHOME GnuPG v2 will convert v1 secret keys and create a socket under $GNUPGHOME. This patch makes sure no state would persist. We no longer need to verify trustdb.gpg, which was added by aae219a99a6e.

hgweb: document why we don't allow untrusted settings to control zlib Added comment per discussion on mercurial-devel.

hgweb: profile HTTP requests Currently, running `hg serve --profile` doesn't yield anything useful: when the process is terminated the profiling output displays results from the main thread, which typically spends most of its time in select.select(). Furthermore, it has no meaningful results from mercurial.* modules because the threads serving HTTP requests don't actually get profiled. This patch teaches the hgweb wsgi applications to profile individual requests. If profiling is enabled, the profiler kicks in after HTTP/WSGI environment processing but before Mercurial's main request processing. The profile results are printed to the configured profiling output. If running `hg serve` from a shell, they will be printed to stderr, just before the HTTP request line is logged. If profiling to a file, we only write a single profile to the file because the file is not opened in append mode. We could add support for appending to files in a future patch if someone wants it. Per request profiling doesn't work with the statprof profiler because internally that profiler collects samples from the thread that *initially* requested profiling be enabled. I have plans to address this by vendoring Facebook's customized statprof and then improving it.

hgweb: abstract call to hgwebdir wsgi function The function names and behavior now matches hgweb. The reason for this will be obvious in the next patch.

profiling: don't error with statprof when profiling has already started statprof.reset() asserts if profiling has already started. So don't call if it profiling is already running.

profiling: add a context manager that no-ops if profiling isn't enabled And refactor dispatch.py to use it. As you can see, the resulting code is much simpler. I was tempted to inline _runcommand as part of writing this series. However, a number of extensions wrap _runcommand. So keeping it around is necessary (extensions can't easily wrap runcommand because it calls hooks before and after command execution).

profiling: make profiling functions context managers (API) This makes profiling more flexible since we can now call multiple functions when a profiler is active. But the real reason for this is to enable a future consumer to profile a function that returns a generator. We can't do this from the profiling function itself because functions can either be generators or have return values: they can't be both. So therefore it isn't possible to have a generic profiling function that can both consume and re-emit a generator and return a value.

dispatch: set profiling.enabled when profiling is enabled We do this for other global command arguments. We don't for --profile for reasons that are unknown to me. Probably because nobody has needed it. An upcoming patch will introduce a new consumer of the profiling code. It doesn't have access to command line arguments. So let's set the config option during argument processing. We also remove a check for "options['profile']" because it is now redundant.

profiling: move profiling code from dispatch.py (API) Currently, profiling code lives in dispatch.py, which is a low-level module centered around command dispatch. Furthermore, dispatch.py imports a lot of other modules, meaning that importing dispatch.py to get at profiling functionality would often result in a module import cycle. Profiling is a generic activity. It shouldn't be limited to command dispatch. This patch moves profiling code from dispatch.py to the new profiling.py. The low-level "run a profiler against a function" functions have been moved verbatim. The code for determining how to invoke the profiler has been extracted to its own function. I decided to create a new module rather than stick this code elsewhere (such as util.py) because util.py is already quite large. And, I foresee this file growing larger once Facebook's profiling enhancements get added to it.

merge with stable

pycompat: avoid using an extra function We have a single line function which just lowercase the letters and replaces "_" with "". Its better to avoid that function call. Moreover we calling this function around 33 times.

pycompat: remove multiple occurences of urlencode By mistake we had two occurences of urlencode.

osx: stamp the hg version into the version field in the pkg This is required for tools like https://github.com/munki/munki, and is also more semantically correct.

performance: disable workaround for an old bug of Python gc Since disabling the gc does things worse for pypy and the bug was fixed in 2.7, let's only enable it in <2.7

merge: always use other, not remote, in user prompts Now that we store and display merge labels in user prompts (not just conflict markets), we should rely on labels to clarify the two sides of a merge operation (hg merge, hg update, hg rebase etc). "remote" is not a great name here, as it conflates "remote" as in "remote server" with "remote" as in "the side of the merge that's further away". In cases where you're merging the "wrong way" around, remote can even be the "local" commit that you're merging with something pulled from the remote server.

merge: use labels in prompts to the user Now that we persist the labels, we can consistently use the labels in prompts for the user without risk of confusion. This changes a huge amount of command output: This means that merge prompts like: no tool found to merge a keep (l)ocal, take (o)ther, or leave (u)nresolved? u and remote changed a which local deleted use (c)hanged version, leave (d)eleted, or leave (u)nresolved? c become: no tool found to merge a keep (l)ocal [working copy], take (o)ther [destination], or leave (u)nresolved? u and remote [source] changed a which local [dest] deleted use (c)hanged version, leave (d)eleted, or leave (u)nresolved? c where "working copy" and "destination" were supplied by the command that requested the merge as labels for conflict markers, and thus should be human-friendly.

journal: use the dirstate parentchange callbacks Instead of hacking into dirstate internals let's use the callbacks to be notified about wd parent change.

dirstate: add callback to notify extensions about wd parent change The journal extension had to touch the dirstate internals to be notified about wd parent change. To make that detection cleaner and reusable let's move it core. Now the extension can register to be notified about parent changes.

revpair: do not optimize tree to check for odd-range spec At cc3a30ff9490, we had to optimize a parsed tree to resolve x^:y ambiguity. Since we've moved the resolution of x^:y to parse(), we no longer have to call optimize(). Therefore, (x:y) can be taken as a single expression, not an odd range expression x:y.

revset: also parse x^: as (x^): Given x^:y is (x^):y, this seems sensible.

revset: resolve ambiguity of x^:y before alias expansion This is purely a parsing problem, which should be resolved before alias expansion.

revset: add test for resolution of infix/suffix ambiguity of x^:y This is the test for 805651777188, and I'm going to fix the failure of 'x^A' where 'revsetalias.A=:y'.

parser: remove unused binding parameter from suffix action Because a suffix action never takes subsequent tokens, it should have no binding strength nor closing character. I've tried if this value could be used to resolve infix/suffix ambiguity of x^:y, but it appears not. So I decided to resend this patch.

extensions: add unwrapfunction to undo wrapfunction Before this patch, we don't have a safe way to undo a wrapfunction because other extensions may wrap the same function and calling setattr will undo them accidentally. This patch adds an "unwrapfunction" to address the issue. It removes the wrapper from the wrapper chain, and re-wraps everything, which is not the most efficient but short and easy to understand. We can revisit the code if we have perf issues with long chains. The "undo" feature is useful in cases like wrapping a function just in a scope. Like, having a "select" command to interactively (using arrow keys) select content from some output (ex. smartlog). It could wrap "ui.label" to extract interesting texts just in the "select" command.

extensions: add getwrapperchain to get a list of wrappers The getwrapperchain returns a list of wrappers + the original function, making it easier to understand what has been wrapped by whom. For example: In : mercurial.extensions.getwrapperchain(mercurial.dispatch, '_runcommand') Out: [<function hgext.pager.pagecmd>, <function hgext.color.colorcmd>, <function hgext.zeroconf.cleanupafterdispatch>, <function mercurial.dispatch._runcommand>] It will also be useful to safely unwrap a function. See the next patch.

extensions: set attributes to wrappers so we can trace them back This patch adds two attributes about the original function and the unbound wrapper. It allows us to get a chain of wrappers. See the next patch.

ui: drop values returned by inspect.*frame*() to avoid cycles "f = inspect.currentframe()" instantly creates a cycle because "f.f_locals['f']" is "f" itself. This patch explicitly sets those frame objects to None to avoid cycles.

dispatch: split global error handling out so it can be reused We may want a similar error handling at worker.py. This patch extracts the error handling logic to "callcatch" so it can be reused.

py3: conditionalize _winreg import _winreg module is renamed to winreg in python 3. Added the conditionalize statements in the respective file because adding this in pycompat will result in pycompat throwing error as this is a windows registry module and we have buildbots and most of the contributors on linux.

py3: conditionalize the raise statement raise E,V,T is not acceptable in Python 3, thats is conditionalized. Moreover this will result in syntax error so we have to use exec() to execute this. Related PEP- https://www.python.org/dev/peps/pep-3109/#id14 My implementation is motivated from the six implementation except they are defining a new function exec_() to prevent adding an extra frame AFAIK :) https://bitbucket.org/gutworth/six/src/ca4580a5a648/six.py#six.py-680

match: added matchessubrepo method to matcher Previously there were three local implementations of this function in cmdutil.files, cmdutil.remove and scmutil.addremove.

changegroup: move branch cache debug message to proper location Before, we logged about performing a branch cache update when we weren't actually doing it. Fix that.

journal: take wlock for writting the 'shared' file As we did for the shared extension itself, we add some locking around the write of the 'shared' file.

debugbuilddag: take wlock to cover '.hg/localtags' This debug command can write local tags. local tags are in the .hg directory and should be covered by the 'wlock'. This is now covered.

fakemergerecord: take wlock to write the merge state The merge state is supposed to be covered by the wlock. We fix the test extensions to comply to that.

shared: take wlock for writting the 'shared' file I do not see a reason why this should not be covered by the wlock.

mq: take wlock when 'qqueue' is doing write operations Apparently when this command was added, the locking was forgotten. No code changes beside the indentation from the locking context.

py3: make check-py3-compat.py use correct module name at loading pure modules Before this patch, check-py3-compat.py implies unintentional ".pure" sub-package name at loading pure modules, because module name is composed by just replacing "/" in the path to actual ".py" file by ".". This makes pure modules belong to "mercurial.pure" package, and prevents them from importing a module belonging to "mercurial" package relatively by "from . import foo" or so. This is reason why pure modules fail to import another module relatively only at examination by check-py3-compat.py.

py3: update output of check-py3-compat.py with python3 Recent work on mercurial/pure/mpatch.py made it import mercurial.policy, and changed output of check-py3-compat.py with python3 in test-check-py3-compat.t. But test-check-py3-compat.t isn't yet updated.

mpatch: raise MemoryError instead of mpatchError if lalloc() failed MemoryError is handled differently in dispatch._runcatch(). Since mpatch_errors[] isn't that useful now, I've changed it to a simple switch statement.

hgweb: config option to control zlib compression level Before this patch, the HTTP transport protocol would always zlib compress certain responses (notably "getbundle" wire protocol commands) at zlib compression level 6. zlib can be a massive CPU resource sink for servers. Some server operators may wish to reduce server-side CPU requirements while requiring more bandwidth. This is common on corporate intranets, for example. Others may wish to use more CPU but reduce bandwidth. This patch introduces a config option to allow server operators to control the zlib compression level. On the "mozilla-unified" generaldelta repository, setting this value to "0" (disable compression) results in server-side CPU utilization for a `hg clone` going from ~180s to ~124s CPU time on my i7-6700K. A level of "1" (which increases the transfer size from ~1,074 MB at level 6 to ~1,222 MB) utilizes ~132s CPU time.

help: don't try to render a section on sub-topics This patch subtly changes the behavior of the parsing of "X.Y" values to not set the "section" variable when rendering a known sub-topic. Previously, "section" would be the same as the sub-topic name. This required the sub-topic RST to have a section named the same as the sub-topic name. When I made this change, the descriptions from help.internalstable started being rendered in command line output. This didn't look correct to me, as it didn't match the formatting of main help pages. I corrected this by moving the top section to help.internalstable and changing the section levels of all the "internals" topics. The end result is that "internals" topics now match the rendering of main topics on both the CLI and HTML. And, "internals" topics no longer require a main section matching the name of the topic.

branchmap: remove extra indent This clean up the rest of the previous changeset.

branchmap: simplify error handlind when writing rev branch cache Now that we have a general try except, we can move the error handling from the individual writes in it. Code will be reindented in the next changeset to help this on readability.

branchmap: acquires lock before writting the rev branch cache We now attempt to acquire a lock and write the branch cache within that lock. This would prevent cache corruption when multiple processes try to write the cache at the same time.

branchmap: preparatory indent of indent the branch rev writing code The rev branch cache is written without a lock, we are going to fix this but we indent the code beforehand to make the next changeset clearer.

mpatch: silence warning about maybe-uninitialized variable It's false positive, but it wouldn't be possible for gcc to know PyBytes_FromStringAndSize() sets PyErr_Occurred(). mercurial/mpatch_module.c:105:47: warning: 'r' may be used uninitialized in this function [-Wmaybe-uninitialized] PyErr_SetString(mpatch_Error, mpatch_errors[-r]);

mpatch: change lalloc() to local function It was mistakenly made public at b9b9f9a92481.

mpatch: remove superfluous whitespaces

cmdutil: remove duplicated badmatch call in cat() Subrepo logic is handled in ctx.walk().

statichttprepo: do not try to write caches The static http repository are read only, there is no hope in any cache writing attempt.

demandimport: omit default value of "level" at construction of _demandmod This makes construction of _demandmod require explicit level value, and helps to avoid issues like issue5208 in the future.

demandimport: import sub-module relatively as expected (issue5208) Before this patch, importing sub-module might (1) fail or (2) success but import incorrect module, because demandimport tries to import sub-module with level=-1 (on Python 2.x) or level=0 (on Python 3.x), which is default value of "level" argument at construction of "_demandmod" proxy object. (1) on Python 3.x, importing sub-module always fails to import existing sub-module (2) both on Python 2.x and 3.x, importing sub-module might import same name module on root level unintentionally On Python 2.x, existing sub-module is prior to this unexpected module. Therefore, this problem hasn't appeared. To import sub-module relatively as expected, this patch specifies "1" as import level explicitly at construction of "_demandmod" proxy object for sub-module.

wireproto: remove gboptslist (API) This variable has been unused since ce25f465e572, which was over 2 years ago. gboptsmap should be used instead. Marking as API because this could break extensions.

wireproto: unescape argument names in batch command (BC) Clients escape both argument names and values when using the batch command. Yet the server was only unescaping argument values. Fortunately we don't have any argument names that need escaped. But that isn't an excuse to lack symmetry in the code. Since the server wasn't properly unescaping argument names, this means we can never introduce an argument to a batchable command that needs escaped because an old server wouldn't properly decode its name. So we've introduced an assertion to detect the accidental introduction of this in the future. Of course, we could introduce a server capability that says the server knows how to decode argument names and allow special argument names to go through. But until there is a need for it (which I doubt there will be), we shouldn't bother with adding an unused capability.

wireproto: consolidate code for obtaining "cmds" argument value Both wireproto.py and sshpeer.py had code for producing the value to the "cmds" argument used by the "batch" command. This patch extracts that code to a standalone function and uses it.

revlog: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

md5sum: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

util: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

url: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.

sshserver: use `iter(callable, sentinel)` instead of while True This is functionally equivalent, but is a little more concise.