revlog: optionally cache the full text when adding revisions revlog instances can cache the full text of a single revision. Typically the most recently read revision is cached. When adding a delta group via addgroup() and _addrevision(), the full text isn't always computed: sometimes only the passed in delta is sufficient for adding a new revision to the revlog. When writing the changelog from a delta group, the just-added full text revision is always read immediately after it is written because the changegroup code needs to extract the set of files from the entry. In other words, revision() is *always* being called and caching the full text of the just-added revision is guaranteed to result in a cache hit, making the cache worthwhile. This patch adds support to _addrevision() for always building and caching the full text. This option is currently only active when processing changelog entries from a changegroup. While the total number of revision() calls is the same, the location matters: buildtext() calls into revision() on the base revision when building the full text of the just-added revision. Since the previous revision's _addrevision() built the full text and the the previous revision is likely the base revision, this means that the base revision's full text is likely cached and can be used to compute the current full text from just a delta. No extra I/O required. The end result is the changelog isn't opened and read after adding every revision from a changegroup. On my 2013 MacBook Pro running OS X 10.10.5 from an SSD and Python 2.7, this patch impacted the time taken to apply ~262,000 changesets from a mozilla-central gzip bundle: before: ~43s after: ~32s ~25% reduction in changelog processing times. Not bad.

revlog: drop local assignment of cache variable The purpose of this code was to provide thread safety. With the conversion of hgweb to use separate localrepository instances per request/thread, we should no longer have any consumers that need to access revlog instances from multiple threads. Remove the code.

revlog: rename generic "i" variable to "indexdata" Increase readability.

hg: always create new localrepository instance cachedlocalrepo.copy() didn't actually create new localrepository instances. This meant that the new thread isolation code in hgweb wasn't actually using separate localrepository instances, even though it was properly using separate cachedlocalrepo instances. Because the behavior of the API changed, the single caller in hgweb had to be refactored to always call _webifyrepo() or it may not have used the proper filter. I confirmed via print() debugging that id(repo) is in fact different on each thread. This was not the case before. For reasons I can't yet explain, this does not fix issue4756. I suspect there is shared cache somewhere that isn't thread safe.

test-bad-extension: reduce dependencies on other things test-bad-extension would jitter if the format of the first line of hg help changed, which isn't relevant to its goal.

help: fix help argument parsing and documentation support combining -c and -e previously -k was misdocumented: * the first line didn't mention it * the help half implied you could do help -k keyword topic with these changes, -k just changes the search method support -c and -e for -k searches

minirst: establish leveling for nested definitions

dispatch: use the right context manager to deactivate demandimport In e86d12404d69 I very embarrassingly wrote a patch with the completely wrong function name. This should fix it.

ui: improve docs on ui.log This makes the documentation on ui.log line up with the use of that interface in blackbox.

templater: switch ctx of list expression to rev of revset() (BC) Because revset() function generates a list of revisions, it seems sensible to switch the ctx as well where a list expression will be evaluated. I think "{revset(...) % "..."}" expression wasn't considered well when it was introduced at cda9d2b6beab.

fileset: handle error of string unescaping

revset: handle error of string unescaping

parser: move unescape helper from templater revset and fileset have a similar problem, so let's make it a common helper function.

unionrepo: take delta base in account with building unified revlog When general delta is enabled, the base is actually meaningful and should be used. With general delta is enabled, test-unionrepo.t crash without this fix.

extdiff: enable -I/-X with --patch Not sure how useful this really is, but it's trivial to add and ignoring the existing arguments supported seems like a bad UI.

extdiff: add a --patch argument for diffing changeset deltas One of the things I missed the most when transitioning from versioned MQ to evolve was the loss of being able to check that rebase conflicts were properly resolved by: $ hg ci --mq -m "before" $ hg rebase -s qbase -d tip $ hg bcompare --mq The old csets stay in the tree with evolve, but a straight diff includes all of the other changes that were pulled in, obscuring the code that was rebased. Diffing deltas can be confusing, but unless radical changes were made during the resolve, it is very clear when individual hunks are added, dropped or modified. Unlike the MQ technique, this can only compare a single pair of csets/patches at a time. Like the MQ method, this also highlights changes in the commit comment and other metadata. I originally tried monkey patching from the evolve extension, but that is too complicated given that it depends on the order the two different extensions are loaded. This functionality is also useful when comparing grafts however, so implementing it in the core is more than just convenience. The --change argument doesn't make much sense for this, but it isn't harmful so I didn't bother blocking it. The -I/-X options are ignored because of a limitation of cmdutil.export(). We'll fix that next.

extdiff: prepare sections of dodiff() for conditionalizing This is purely indenting under an unconditional branch, so that the actual changes in the next patch are clear. Feel free to fold them if desired.

hgweb: drop unused import

help/config: back out 5f2a1ebd6e78 This breaks building manpages, and by association breaks building debs. timeless has a fix coming, but it turns out we'll need to back out 5f2a1ebd6e78 anyway, so just back it out now to fix building packages.

filemerge: add non-interactive :merge-local and :merge-other There are two non-interactive internal merge tools, :other and :local, but they don't really merge, they just pick all changes from the local or other version of the file. In some situations, it is known that we want a merge and also know that all merge conflicts should be resolved in one direction. Although external merge tools can do this, sometimes it can be convenient to do so from within hg, without invoking a merge tool. These new :merge-local and :merge-other tools can do just that.

simplemerge: enable option to resolve conflicts one way With this change, the simplemerge algorithm grows an option to only return the local or the other hunk in a conflicting region.

templater: add new docheader/footer components for XML (issue4135) The existing header/footer components were templated per-changeset, and thus couldn't be correctly printed for an empty log

import-checker: use modern .endswith for multiple suffixes Suggested by Anton Shestakov <engored@ya.ru> on the list. Thanks!

hgweb: use separate repo instances per thread Before this change, multiple threads/requests could share a localrepository instance. This meant that all of localrepository needed to be thread safe. Many bugs have been reported telling us that localrepository isn't actually thread safe. While making localrepository thread safe is a noble cause, it is a lot of work. And there is little gain from doing so. Due to Python's GIL, only 1 thread may be processing Python code at a time. The benefits to multi-threaded servers are marginal. Thread safety would be a lot of work for little gain. So, we're not going to even attempt it. This patch establishes a pool of repos in hgweb. When a request arrives, we obtain the most recently used repository from the pool or create a new one if none is available. When the request has finished, we put that repo back in the pool. We start with a pool size of 1. For servers using a single thread, the pool will only ever be of size 1. For multi-threaded servers, the pool size will grow to the max number of simultaneous requests the server processes. No logic for pruning the pool has been implemented. We assume server operators either limit the number of threads to something they can handle or restart the Mercurial process after a certain amount of requests or time has passed.

hg: establish a cache for localrepository instances hgweb contained code for determining whether a cached localrepository instance was up to date. This code was way too low-level to be in hgweb. This functionality has been moved to a new "cachedlocalrepo" class in hg.py. The code has been changed slightly to facilitate use inside a class. hgweb has been refactored to use the new API. As part of this refactor, hgweb.repo no longer exists! We're very close to using a distinct repo instance per thread. The new cache records state when it is created. This intelligence prevents an extra localrepository from being created on the first hgweb request. This is why some redundant output from test-extension.t has gone away.

hgweb: create function to perform actions on new repo We perform some common tasks when a new repo instance is obtained. In preparation for changing how we obtain repo instances, factor this functionality into a standalone function.

hgweb: remove proxy to hgweb instance We were temporarily routing attributes until all request-specific attributes from hgweb were moved to requestcontext. We have finally reached that juncture and we can remove the proxy. At this point, only the repo instance is prone to race conditions between threads. This will be dealt with shortly.

dispatch: disable demandimport when invoking the debugger Turns out that demandimport confuses pudb, which does some pretty complicated imports. I think it's reasonable to disable demandimport here.

templater: create string unescape helper (issue4798) This gives us a unified place to do error-handling of string-escaping syntax errors

parsers: use PyTuple_New and SET_ITEM to construct metadata pair of markers With these 2 patches, fm1readmarkers() gets slightly faster: obsolete._fm1readmarkers() for 78644 entries 58.0 -> 56.2msec

parsers: use PyTuple_SET_ITEM() to fill new marker tuples Because we know these tuples have no member yet, PyTuple_SetItem() isn't necessary.

revset: uncache filteredset.__contains__ Since 96b6b3d78697, condition function returns a cached value, so there's little benefit to cache __contains__. No measurable difference found in contrib/base-revsets.txt.

hgweb: assign ctype to requestcontext The very existence of ctype is a bit hacky. But we roll with it. Before this patch, there was possibly a race condition between 2 threads handling file requests: 1 thread could set the ctype and another serving a different file would read and use that potentially wrong ctype.

hgweb: add reponame to requestcontext We have to use object.__setattr__ until the app proxy is gone.

hgweb: don't access self.repo during request processing We want all repository accesses to go through requestcontext.repo so the request is isolated from the application.

hgweb: extract _getview to own function While we're refactoring code, we might as well remove a method that doesn't need to be a method.

hgweb: regenerate web substitutions when repo is refreshed Previously, changes to the configuration would not be picked up by a running server. That feels like a bug. Regenerate the web substitutions table when the repository changes.

add: pass full=False to dirstate walk Previously cmdutil.add would call wctx.walk(), which under the hood calls dirstate.walk with full=True. This means it returns all of the clean files (which we don't need when computing the add set), as well as the unclean files. This results in 1) a lot more work being done and 2) this code path circumventing the hgwatchman extension, resulting in worse performance in hgwatchman environments ('hg add .' went from 9s to 1.8s).

hgweb: drop unused import

obsolete: improve English of successorssets

histedit: fix grammar in cleanupnode comment

hgweb.server: fix _httprequesthandlerssl help text

util: capitalize Python in MBTextWrapper._wrap_chunks comment

hgweb: remove ErrorResponse.message BaseException.message is deprecated: https://www.python.org/dev/peps/pep-0352/#retracted-ideas

manifest: switch add() to heapq.merge (available in Py2.6+)

archival: drop self.filename - deprecated in py2.6

templater: fix get English

help: fix makeitemsdoc English description

help: filesets show hg resolve command

help: filesets show hg status command

hgmanpage: fix grammar remove 's's from places where they don't belong insert the preposition 'in'

check-seclevel: fix file description grammar

profiling: allow logging profile to the blackbox This allows specifying '--config profiling.output=blackbox' which will log the profile output to the blackbox (if enabled). This is useful for doing profiling on the server since it allows us to record the command, it's result, any exceptions, and it's profile, all in one spot. And we get log rotation for free.

bookmark: improve ambiguous documentation for rename

ui: change default path fallback mechanism (issue4796) The previous paths API code always fell back to the default path. This was wrong because if a requested path doesn't exist, that should error. Only if no path was requested should we fall back to the default. As part of implementing the test case for issue 4796, it was discovered that the "repository does not exist" error message raised by localrepository.__init__ wasn't being seen because the paths API validates paths before localrepository.__init__ was being called. The exception and error message from localrepository.__init__ has been introduced to getpath(). This necessitated rewriting expandpath() both to catch the exception and to have proper default fallback. This code is more complicated than I'd like. But making all tests pass was a big chore. As more code moves to getpath(), there will likely be opportunities to improve things a bit.

templater: catch regexp error at sub() function This patch splits re.sub() into re.compile() and sub() so that it can distinguish which argument causes re.error.

graphmod: compute slow revset query once prior to reachableroots (issue4782) Because revsets query is evaluated lazily, "list(revs)" may take long for complicated query. So we shouldn't iterate revs many times. This patch is the easiest workaround for the issue4782. We could introduce more aggressive caching, but it wouldn't be as fast as the simple baseset operation. Gregory Szorc said "this makes `hg wip` on my Firefox clone ~4x faster than 3.5.1 (~6.5s to ~1.5s). This is after a regression in @ to ~45s."

profiling: add config option for enabling profiling Previously you could only enable profiling via the --profile option. This is awkward when trying to debug a server side operation. Let's add a config option to enable profiling. In the future, this could be extended to allow profiling a certain percentage of operations (and potentially reporting that information to an external service).

blackbox: add pid to output This adds the process id to the line header for the blackbox output. This is useful for distinguishing processes when using the blackbox on a server and many processes are writing to the blackbox at once.

exchange: allow fallbackheads to use lazy set behavior The common ancestor set implementation was made lazy a couple years ago, but this piece of code still required processing the entire repo by putting set() around the lazy set. The code was introduced in 5653f2d166ea, a year before the lazy ancestor set was added. Dropping the set() shaves 3.5 seconds off of 'push -r' in repos with hundreds of thousands of commits.