caches: introduce a 'debugupdatecaches' command That command make sure caches are updated. This is based on 'localrepo.updatecaches' so when we move support for new cache in that function this command will benefit from it.

caches: call 'repo.updatecache()' in 'repo.destroyed()' Regenerating the cache after a 'strip' or a 'rollback' is useful. So we call the generic cache warming function as other caches than just branchmap will be updated there in the future. To do so, we have to make 'repo.updatecache()' able to take no arguments. In such cases, we reload all caches.

caches: introduce a function to warm cache We have multiple caches that gain from being kept up to date. For example in a server setup, we want to make sure the branchcache cache is hot for other read-only clients. Right now each cache tries to update themself in place where new data have been added. However the approach is error prone (we might miss some spot) and fragile. When nested transaction are involved, such cache updates might happen before a top level transaction is committed. Writing caches for uncommitted data on disk. Having a single entry point, run at the end of each successful transaction, helps to ensure the cache is up to date and refreshed at the right time. We start with updating the branchmap cache but other will come.

transaction: track newly introduced revisions Tracking revisions is not the data that will unlock the most new capability. However, they are the simplest thing to track and still unlock some nice improvements in regard with caching. We plug ourself at the changelog level to make sure we do not miss any revision additions. The 'revs' set is configured at the repository level because the transaction itself does not needs to know that much about the business logic.

transaction: introduce "changes" dictionary to precisely track updates The transaction is already tracking some data intended for hooks (in 'hookargs'). However, that information is minimal as we optimise for passing data to other processes through environment variables. There are multiple places were we could use more complete and lower level information locally (eg: cache update, better report of changes to hooks, etc...). For this purpose we introduces a 'changes' dictionary on the transaction. It is intended to track every changes happening to the repository (eg: new revs, bookmarks move, phases move, obs-markers, etc). For now we just adds the 'changes' dictionary. We'll adds more tracking and usages over time.

clone: add a server-side option to disable full getbundles (pull-based clones) For large enough repositories, pull-based clones take too long, and an attempt to use them indicates some sort of configuration or other issue or maybe an outdated Mercurial. Add a config option to disable them.

clone: warn when streaming was requested but couldn't be performed This helps both users and the people who support them figure out why a stream clone couldn't be performed. In an upcoming patch we're going to add a way for servers to hard abort on a full getbundle. In those cases servers might expect clients to perform a stream clone, so it's important to communicate why one couldn't be done.

clone: test streaming disabled because client is missing requirement Turns out we had no coverage for this important case.

bundle2: don't check for whether we can do stream clones At the moment this isn't used and all stream clones use the legacy protocol. In an upcoming diff, canperformstreamclone will print out a message if a stream clone was requested but couldn't happen for some reason. Removing this call ensures the message isn't printed twice.

debugcommands: add debugpickmergetool to examine which merge tool is chosen Before this patch, there is no convenient way to know which merge tool is chosen for each managed files without actual merging.

filemerge: add internal merge tool to dump files forcibly Internal merge tool :dump implies premerge. Therefore, files aren't dumped, if premerge runs successfully. This undocumented behavior might confuse users, if they want to always dump files. But just making :dump omit premerge might cause backward compatibility issue for existing automation. This patch adds new internal merge tool :forcedump, which works as same as :dump, but omits premerge always. Internal tools annotated with "nomerge" should merge "change and delete" correctly, but _forcedump() can't. Therefore, it is annotated with "mergeonly" to always omit premerge, even though it doesn't merge files actually. This patch also adds explanation about premerge to :dump, to clarify how :dump actually works. BTW, this patch specifies internal tools with "internal:" prefix in newly added test scenario in test-merge-tools.t, even though this prefix is already deprecated. This is only for similarity to other tests in test-merge-tools.t.

filemerge: make warning message more i18n friendly Before this patch, " specified for " part of warning messages (e.g. "couldn't find merge tool TOOL specified for PAT") isn't translatable.

filemerge: show warning about choice of :prompt only at an actual fallback Before this patch, internal merge tool :prompt shows "no tool found to merge FILE" line, even if :prompt is explicitly specified as a tool to be used. This patch shows warning message about choice of :prompt only at an actual fallback, in which case any tool is rejected by capability for binary or symlink. It is for backward compatibility to omit warning message in "changedelete" case.

treemanifest: allow manifestrevlog to take an explicit treemanifest arg Previously we relied on the opener options to tell the revlog to be a tree manifest. This makes it complicated for extensions to create treemanifests and normal manifests at the same time. Let's add a construtor argument to create a treemanifest revlog as well. I considered removing the options['treemanifest'] logic from manifestrevlog entirely, but doing so shifts the responsibility to the caller which ends up requiring changes in localrepo, bundlerepo, and unionrepo. I figured having the dual mechanism was better than polluting other parts of the code base with treemanifest knowledge.

policy: relax the default for in-place build We're going to make the 'c' policy more strict, where no missing attribute will be allowed. Since we want to run 'hg bisect' without rebuilding the C extension modules, we'll need a looser policy for development environment. The default for system installation isn't changed. Note that the current 'c' policy is practically 'allow'-ish as we have lots of adhoc fallbacks to pure functions.

verify: always check rawsize Previously, verify only checks "rawsize == len(rawtext)", if "len(fl.read()) != fl.size()". With flag processor, "len(fl.read()) != fl.size()" does not necessarily mean "rawsize == len(rawtext)". So we may miss a useful check. This patch removes the "if len(fl.read()) != fl.size()" condition so the rawsize check is always performed. With the condition removed, "fl.read(n)" looks unnecessary so a comment was added to explain the side effect is wanted.

rebase: rename "target" to "destination" in messages The help text for rebase calls it "the destination" (never "target"), so let's use that in messages as well.

rebase: rename "target" to "dest" in variable names It took me a while to figure out that "target" was actually what's passed to --dest.

sslutil: remove conditional cipher code needed for Python 2.6 We dropped support for Python 2.6. So this code to work around a missing feature on 2.6 is no longer necessary.

merge: use repo.wvfs.setflags() instead of util.setflags() Most merge.py code goes through the vfs instead of maniulating files directly, so let's do the same here.

merge with stable

revlog: move part of "addrevision" to "addrawrevision" "addrawrevision" will be the public API to reuse revision rawdata elsewhere. It will be used by a future patch.

filectx: add an overlayfilectx class The end goal is to make it possible to avoid potential expensive fctx.data() when unnecessary. While memctx is useful for creating new file contexts, there are many cases where we could reuse an existing raw file revision (amend, histedit, rebase, process a revision constructed by a remote peer, etc). The overlayfilectx class is made to support such reuse cases. Together with a later patch, hash calculation and expensive flag processor could be avoided.

filectx: remove __new__ It does not seem to be used anywhere, and breaks a later patch.

filectx: add a rawflags method The new method returns the low-level revlog flag. We already have "rawdata" so a "rawflags" makes sense. Both "rawflags" and "rawdata" will be used in a later patch.

filectx: move size to basefilectx See previous patch for context - avoid code duplication.

filectx: make renamed a property cache See previous patch for context - mainly to avoid code duplication.

filectx: make flags a property cache Make basefilectx._flags a property cache, so basefilectx.flags() could be reasonably reused. This avoids code duplication between memfilectx and a class added in a later patch.

commandserver: move printbanner logic to bindsocket bindsocket now handles listen automatically. "printbanner" seems to be just a part of "bindsocket". This simplifies the interface a bit.

commandserver: move "listen" responsibility from service to handler This enables chg to replace a server socket in an atomic way: 1. bind to a temp address 2. listen 3. rename Currently 3 happens before 2 so a client may see the socket file but fails to connect to it.

hghave: remove py27+ capability It is now unused. And we require Python 2.7+ now so this check is not necessary.

tests: remove test targeting Python 2.6 We just removed support for Python 2.7. This test is dead since it only ran on <2.7.

setup: drop support for Python 2.6 (BC) Per discussion on the mailing list and elsewhere, we've decided that Python 2.6 is too old to continue supporting. We keep accumulating hacks/fixes/workarounds for 2.6 and this is taking time away from more important work. So with this patch, we officially drop support for Python 2.6 and require Python 2.7 to run Mercurial.

perf: move revlog construction and length calculation out of benchmark We don't need to measure the time it takes to open the revlog or calculate its length. This is more consistent with what other perf* functions do. While I was here, I also renamed the revlog variable from "r" to "rl" - again in the name of consistency.

perf: clear revlog caches on every iteration cmdutil.openrevlog() may return a cached revlog instance. This /may/ be a recent "regression" due to refactoring of the manifest API. I'm not sure. Either way, this perf command was broken for at least manifests because subsequent invocations of the perf function would get cache hits from previous invocations, invalidating results. In the extreme case, testing the last revision in the revlog resulted in near-instantanous execution of subsequent runs (since the fulltext is cached). A time of ~1us would be reported in this case.

perf: don't convert rev to node before calling revlog.revision()

revlog: rename _chunkraw to _getsegmentforrevs() This completes our rename of internal revlog methods to distinguish between low-level raw revlog data "segments" and higher-level, per-revision "chunks." perf.py has been updated to consult both names so it will work against older Mercurial versions.

perf: store reference to revlog._chunkraw in a local variable To prepare for renaming revlog._chunkraw, we stuff a reference to this metho in a local variable. This does 2 things. First, it moves the attribute lookup outside of a loop, which more accurately measures the time of the code being invoked. Second, it allows us to alias to different methods depending on their presence (perf.py needs to support running against old Mercurial versions). Removing an attribute lookup from a tigh loop appears to shift the numbers slightly with mozilla-central: $ hg perfrevlogchunks -c ! read ! wall 0.354789 comb 0.340000 user 0.330000 sys 0.010000 (best of 28) ! wall 0.335932 comb 0.330000 user 0.290000 sys 0.040000 (best of 30) ! read w/ reused fd ! wall 0.342326 comb 0.340000 user 0.320000 sys 0.020000 (best of 29) ! wall 0.332857 comb 0.340000 user 0.290000 sys 0.050000 (best of 30) ! read batch ! wall 0.023623 comb 0.020000 user 0.000000 sys 0.020000 (best of 124) ! wall 0.023666 comb 0.020000 user 0.000000 sys 0.020000 (best of 125) ! read batch w/ reused fd ! wall 0.023828 comb 0.020000 user 0.000000 sys 0.020000 (best of 124) ! wall 0.023556 comb 0.020000 user 0.000000 sys 0.020000 (best of 126)

revlog: rename internal functions containing "chunk" to use "segment" Currently, "chunk" is overloaded in revlog terminology to mean multiple things. One of them refers to a segment of raw data from the revlog. This commit renames various methods only used within revlog.py to have "segment" in their name instead of "chunk." While I was here, I also made the names more descriptive. e.g. "_loadchunk()" becomes "_readsegment()" because it actually does I/O.

fsmonitor: do not nuke dirstate filecache In the future, chg may prefill repo's dirstate filecache so it's valuable and should be kept. Previously we drop both filecache and property cache for dirstate during fsmonitor reposetup, this patch changes it to only drop property cache but keep the filecache.

perf: move gettimer() call This is more consistent with other perf* functions.

perf: don't clobber startrev variable Previously, the "startrev" argument would be ignored due to "startrev = 0" in the benchmark function. This meant that `hg perfrevlog` always started at revision 0. Rename the local variable to "beginrev" so the variable does the right thing.

bundle: add optional 'tagsfnodecache' data to on disk bundle (issue5543) This should help performance when unbundling.

bundle2: move tagsfnodecache generation in a generic function This will help us reusing the logic for `hg bundle`.

bundle: introduce an higher level function to write bundle on disk The current function ('writebundle') is focussing on getting an existing changegroup to disk. It is no easy ways to includes more part in the generated bundle2. So we introduce a slightly higher level function that is fed the 'outgoing' object (that defines the bundled spec) and the bundlespec parameters (to control the changegroup generation and inclusion of other parts). This is creating the third logic dedicated to create a consistent bundle2 (the other 2 are the push code and the getbundle code). We should probably reconcile them at some points but they all takes different types of input. So we need to introduce an intermediate "object" that each different input could be converted to. Such unified "bundle2 specification" could be fed to some unified code. We start by having the `hg bundle` related code on its own to helps defines its specific needs first. Once the common and specific parts of each logic will be known we can start unification.

bundle: handle compression earlier We can also handle that part before starting any generation.

bundle: check changegroup version earlier We can check if we know how to bundle this changegroup version before actually starting to generate the changegroup.

bundle: check lack of revs to bundle before generating the changegroup We already have the information so we can check it earlier.

extdiff: copy back files to the working directory if the size changed In theory, it should be enough to pay attention only to the modification time when detecting if a snapshotted working directory file changed. In practice, BeyondCompare preserves all file attributes when syncing files at the directory level. (If you open the file and sync individual hunks, then mtime does change, and everything was being copied back as desired.) I'm not sure how many other synchronization tools would trigger this issue, but it's annoyingly inconsistent (if a single file is diffed, it isn't snapshotted, so the same BeyondCompare file sync operation _is_ visible, because wdir() is updated in place. I filed a bug with them, and they stated it is on their wish list, but won't be fixed in the near term. This isn't a complete fix (there is still the case of the size not changing), but this seems like a trivial enough change to fix most of the problem. I suppose we could fool around with making files in the other snapshot readonly, and copy back if we see the readonly bit copied. That seems pretty hacky though, and only works if the external tool copies all attributes.

test-extdiff: enable a previously failing test on Windows

test-extdiff: narrow the range of an '#if execbit' block Now that output can be conditionalized, the few `chmod +x` specific outputs can be conditionalized, and the rest of the tests run as normal. Disable one test that is failing on Windows for now.

test-extdiff: deduplicate tests

test-extdiff: fill in a missing Windows test

policy: eliminate ".pure." from module name only if marked as dual So we can switch cext/pure modules to new layout one by one.

policy: add "cext" package which will host CPython extension modules I'm going to restructure cext/pure modules and get rid of our hgimporter hack. C extension modules will be moved to cext/ directory so old and new compiled modules can coexist in development tree. This is necessary to run 'hg bisect' without recompiling. New extension modules will be loaded by an importer function: base85 = policy.importmod('base85') # select pure.base85 or cext.base85 This will also allow us to split cffi from pure modules, which is currently difficult because pure modules can't be imported by name.

policy: mark all string literals as sysstr or bytes The policy module won't be imported early in future, which means string literals will be processed by our Python 3 loader.

debuginstall: check C extensions only if they are loadable per policy This check is useless in pure installation and I want to make it directly import C extension modules.

osutil: proxy through util (and platform) modules (API) See the previous commit for why. Marked as API change since osutil.listdir() seems widely used in third-party extensions. The win32mbcs extension is updated to wrap both util. and windows. aliases.

mpatch: proxy through mdiff module See the previous commit for why.

bdiff: proxy through mdiff module See the previous commit for why. mdiff seems a good place to host bdiff functions. bdiff.bdiff was already aliased as textdiff, so we use it.