util: rewrite sortdict using Python 2.7's OrderedDict Pattern copied from https://docs.python.org/dev/library/collections.html#ordereddict-examples-and-recipes.

encoding: use i.startswith() instead of i[0] to eliminate py2/3 divergence

test-casefolding: sync with latest code changes This goes with c2380b448265.

perf: always pass node to revlog.revision() I removed this in 73c3e226d2fc thinking it wasn't necessary. In fact, we need to always pass a node so the code is compatible with revisions before d7d64b89a65c. The new code uses a variable to avoid check-style complaining about "r.revision(r.node(" patterns.

hg: backout optimizing for treemanifests It turns out that the files list is not sufficient to identify with revlogs have changed. In a merge commit, no files could've changed but directories would have. For now let's just backout this optimization.

test-hardlink: do not test .hg/cache/checklink Linux and BSD have different behavior on "os.link(src, dst)" where "src" is a symlink. That causes test difference. According to POSIX [1]: If path1 names a symbolic link, it is implementation-defined whether link() follows the symbolic link, or creates a new link to the symbolic link itself. So both behaviors are correct. This patch removes the trouble maker "checklink" to make the test pass on both platforms. [1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/link.html

test-hardlinks: unify two test files into one The "whitelisted" test could be unified using "#if"s.

style: ban [ foo == bar] bashism in tests

changelog: load pending file directly When changelogs are written, a copy of the index (or inline revlog) may be written to an 00changelog.i.a file to facilitate hooks and other processes having access to the pending data before it is finalized. The way it works today, the localrepo class loads the changelog like normal. Then, if it detects a pending transaction, it asks the changelog class to load a pending changelog. The changelog class looks for a 00changelog.i.a file. If it exists, it is loaded and internal data structures on the new revlog class are copied to the original instance. The existing mechanism is inefficient because it loads 2 revlog files. The index, node map, and chunk cache for 00changelog.i are thrown away and replaced by those for 00changelog.i.a. The existing mechanism is also brittle because it is a layering violation to access the data structures being accessed. For example, the code copies the "chunk cache" because for inline revlogs this cache contains the raw revision chunks and allows the original changelog/revlog instance to access revision data for these pending revisions. This whole behavior of course relies on the revlog constructor reading the entirety of an inline revlog into memory and caching it. That's why it is brittle. (I discovered all this as part of modifying behavior of the chunk cache.) This patch streamlines the loading of a pending 00changelog.i.a revlog by doing it directly in the changelog constructor if told to do so. When this code path is active, we no longer load the 00changelog.i file at all. The only negative outcome I see from this change is if loading 00changelog.i was somehow facilitating a role. But I can't imagine what that would be because we throw away its data (the index data structures are replaced and inline revision data is replaced via the chunk cache) and since 00changelog.i.a is a copy of 00changelog.i, file content should be identical, so there should be no meaninful file integrity checking at play. I think this was all just sub-optimal code.

cleanup: use set literals We no longer support Python 2.6, so we can now use set literals.

py3: convert date and format arguments str before passing in time.strptime time.strptime() raises ValueError if the arguments are not str. Source Code: https://hg.python.org/cpython/file/3.5/Lib/_strptime.py#l307

py3: convert kwargs' keys to str using pycompat.strkwargs

verify: add a config option to skip certain flag processors Previously, "hg verify" verifies everything, which could be undesirable when there are expensive flag processor contents. This patch adds a "verify.skipflags" developer config. A flag processor will be skipped if (flag & verify.skipflags) == 0. In the LFS usecase, that means "hg verify --config verify.skipflags=8192" will not download all LFS blobs, which could be too large to be stored locally. Note: "renamed" is also skipped since its default implementation may call filelog.data() which will trigger the flag processor.

changegroup: add bundlecaps back Commit 282b288aa20c333c removed the unused bundlecaps argument from the changegroup code. While it is unused in core Mercurial, it was an important feature for the remotefilelog extension because it allowed the exchange layer to communicate to the changegroup packer that this was a shallow repo and that filelogs should not be included. Without bundlecaps, there is currently no other way to pass that information along without a more extensive refactor of exchange, bundle, and changegroup code. This patch backs out the original removal, and merges it with some recent changes to changegroup apis.

flagprocessor: add a fast path when flags is 0 When flags is 0, _processflags could be a no-op instead of iterating through the flag bits.

shelve: make shelvestate use simplekeyvaluefile Currently shelvestate uses line ordering to differentiate fields. This makes it hard for extensions to wrap shelve, since if two alternative versions of code add a new line, correct merging is going to be problematic. simplekeyvaluefile was introduced fot this purpose specifically. After this patch: - shelve will always write a simplekeyvaluefile - unshelve will check the first line of the file for a version, and if the version is 1, will read it in a position-based way, if the version is 2, will read it in a key-value way As discussed with Yuya previously, this will be able to handle old-style shelvedstate files, but old Mercurial versions will fail on the attempt to read shelvedstate file of version 2 with a self-explanatory message: 'abort: this version of shelve is incompatible with the version used in this repo'

shelve: refactor shelvestate loading This is a preparatory patch which separates file reading from the minimal validation we have (like turning version into int and checking that this version is supported). The purpose of this patch is to be able to read statefile form simplekeyvaluefile, which is implemented in the following patch.

extdiff: copy back execbit-only changes to the working directory Some tools like BeyondCompare allow the file mode to be changed. The change was previously applied if the content of the file changed (either according to size or mtime), but was not being copied back for a mode-only change. That would seem to indicate handling this in an 'elif' branch, but I opted not to in order to avoid copying back the mode without the content changes when mtime and size are unchanged. (Yes, that's a rare corner case, but all the more reason not to have a subtle difference in behavior.) The only way I can think to handle this undetected change is to set each file in the non-wdir() snapshot to readonly, and check for that attribute (as well as mtime) when deciding to copy back. That would avoid the overhead of copying the whole file when only the mode changed. But a chmod in a diff tool is likely rare. See also affd753ddaf1.

tests: remove regular expression matching for Python 2.6 This effectively reverts 52cca17ac523. Some lines still have (re) due to variable length port numbers. There's not much we can do about that. But at least this change removes most of the ugliness.

branchmap: remove use of buffer() to support Python 2.6 The use of buffer() was added in 7359157b9e46 to support Python 2.6, which we no longer support.

py3: remove delayed import of importlib All supported versions of Python now have importlib. This effectively reverts b85fa6bf298b.

tests: use context manager form of assertRaises Support for using unittest.TestCase.assertRaises as a context manager was added in Python 2.7. This form is more readable, especially for complex tests. While I was here, I also restored the use of assertRaisesRegexp, which was removed in c6921568cd20 for Python 2.6 compatibility.

obsolete: use 2 argument form of enumerate() The 2 argument form of enumerate was added in Python 2.6. This change effectively reverts 10880c8aad85.

tests: remove special handling for undefined memoryview 'memoryview' was introduced in Python 2.7. 4adc090fa2fb added code to filterpyflakes.py to ignore "undefined name 'memoryview'" pyflakes warnings. Since we no longer support <Python 2.7, we can remove this workaround.

encoding: remove workaround for locale.getpreferredencoding() locale.getpreferredencoding() was buggy in OS X for Python <2.7. Since we no longer support Python <2.7, we no longer need this workaround. This essentially reverts 2be70ca17311.

mail: remove code to support < Python 2.7 This code was added in 594b98846ce1. Since we no longer support Python <2.7, it can be removed.

help: clarify that colons are allowed in fingerprints values This was suggested by Lars Rohwedder in issue5559.

sslutil: tweak the legacy [hostfingerprints] warning message Lars Rohwedder noted in issue5559 that the previous wording was confusing. I agree.

rebase: allow rebase even if some revisions need no rebase (BC) (issue5422) This allows you to do e.g. "hg rebase -d @ -r 'draft()'" even if some drafts are already based off of @. You'd still need to exclude obsolete and troubled revisions, though. We will deal with those cases later. Implemented by treating state[rev]==rev as "no need to rebase". I considered adding another fake revision number like revdone=-6. That would make the code clearer in a few places, but would add extra code in other places. I moved the existing test out of test-rebase-base.t and into a new file and added more tests there, since not all are using --base.

chgserver: more explicit about sensitive environ variables Environment variables like HGUSER, HGEDITOR, HGEDITFROM should not trigger a new chgserver. This patch uses a whitelist for environ variables starting with "HG" to reduce the number of servers. I have went through `grep -o "[\"']HG[A-Z_0-9]*['\"]" -hR . | sort -u` so the list should be up-to-date.

scmutil: make simplekeyvaluefile able to have a non-key-value first line To ease migration from files with version numbers in their first lines, we want simplekeyvaluefile to support a non-key-value first line. In this way, old versions of Mercurial will read such files, discover a newer version than the one they know how to handle and fail gracefully, rather than with exception. Shelve's shelvestate file is an example.

scmutil: add simplekeyvaluefile reading test Before this patch, mockvfs did not emulate readlines correctly and there was no test for simplekeyvaluefile reading.

caches: stop warming the cache after changegroup application Now that we garantee that branchmap cache is updated at the end of the transaction we can drop this update. This removes a problematic case with nested transaction where the new cache could be written on disk before the transaction is finished (and even roll-backed) Such premature cache write was visible in the following test: * tests/test-acl.t * tests/test-rebase-conflicts.t In addition, running the cache update later means having more date about the state of the repository (in particular: phases). So we can generate caches with more information. This creates harmless changes to the following tests: * tests/test-hardlinks-whitelisted.t * tests/test-hardlinks.t * tests/test-phases.t * tests/test-tags.t * tests/test-inherit-mode.t

caches: move the 'updating the branch cache' message in 'updatecaches' We are about to remove the branchmap cache update in changegroup application. There is a debug message alongside this update that we do not want to loose. We move the message beforehand to simplify the test update in the next changeset. The message move is quite noisy and isolating that noise is useful. Most tests update are just line reordering since the message is issued at a later point during the transaction. After this changes, the message is displayed in more case since local commit creation also issue it.

caches: stop warming the cache after 'localrepo.commitctx' Now that we garantee that branchmap cache are updated at the end of the transaction we can drop that one. This removes a problematic case with nested transaction where the new cache could be written on disk before the transaction is finished. The test change is harmless, since we update the cache at a later point, the dirstate have been updated in between.

caches: introduce a 'debugupdatecaches' command That command make sure caches are updated. This is based on 'localrepo.updatecaches' so when we move support for new cache in that function this command will benefit from it.

caches: call 'repo.updatecache()' in 'repo.destroyed()' Regenerating the cache after a 'strip' or a 'rollback' is useful. So we call the generic cache warming function as other caches than just branchmap will be updated there in the future. To do so, we have to make 'repo.updatecache()' able to take no arguments. In such cases, we reload all caches.

caches: introduce a function to warm cache We have multiple caches that gain from being kept up to date. For example in a server setup, we want to make sure the branchcache cache is hot for other read-only clients. Right now each cache tries to update themself in place where new data have been added. However the approach is error prone (we might miss some spot) and fragile. When nested transaction are involved, such cache updates might happen before a top level transaction is committed. Writing caches for uncommitted data on disk. Having a single entry point, run at the end of each successful transaction, helps to ensure the cache is up to date and refreshed at the right time. We start with updating the branchmap cache but other will come.

transaction: track newly introduced revisions Tracking revisions is not the data that will unlock the most new capability. However, they are the simplest thing to track and still unlock some nice improvements in regard with caching. We plug ourself at the changelog level to make sure we do not miss any revision additions. The 'revs' set is configured at the repository level because the transaction itself does not needs to know that much about the business logic.

transaction: introduce "changes" dictionary to precisely track updates The transaction is already tracking some data intended for hooks (in 'hookargs'). However, that information is minimal as we optimise for passing data to other processes through environment variables. There are multiple places were we could use more complete and lower level information locally (eg: cache update, better report of changes to hooks, etc...). For this purpose we introduces a 'changes' dictionary on the transaction. It is intended to track every changes happening to the repository (eg: new revs, bookmarks move, phases move, obs-markers, etc). For now we just adds the 'changes' dictionary. We'll adds more tracking and usages over time.

clone: add a server-side option to disable full getbundles (pull-based clones) For large enough repositories, pull-based clones take too long, and an attempt to use them indicates some sort of configuration or other issue or maybe an outdated Mercurial. Add a config option to disable them.

clone: warn when streaming was requested but couldn't be performed This helps both users and the people who support them figure out why a stream clone couldn't be performed. In an upcoming patch we're going to add a way for servers to hard abort on a full getbundle. In those cases servers might expect clients to perform a stream clone, so it's important to communicate why one couldn't be done.

clone: test streaming disabled because client is missing requirement Turns out we had no coverage for this important case.

bundle2: don't check for whether we can do stream clones At the moment this isn't used and all stream clones use the legacy protocol. In an upcoming diff, canperformstreamclone will print out a message if a stream clone was requested but couldn't happen for some reason. Removing this call ensures the message isn't printed twice.

debugcommands: add debugpickmergetool to examine which merge tool is chosen Before this patch, there is no convenient way to know which merge tool is chosen for each managed files without actual merging.

filemerge: add internal merge tool to dump files forcibly Internal merge tool :dump implies premerge. Therefore, files aren't dumped, if premerge runs successfully. This undocumented behavior might confuse users, if they want to always dump files. But just making :dump omit premerge might cause backward compatibility issue for existing automation. This patch adds new internal merge tool :forcedump, which works as same as :dump, but omits premerge always. Internal tools annotated with "nomerge" should merge "change and delete" correctly, but _forcedump() can't. Therefore, it is annotated with "mergeonly" to always omit premerge, even though it doesn't merge files actually. This patch also adds explanation about premerge to :dump, to clarify how :dump actually works. BTW, this patch specifies internal tools with "internal:" prefix in newly added test scenario in test-merge-tools.t, even though this prefix is already deprecated. This is only for similarity to other tests in test-merge-tools.t.

filemerge: make warning message more i18n friendly Before this patch, " specified for " part of warning messages (e.g. "couldn't find merge tool TOOL specified for PAT") isn't translatable.

filemerge: show warning about choice of :prompt only at an actual fallback Before this patch, internal merge tool :prompt shows "no tool found to merge FILE" line, even if :prompt is explicitly specified as a tool to be used. This patch shows warning message about choice of :prompt only at an actual fallback, in which case any tool is rejected by capability for binary or symlink. It is for backward compatibility to omit warning message in "changedelete" case.

treemanifest: allow manifestrevlog to take an explicit treemanifest arg Previously we relied on the opener options to tell the revlog to be a tree manifest. This makes it complicated for extensions to create treemanifests and normal manifests at the same time. Let's add a construtor argument to create a treemanifest revlog as well. I considered removing the options['treemanifest'] logic from manifestrevlog entirely, but doing so shifts the responsibility to the caller which ends up requiring changes in localrepo, bundlerepo, and unionrepo. I figured having the dual mechanism was better than polluting other parts of the code base with treemanifest knowledge.

policy: relax the default for in-place build We're going to make the 'c' policy more strict, where no missing attribute will be allowed. Since we want to run 'hg bisect' without rebuilding the C extension modules, we'll need a looser policy for development environment. The default for system installation isn't changed. Note that the current 'c' policy is practically 'allow'-ish as we have lots of adhoc fallbacks to pure functions.

verify: always check rawsize Previously, verify only checks "rawsize == len(rawtext)", if "len(fl.read()) != fl.size()". With flag processor, "len(fl.read()) != fl.size()" does not necessarily mean "rawsize == len(rawtext)". So we may miss a useful check. This patch removes the "if len(fl.read()) != fl.size()" condition so the rawsize check is always performed. With the condition removed, "fl.read(n)" looks unnecessary so a comment was added to explain the side effect is wanted.

rebase: rename "target" to "destination" in messages The help text for rebase calls it "the destination" (never "target"), so let's use that in messages as well.

rebase: rename "target" to "dest" in variable names It took me a while to figure out that "target" was actually what's passed to --dest.

sslutil: remove conditional cipher code needed for Python 2.6 We dropped support for Python 2.6. So this code to work around a missing feature on 2.6 is no longer necessary.

merge: use repo.wvfs.setflags() instead of util.setflags() Most merge.py code goes through the vfs instead of maniulating files directly, so let's do the same here.

merge with stable

revlog: move part of "addrevision" to "addrawrevision" "addrawrevision" will be the public API to reuse revision rawdata elsewhere. It will be used by a future patch.

filectx: add an overlayfilectx class The end goal is to make it possible to avoid potential expensive fctx.data() when unnecessary. While memctx is useful for creating new file contexts, there are many cases where we could reuse an existing raw file revision (amend, histedit, rebase, process a revision constructed by a remote peer, etc). The overlayfilectx class is made to support such reuse cases. Together with a later patch, hash calculation and expensive flag processor could be avoided.

filectx: remove __new__ It does not seem to be used anywhere, and breaks a later patch.

filectx: add a rawflags method The new method returns the low-level revlog flag. We already have "rawdata" so a "rawflags" makes sense. Both "rawflags" and "rawdata" will be used in a later patch.