lfs: restore the local blob store after a repo upgrade This also ends up testing the local extension wrapping for dstrepo during upgrade, which was fixed in 06987c6971be.

lfs: override walk() in lfsvfs In order to fix the missing lfs store after an upgrade, I attempted to walk the store vfs to hardlink to the upgraded repo's store. But the custom join() clashes with the default walk() implementation. First, 'path=None' blew up in the regex matcher, because it wanted a string. But even if that is fixed, the join to walk the root of the vfs wouldn't match the required xx/xx...xx pattern. The first cut of this was a copy/paste/tweak of the base implementation, but this version of walk() hides the internal directories, and treats the vfs as a flat store. I think this makes sense because most vfs methods call join() on input paths, which wants the simple oid format. It also relieves the caller from having to deal with bogus files/directories in the store.

tests: add coverage for preserving 'lfs' requirement on repo upgrade The test also shows that the local blob store is erroneously lost.

test-upgrade-repo: glob away timing values

py3: handle keyword arguments correctly in httppeer.py Differential Revision: https://phab.mercurial-scm.org/D1635

py3: handle keyword arguments correctly in httpconnection.py Differential Revision: https://phab.mercurial-scm.org/D1634

py3: handle keyword arguments correctly in hook.py Differential Revision: https://phab.mercurial-scm.org/D1633

py3: handle keyword arguments correctly in help.py Differential Revision: https://phab.mercurial-scm.org/D1632

py3: handle keyword arguments correctly in exchange.py Differential Revision: https://phab.mercurial-scm.org/D1631

py3: handle keyword arguments correctly in dispatch.py Differential Revision: https://phab.mercurial-scm.org/D1630

py3: handle keyword arguments correctly in context.py Differential Revision: https://phab.mercurial-scm.org/D1629

py3: handle keyword arguments correctly in commands.py Differential Revision: https://phab.mercurial-scm.org/D1628

py3: handle keyword arguments correctly in color.py Differential Revision: https://phab.mercurial-scm.org/D1627

py3: handle keyword arguments correctly in cmdutil.py Differential Revision: https://phab.mercurial-scm.org/D1626

py3: handle keyword arguments correctly in archival.py Differential Revision: https://phab.mercurial-scm.org/D1625

py3: handle keyword arguments correctly in hgext/largefiles/ Keys of keyword arguments must be str(unicode) on Python 3. The transformer which we use on Python 3, appends b'' in front of each string literal, so this may lead in KeyError or None return even when the key is present by we are using bytes value and it's stored in unicodes. This patch and all the similar patches handle this by either converting the keys of kwargs to bytes using 'pycompat.byteskwargs()' or adding r'' so that the transformer won't append b''. This next 23 patches follows the above mentioned way to handle keyword arguments. Differential Revision: https://phab.mercurial-scm.org/D1624

remotenames: rename related file and storage dir to logexchange This patch renames remotenames.py to logexchange.py, test-remotenames.t to test-logexchange.t. Also this patch renames the directory in which the data is stored from remotenames to logexchange. After this patch, data about bookmarks and branches from a server we pull is stored in `.hg/logexchange/(bookmarks|branches)` files. Thanks to smf for the suggestion. Differential Revision: https://phab.mercurial-scm.org/D1607

lfs: allow to run 'debugupgraderepo' on repo with largefiles The extensions wrap the necessary function to ensure the 'lfs' requirements won't be dropped. It is now possible to run `hg debugupgraderepo` on a repository with lfs.

upgrade: add a 'redeltafullall' mode We add a new mode for delta recomputation, when selected, each full text will go through the full "addrevision" mechanism again. This is slower than "redeltaall" but this gives the opportunity for extensions to trigger special logic. For example, the lfs extensions can decide to promote some revision to lfs storage during the upgrade.

upgrade: use actual filelog to convert filelog Extensions can add extra logic related to the config, so we must use the actual class. The path used needs minimal transformation for this to work.

upgrade: more standard creation of the temporary repository By using the standard path to create a repository we fill some hole in the current initialization process. The one who triggered this changeset was the lack of extensions initialization.

upgrade: use the repository 'ui' as the base for the new repository The `repo.baseui` contains all the configuration but the one specific to the repository (so it can be used when dealing with local peer and sub- repository). However, we need the repository config to be taken into account when doing the upgrade. Otherwise, the upgrade related config that exists in the repository config won't be taken into account when performing the update. A buggy and surprising behavior. We had to work around protection set around `repo.ui.copy` since we are an uncommon case.

upgrade: add a test to show the repository config being ignored The upgrade process ignores the config within the repository. The next changeset fixes it, but we introduce this test before to show it actually tests our target.

upgrade: register compression as a format variants Compression is a promising vector for speedup, let us make it easier to check the compression used and upgrade existing repository.

debugformat: handle non-boolean value for variant

debugformat: update label depending on value difference The new label highlight areas where the repo format differs from current config or default. This should help people spot area where a repository mismatch with the expected state.

debugformat: add data about the config when verbose In verbose mode, the command also displays the current configuration choice for the variant and the global Mercurial default for it.

debugformat: add a 'debugformat' command The command displays basic data about all format variants registered for repo upgrades. This gives a quick way to peek into a repository format. The 'fm.write()' calls are very independent because more data will be added in later changeset. Having more separate call make the later patch clearer.

upgrade: rename 'removecldeltachain' to 'plain-cl-delta' The new naming is more descriptive of a "state" while the older one was more about "action". I'm looking into command exposing more of data about the state of the repository so "state" oriented work better there. The key has not been made public anywhere outside the debug area so it is fine to update it.

tests: glob away ' that doesn't occur on all platforms in lfs test Differential Revision: https://phab.mercurial-scm.org/D1623

rebase: extract _assignworkingcopy Slightly cleaner, and makes it easier to hook this in an extension. Differential Revision: https://phab.mercurial-scm.org/D1617

rebase: disable `inmemory` if the rebaseset contains the working copy As described in the comment, rebasing the working copy parent with in-memory merge, and then updating to the new commit, isn't much faster because of the extra overhead of uppdating. Best to leave it off in that case. This commit makes deploying in-memory merge via an extension easier, because you can just set `inmemory=True` based on some config or probability, and this will turn off the cases where it's not desired. Differential Revision: https://phab.mercurial-scm.org/D1616

clone: add support for storing remotenames while cloning If `experimental.remotenames` is set to True, we store the remotenames in case of `hg pull`. This patch adds that support to clone command also. Differential Revision: https://phab.mercurial-scm.org/D1601

revset: use phasecache.getrevset to calculate public() Other revsets like secret(), draft(), _nonpublic() are using phasescache.getrevset already. The latter is more efficient after D1606. So let's migrate the public() revset function too. Tested using: $ hg debugshell --hidden --cwd hg-committed` In [1]: %timeit len(repo.revs('public()')) * Before D1606: 10 loops, best of 3: 22.5 ms per loop * Before this change, after D1606: 10 loops, best of 3: 28.6 ms per loop * After this change: 10 loops, best of 3: 20.2 ms per loop Therefore `public()` revset becomes even slightly faster after the data structure change by D1606. A similar performance win could also be observed on a large repo. A side effect is `phasecache.getrevset` needs to take a `subset` parameter. That was added with a default value so it won't cause BC issues. Differential Revision: https://phab.mercurial-scm.org/D1620

highlight: eagerly discover plugin lexers while demandimport is off Since highlight is only relevant for servers, it seems worthwhile to just trigger this eagerly, which avoids really weird traceback problems caused by demandimport messing with some of the lexer plugins. Differential Revision: https://phab.mercurial-scm.org/D1619

patch: move part of tabsplitter logic in _inlinediff It cannot be entirely moved within _inlinediff as long as worddiff is experimental (when turned off, matches is always an empty dict).

overlayworkingctx: invalidate the manifest cache when changing parents This fixes problems noticeable when rebasing several commits into one destination commit using ``--collapse``. The manifest cache needs to be cleared each time. Differential Revision: https://phab.mercurial-scm.org/D1244

overlayworkingctx: add `_compact()` Alas, presence of a key in the cache isn't sufficient evidence that the file is actually dirty. Differential Revision: https://phab.mercurial-scm.org/D1243

overlayworkingctx: add ``tomemctx()`` Differential Revision: https://phab.mercurial-scm.org/D1242

overlayworkingctx: add _auditconflicts to write() Alas, part of Mercurial's conflict detection (for file<->folder conflicts, for example) depends on the filesystem. We don't have the filesystem with IMM, so we have to run these checks ourselves. Differential Revision: https://phab.mercurial-scm.org/D1241

overlayworkingctx: inherit from committablectx instead of workingctx We should find a better name for this context at some point. (mutablememctx?) Differential Revision: https://phab.mercurial-scm.org/D1240

overlayworkingctx: add a no-op ``clearunknown()`` We no longer inherit ``workingctx``'s version, but we also don't need to do anything anymore. Differential Revision: https://phab.mercurial-scm.org/D1239

overlayworkingctx: add _manifest, files(), added(), removed(), modified() Differential Revision: https://phab.mercurial-scm.org/D1238

rebase: rerun a rebase on-disk if IMM merge conflicts arise Differential Revision: https://phab.mercurial-scm.org/D1249

rebase: add concludememorynode(), and call it when rebasing in-memory Differential Revision: https://phab.mercurial-scm.org/D1248

rebase: pass the wctx object (IMM or on-disk) to merge.update Differential Revision: https://phab.mercurial-scm.org/D1247

rebase: do not update if IMM; instead, set the overlaywctx's parents Differential Revision: https://phab.mercurial-scm.org/D1246

rebase: pass wctx to rebasenode() Differential Revision: https://phab.mercurial-scm.org/D1245

check-code: allow tabs in heredoc

hgweb: rewrite `template = A and B or C` to be a proper ternary operator

fsmonitor: fsmonitor should send wlock notifications to watchman The fsmonitor extension should send state-enter and state-leave notifications to watchman when the wlock is acquired/release, respectively. This will allow watchman and watchman subscribers to customize behavior based on whether source control operations are occurring. Test Plan: Tested checkout, update and working copy changes with extension enabled. Differential Revision: https://phab.mercurial-scm.org/D1612

fsmonitor: remove watchman transaction and working copy change notifications Remove working copy change and transaction notifications. We were relying upon callbacks on transaction function. This caused issues with lock ordering. A different approach will be adopted in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D1611

patch: catch unexpected case in _inlinediff If operation is neither 'diff.inserted' or 'diff.deleted', label and token won't be define. This patch explicitely catches that exception.

patch: reverse _inlinediff output for consistency

phases: drop the list with phase of each rev, always comput phase sets Change the C implementation of phasecache.loadphaserevs to provide only the sets for draft and secret phase as well as the number of revisions seen. Change the pure Python implementation of the same functino to compute the sets instead of the list of phases for each revision. Change phasecache.phase to check the phase sets and assume public if the revision is in neither draft nor secret set. This is computationally slightly more expensive. Change phasecache.getrevset for public() based queries to compute the set of non-matching revisions and return the result as filtered fullreposet. A shortcut is taken when no draft or secret revision exists. Bump the module version for the changed interface contract. Overall, this saves around 16 Bytes per revision whenever the phasecache is used, for the test case in issue5691 it is around 3MB. getrevset() for a large repository is around 13% slower here, that seems an acceptable trade off. Performance impact for phase() should be similar. Differential Revision: https://phab.mercurial-scm.org/D1606

transaction: build changes['revs'] as range instead of a set Revisions are added consecutively, so a range can easily represent them in the changes list. This saves around 45 Bytes / revision on 64bit platforms and reduces the memory footprint of issue5691 by 15MB. Don't copy changes['revs'] in getobsoleted. Ranges have a very efficient contains implementation already. Differential Revision: https://phab.mercurial-scm.org/D1615

ui: add diff.showfunc to tweakdefaults This is a little risky, as I think we can have some encoding weirdness crop up. showfunc also isn't the most robust feature, but it's still often useful context... Differential Revision: https://phab.mercurial-scm.org/D1610

ui: add curses interface to tweakdefaults This was part of the original proposal, and while *I* don't like the curses interface, most users anecdotally seem to greatly prefer it to plain text interfaces. Differential Revision: https://phab.mercurial-scm.org/D1609

push: restrict common discovery to the pushed set This changeset make use of the ability of the set discovery to only search common changeset for a subset of the repository. Restricting that search to the pushed set avoid potential waste of time finding out the status of many unrelated related revision. Repository with many heads were especially badly affected by this. Here is an example of findcommonhead discovery for pushing 11 outgoing changeset on a repository with tens of thousand of unrelated heads. (discovery run over a ssh link to localhost). Before: queries: 92 time: 44.1996s After: queries: 3 time: 0.6938s A x63 speedup even with a network link without latency.

setdiscover: allow to ignore part of the local graph Currently, the push discovery first determines the full set of common nodes before looking into what changesets are outgoing. When pushing a specific subset, this can lead to pathological situations where we search for the status of thousand of local heads that are unrelated to the requested pushes. To fix this, we need to teach the discovery to ignores part of the graph. Most of the necessary pieces were already in place. This changeset just makes them available to higher level API and tests them. Change actually impacting pushes are coming in a later changeset.