simplestore: correctly implement flag processors There were a couple of bugs around the implementation of flags processing with the simple store. After these changes, test-flagprocessor.t now passes! test-flagprocessor.t was also updated to include explicit test coverage that pushed data is as expected on the server. The test extension used by test-flagprocessor.t has been updated so it monkeypatches the object returned from repo.file() instead of monkeypatching filelog.filelog. This allows it to work with extensions that return custom types from repo.file(). The monkeypatching is rather hacky and probably is performance prohibitive for real repos. We should probably come up with a better mechanism for registering flag processors so monkeypatching isn't needed. Differential Revision: https://phab.mercurial-scm.org/D3116

tests: `hg init` after resetting HGRCPATH Otherwise extensions loaded via --extra-config-opt could prevent access to the repo by introducing requirements file. This does mean that custom extensions loaded in this way won't impact this test. I'm fine with that. Differential Revision: https://phab.mercurial-scm.org/D3115

tests: work around potential repo incompatibility test-run-tests.t invokes run-tests.py. But custom extensions providing new repo requirements may be in play and may not get inherited by the new run-tests.py. We ensure our repo is created with a vanilla config to mitigate extension-caused badness. Differential Revision: https://phab.mercurial-scm.org/D3114

tests: disable test-keyword.t with simple store The keyword extension is hooking into repo.file() and defining its own filelog class. It will likely require a more formal storage interface before keywords are usable with alternate storage backends. Differential Revision: https://phab.mercurial-scm.org/D3113

tests: conditionalize test-treemanifest.t Parts of the test were assuming the use of revlogs with fnstore path encoding. Other parts of the test assumed we could create repos with different store encodings and that stream clone bundles worked. Make all of this conditional on running a revlog repo. Differential Revision: https://phab.mercurial-scm.org/D3112

tests: use unbundle in test-symlink-os-yes-fs-no.py The test (which should probably be rewritten as a .t test - the test was initially authored in 2009 and this may have predated some test harness features allowing us to implement it as a .t test) is verifying symlink behavior with regards to working directory operations. How it pulls bundle data into a repo is not relevant. So we can switch from pull to unbundle so we can support environments where bundlerepos don't work. Differential Revision: https://phab.mercurial-scm.org/D3111

tests: disable `hg clone --stream` test with simple store We mass disabled stream clone tests in a previous commit. Looks like one was missed. Differential Revision: https://phab.mercurial-scm.org/D3110

tests: use `hg unbundle` in test-setdiscovery.t This is testing how discovery between 2 well-defined repos works, not that `hg pull` works with bundles. Switch to `hg unbundle` so it works in environments where bundlerepos aren't supported. Differential Revision: https://phab.mercurial-scm.org/D3109

tests: require revlog store for test-verify.t This tests is doing tons of revlog-y things. It should be possible to make verification work across multiple stores. But it will likely have to wait until we have better abstractions in place. Differential Revision: https://phab.mercurial-scm.org/D3108

tests: conditionalize test-commandserver.t based on extra extensions If running with extra extensions we get an output difference. Differential Revision: https://phab.mercurial-scm.org/D3107

tests: conditionalize test-bundle.t This test is massive and could probably be split up. This change essentially requires the revlog store for stream clone tests and support for bundlerepos for various tests operating on bundle files. Differential Revision: https://phab.mercurial-scm.org/D3106

tests: require revlog store with test-repair-strip.t This test is doing a number of low-level things with revlogs. Mark it as requiring the revlog store. Differential Revision: https://phab.mercurial-scm.org/D3105

simplestore: back up index when adding a revision This ensures that transaction rollback reverts the index to its previous state. This fixed a few test failures due to `hg verify` complaining about a reference to an undefined changeset revision. Differential Revision: https://phab.mercurial-scm.org/D3104

tests: disable shallow narrow tests with simple store I think these are failing because of issues with flags processing in the simple store. Let's revisit this later. Differential Revision: https://phab.mercurial-scm.org/D3103

tests: skip test-hgweb-bundle.t if we don't support bundlerepos Differential Revision: https://phab.mercurial-scm.org/D3102

tests: disable test-audit-path.t with simple store The simple store is using a primitive filename encoding scheme. Many tests in this file fail. So let's disable it for now. At some point, we should support a mode where we can run tests without fnstore or dotencode and have all tests assuming use of these filename encoding mechanisms be conditional on their usage. Differential Revision: https://phab.mercurial-scm.org/D3101

tests: port test-convert-filemap.t to simple store Differential Revision: https://phab.mercurial-scm.org/D3100

tests: disable test-static-http.t with simple store It /should/ be possible to support repos statically hosted on HTTP servers with alternate stores. But it's more trouble than it is worth right now. Let's just disable the test. Differential Revision: https://phab.mercurial-scm.org/D3099

tests: don't drop global hgrc Various tests are failing when the simple store extension is loaded because the test overrides HGRCPATH and preempts loading of extensions that were injected via --extra-config-opt. In most cases, it is acceptable to always load the global HGRCPATH. So this commit changes a couple of tests so the global HGRCPATH is still pulled in. Differential Revision: https://phab.mercurial-scm.org/D3098

verify: allow suppressing warnings about extra files The verifier issues warnings when the set of files in .hg/store doesn't align with the set of files that are advertised via repo.file(f).files() for all files seen in ctx.files() changelog traversal. This logic is reasonable for a default implementation. But some stores may have extra files whose presence is harmless. Or those stores may not have the same transaction rollback semantics that unlink files as other stores. This commit adds support for disabling the warning for orphaned files. The simple store extension has been taught to set this flag. Differential Revision: https://phab.mercurial-scm.org/D3097

tests: conditionalize tests for various repo features Working my down the long tail of test failures due to the simple store. We're now down to 38 failures with the simple store. Differential Revision: https://phab.mercurial-scm.org/D3096

simplestore: use a custom store for the simple store repo Before, we used the default store, which was based on fncache and dotencode. After attempting to port tests to work with the simple store, I realized that fncache was more trouble than it is worth. This commit implements a proper store type for the simple repo - one that isn't based off fncache. This causes a number of new test failures because of tests expecting the full fncache store filename encoding. I may extend the store format in a subsequent commit to take the filename encoding parts of fncache that we can take (basically everything except hash encoding, since that isn't reversible). But for now, let's use encoded store. As part of this, we implement proper requirements support for repos created with the simple store. This should have been done from the beginning, as a requirement is needed to lock out clients that don't understand a storage format. A new hghave feature advertising the presence of fncache in repos has been added. Most tests touching the fncache are now conditional on that feature. Other tests have added the optional repo requirement to output. Differential Revision: https://phab.mercurial-scm.org/D3095

wireproto: allow direct stream processing for unbundle Introduce a new option server.streamunbundle which starts a transaction immediately to apply a bundle instead of writing it to a temporary file first. This side steps the need for a large tmp directory at the cost of preventing concurrent pushes. This is a reasonable trade-off for many setups as concurrent pushes for the main branch at least are disallowed anyway. The option defaults to off to preserve existing behavior. Change the wireproto interface to provide a generator for reading the payload and make callers responsible for consuming all data. Differential Revision: https://phab.mercurial-scm.org/D2470

wireproto: send server capabilities in canonical order Differential Revision: https://phab.mercurial-scm.org/D3171

wireproto: don't special case bundlecaps, but sort all scsv arguments Differential Revision: https://phab.mercurial-scm.org/D3170

wireproto: turn client capabilities into sets, sorted on the wire Differential Revision: https://phab.mercurial-scm.org/D3169

verify: drop "revlog" from warning message Not all stores may be backed by revlogs. Switch to a more generic error message. Differential Revision: https://phab.mercurial-scm.org/D3094

store: make file filtering during walk configurable Previously, the walking mechanism assumed the use of revlogs for storage. Making the file filter configurable will enable custom stores to override _walk() so it recognizes additional files. Differential Revision: https://phab.mercurial-scm.org/D3093

simplestore: shore up lookup errors When revisions or nodes can't be resolved, we're expected to raise an error.LookupError. When I ported code from revlog.py, I failed to realize that "LookupError" in that module is aliased to error.LookupError. I thought we were using the builtin LookupError instead. This commit switches us to error.LookupError. It also fixes rev() to raise error.LookupError instead of KeyError. Differential Revision: https://phab.mercurial-scm.org/D3092

tests: extract dumprevlog tests to own file And mark the test as requiring the revlog store. Differential Revision: https://phab.mercurial-scm.org/D3091