ui: factor out function that writes data to fout/ferr with labeling I'm thinking of adding an option to send status messages to stderr (or a dedicated command-server channel) so that structured output (e.g. JSON) would never be interleaved with non-formatter output. A unified write() interface helps to do that.

ui: consolidate places where _progclear() is called The progress bar has to be cleared when we start writing some data to the output stream. Let's make it always triggered immediately before switching by _colormode, so that we can easily factor out helper functions.

templater: compute revset lazily This speeds up e.g. "{ifcontains(rev, revset('::.'), ...)}" in common cases where 'rev' is near the working parent. The templater API is ugly, but it helps here. 'f' can be either a generator or a function returning a generator.

remotefilelog: drop compat code for "getbundle_shallow" wireprotocol command Doing some annotate on hgexperimental shows that getbundle_shallow used to exist in 2013 or before. We don't have any pre-2013 remotefilelog users except Fb themselves and I doubt they are going to use in-core remotefilelog. So it's safe to remove this. Differential Revision: https://phab.mercurial-scm.org/D5193

tests: remove lz4 as dependency while running tests One of the previous patch authored by Augie rips out the lz4 dependency and things should work without it. Now there are just 2-3 tests failing because of same change in emitrevisions() API. Differential Revision: https://phab.mercurial-scm.org/D5192

remotefilelogserver: add a matcher argument to _walkstreamfiles() Implementing narrow stream clones in core, I added an optional matcher argument. The function in remotefilelogserver.py does not know about that argument and does not accept that and hence some tests fails. Differential Revision: https://phab.mercurial-scm.org/D5191

shallowutil: introduce a helper function isenabled() This patch introduces a inenabled() function which will check whether remotefilelog is enabled or not. The function is then also used at all the places where check whether remotefilelog is enabled or not. The new function makes code easy to read without need to understand what is the constant involved and why we are checking repo.requirements. Differential Revision: https://phab.mercurial-scm.org/D5190

remotefilelog: remove some BC code related to streamclone Since remotefilelog is now moved to core and we have streamclone._walkstreamfiles() in core, we don't need to have the backward compatibility code. People with old mercurial version should use remotefilelog from hg-experimental repo as IMO remotefilelog will go under a good refactoring and old clients will break. # no-check-commit foo_bar function name Differential Revision: https://phab.mercurial-scm.org/D5189

remotefilelog: add some docstring Differential Revision: https://phab.mercurial-scm.org/D5131

remotefilelog: rename wireproto methods and remaining capabilities This is a mess, in part because there should be more constants throughout. I know we typically do exp- instead of the x_ business in this change, but I also had to use this in some function names, so I figured until I can break that coupling I'd go with this. If it's too unpleasant during review, let me know and I can probably clean it up some more. # no-check-commit due to new foo_bar naming - too hard to avoid right now :( Differential Revision: https://phab.mercurial-scm.org/D5129

remotefilelog: consolidate and rename on-disk store requirement The value of this constant appeared in too many places. While we're here, rename it to be more consistent with our naming conventions for experimental functionality. Differential Revision: https://phab.mercurial-scm.org/D5128

remotefilelog: consolidate and rename bundle2 capability Bonus: we catch a spot where the shallowrepo requirement was being misused as the bundle2 capability. Differential Revision: https://phab.mercurial-scm.org/D5127

remotefilelog: rename capability for legacy ssh file fetching method I think I want to delete this, but for now we'll just rename this so it's easy to tell apart from other strings that have the same value. Differential Revision: https://phab.mercurial-scm.org/D5126

remotefilelog: rip out lz4 support Some methods are left teased out so it's easier to build a migration extension from v1 packs to v2. I also anticipate those methods will be of use if we get around to adding configurable compression engines before we end up jettisoning packs. I think it would make sense to register one-shot `compress` and `decompress` methods on our compression engines in util.py, but indygreg mentioned that might not be straightforward. Differential Revision: https://phab.mercurial-scm.org/D4944

remotefilelog: bump pack file formats to version 2 Drop support for format 0 and 1 packs. Version 2 will be different in that it'll use zlib compression instead of lz4, as the latter isn't really suitable for core. If I can make it work, I'll make compression engines pluggable, but the logic for handling these files is spread out enough it's a little tricky to get that right. Test changes are only due to pack files being renamed. The contents of the packs are the same. Differential Revision: https://phab.mercurial-scm.org/D4943

remotefilelog: avoid hard-coding pack names in tests I'm about to alter the pack format, which alters the content-addressed name of the pack file. In order to reduce my own confusion I found it helpful to use an ls | head pipeline since we always want to inspect the newest pack. Differential Revision: https://phab.mercurial-scm.org/D4942

remotefilelog: remove now-unused wirepack code Differential Revision: https://phab.mercurial-scm.org/D5124

remotefilelogserver: remove pack-serving functionality Differential Revision: https://phab.mercurial-scm.org/D5123

remotefilelog: remove support for fetching packs We've never used this at Google, and it's strictly an optimization that indygreg and I are confident we can handle more elegantly later. Let's rip it out to keep the surface area of the imported remotefilelog small. Some tests have changes because the tests for regular packs and repacking were mixed with tests of fetching packs. I'm pretty sure I've done the right thing here. Differential Revision: https://phab.mercurial-scm.org/D5122

remotefilelog: jettison example cache client We can always reconstruct it from history if it becomes interesting. Differential Revision: https://phab.mercurial-scm.org/D4941

remotefilelog: prune obsolete method Differential Revision: https://phab.mercurial-scm.org/D4940

logtoprocess: use new runbgcommand from procutil Differential Revision: https://phab.mercurial-scm.org/D4939

procutil: port over windows encoding fixes from logtoprocess Differential Revision: https://phab.mercurial-scm.org/D5121

remotefilelog: transplant runbgcommand to procutil While cleaning up the deprecated runshellcommand I noticed a near-clone of this in logtoprocess, so I'm standardizing on what appears to be the newer one by moving it to procutil. Differential Revision: https://phab.mercurial-scm.org/D4938

remotefilelog: remove function that was described as deprecated Differential Revision: https://phab.mercurial-scm.org/D4937

remotefilelog: import pruned-down remotefilelog extension from hg-experimental This is remotefilelog as of my recent patches for compatibility with current tip of hg, minus support for old versions of Mercurial and some FB-specific features like their treemanifest extension and fetching linkrev data from a patched phabricator. The file extutil.py moved from hgext3rd to remotefilelog. This is not yet ready to be landed, consider it a preview for now. Planned changes include: * replace lz4 with zstd * rename some capabilities, requirements and wireproto commands to mark them as experimental * consolidate bits of shallowutil with related functions (eg readfile) I'm certainly open to other (small) changes, but my rough mission is to land this largely as-is so we can use it as a model of the functionality we need going forward for lazy-fetching of file contents from a server. # no-check-commit because of a few foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D4782

store: introduce _matchtrackedpath() and use it to filter store files This patch introduces a function to filter store files on the basis of the path which they are tracking. The function assumes that the entries can be of two types, 'meta/*' and 'data/*' which means it will just work on revlog based storage and not with another storage ways. For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the beginning and the end then pass that to matcher. For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from beginning and end then call matcher.visitdir() with it to make sure all the parent directories are also downloaded. Since the storage filtering for narrow stream clones is implemented with this patch, we remove the un-implemented error message, add some more tests and add the treemanifest case to tests too. The tests demonstrate that it works correctly. After this patch, we have now narrow stream clones working. Narrow stream clones are a very important feature for large repositories who have good internet connection because they use streamclones for cloning and if they do normal narrow clone, that takes more time then a full streamclone. Also narrow-stream clone will drastically speed up clone timings. Differential Revision: https://phab.mercurial-scm.org/D5139

narrow: only send includepats and excludepats if they are not empty If we send an empty includepats or excludepats argument to getbundle, it's translated to `['']` on the server which causes problems because even though it's empty, bool of that value if True and we end up creating differencematcher with narrowspec.match() which results in unexpected behavior. Differential Revision: https://phab.mercurial-scm.org/D5138

exchange: pass includepats and excludepats as arguments to getbundle() This will help in implementing narrow stream clones. Also narrow extension used to add these arguments, now we add them by default if they are not empty. Since reading includepats and excludepats on the server only works when narrow is enabled, we check if narrow if enabled or not before passing them. Differential Revision: https://phab.mercurial-scm.org/D5119

catapult: add a bit more documentation on how to use catapult tracing Differential Revision: https://phab.mercurial-scm.org/D5217