Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 24 Oct 2018 17:54:34 +0300] rev 40566
tests: remove lz4 as dependency while running tests
One of the previous patch authored by Augie rips out the lz4 dependency and
things should work without it.
Now there are just 2-3 tests failing because of same change in emitrevisions()
API.
Differential Revision: https://phab.mercurial-scm.org/D5192
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 24 Oct 2018 17:51:49 +0300] rev 40565
remotefilelogserver: add a matcher argument to _walkstreamfiles()
Implementing narrow stream clones in core, I added an optional matcher argument.
The function in remotefilelogserver.py does not know about that argument and
does not accept that and hence some tests fails.
Differential Revision: https://phab.mercurial-scm.org/D5191
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 05 Nov 2018 11:52:10 -0800] rev 40564
shallowutil: introduce a helper function isenabled()
This patch introduces a inenabled() function which will check whether
remotefilelog is enabled or not. The function is then also used at all the
places where check whether remotefilelog is enabled or not. The new function
makes code easy to read without need to understand what is the constant involved
and why we are checking repo.requirements.
Differential Revision: https://phab.mercurial-scm.org/D5190
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 24 Oct 2018 16:59:05 +0300] rev 40563
remotefilelog: remove some BC code related to streamclone
Since remotefilelog is now moved to core and we have
streamclone._walkstreamfiles() in core, we don't need to have the backward
compatibility code.
People with old mercurial version should use remotefilelog from hg-experimental
repo as IMO remotefilelog will go under a good refactoring and old clients will
break.
# no-check-commit foo_bar function name
Differential Revision: https://phab.mercurial-scm.org/D5189
Augie Fackler <augie@google.com> [Wed, 17 Oct 2018 09:12:29 -0400] rev 40562
remotefilelog: add some docstring
Differential Revision: https://phab.mercurial-scm.org/D5131
Augie Fackler <augie@google.com> [Thu, 04 Oct 2018 10:03:10 -0400] rev 40561
remotefilelog: rename wireproto methods and remaining capabilities
This is a mess, in part because there should be more constants
throughout. I know we typically do exp- instead of the x_ business in
this change, but I also had to use this in some function names, so I
figured until I can break that coupling I'd go with this. If it's too
unpleasant during review, let me know and I can probably clean it up
some more.
# no-check-commit due to new foo_bar naming - too hard to avoid right now :(
Differential Revision: https://phab.mercurial-scm.org/D5129
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 17:02:48 -0400] rev 40560
remotefilelog: consolidate and rename on-disk store requirement
The value of this constant appeared in too many places. While we're
here, rename it to be more consistent with our naming conventions for
experimental functionality.
Differential Revision: https://phab.mercurial-scm.org/D5128
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 17:30:47 -0400] rev 40559
remotefilelog: consolidate and rename bundle2 capability
Bonus: we catch a spot where the shallowrepo requirement was being
misused as the bundle2 capability.
Differential Revision: https://phab.mercurial-scm.org/D5127
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 16:38:40 -0400] rev 40558
remotefilelog: rename capability for legacy ssh file fetching method
I think I want to delete this, but for now we'll just rename this so
it's easy to tell apart from other strings that have the same value.
Differential Revision: https://phab.mercurial-scm.org/D5126
Augie Fackler <augie@google.com> [Thu, 04 Oct 2018 00:11:37 -0400] rev 40557
remotefilelog: rip out lz4 support
Some methods are left teased out so it's easier to build a migration
extension from v1 packs to v2. I also anticipate those methods will be
of use if we get around to adding configurable compression engines
before we end up jettisoning packs.
I think it would make sense to register one-shot `compress` and
`decompress` methods on our compression engines in util.py, but
indygreg mentioned that might not be straightforward.
Differential Revision: https://phab.mercurial-scm.org/D4944
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 23:25:31 -0400] rev 40556
remotefilelog: bump pack file formats to version 2
Drop support for format 0 and 1 packs. Version 2 will be different in
that it'll use zlib compression instead of lz4, as the latter isn't
really suitable for core.
If I can make it work, I'll make compression engines pluggable, but
the logic for handling these files is spread out enough it's a little
tricky to get that right.
Test changes are only due to pack files being renamed. The contents of
the packs are the same.
Differential Revision: https://phab.mercurial-scm.org/D4943
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 23:47:43 -0400] rev 40555
remotefilelog: avoid hard-coding pack names in tests
I'm about to alter the pack format, which alters the content-addressed
name of the pack file. In order to reduce my own confusion I found it
helpful to use an ls | head pipeline since we always want to inspect
the newest pack.
Differential Revision: https://phab.mercurial-scm.org/D4942
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 14:04:41 -0400] rev 40554
remotefilelog: remove now-unused wirepack code
Differential Revision: https://phab.mercurial-scm.org/D5124
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 14:04:11 -0400] rev 40553
remotefilelogserver: remove pack-serving functionality
Differential Revision: https://phab.mercurial-scm.org/D5123
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 13:17:16 -0400] rev 40552
remotefilelog: remove support for fetching packs
We've never used this at Google, and it's strictly an optimization
that indygreg and I are confident we can handle more elegantly
later. Let's rip it out to keep the surface area of the imported
remotefilelog small.
Some tests have changes because the tests for regular packs and
repacking were mixed with tests of fetching packs. I'm pretty sure
I've done the right thing here.
Differential Revision: https://phab.mercurial-scm.org/D5122
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 17:20:44 -0400] rev 40551
remotefilelog: jettison example cache client
We can always reconstruct it from history if it becomes interesting.
Differential Revision: https://phab.mercurial-scm.org/D4941
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 17:18:20 -0400] rev 40550
remotefilelog: prune obsolete method
Differential Revision: https://phab.mercurial-scm.org/D4940
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 14:01:22 -0400] rev 40549
logtoprocess: use new runbgcommand from procutil
Differential Revision: https://phab.mercurial-scm.org/D4939
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 14:42:57 -0400] rev 40548
procutil: port over windows encoding fixes from logtoprocess
Differential Revision: https://phab.mercurial-scm.org/D5121
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 14:01:04 -0400] rev 40547
remotefilelog: transplant runbgcommand to procutil
While cleaning up the deprecated runshellcommand I noticed a
near-clone of this in logtoprocess, so I'm standardizing on what
appears to be the newer one by moving it to procutil.
Differential Revision: https://phab.mercurial-scm.org/D4938
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 13:54:45 -0400] rev 40546
remotefilelog: remove function that was described as deprecated
Differential Revision: https://phab.mercurial-scm.org/D4937
Augie Fackler <augie@google.com> [Thu, 27 Sep 2018 13:03:19 -0400] rev 40545
remotefilelog: import pruned-down remotefilelog extension from hg-experimental
This is remotefilelog as of my recent patches for compatibility with
current tip of hg, minus support for old versions of Mercurial and
some FB-specific features like their treemanifest extension and
fetching linkrev data from a patched phabricator. The file extutil.py
moved from hgext3rd to remotefilelog.
This is not yet ready to be landed, consider it a preview for
now. Planned changes include:
* replace lz4 with zstd
* rename some capabilities, requirements and wireproto commands to mark
them as experimental
* consolidate bits of shallowutil with related functions (eg readfile)
I'm certainly open to other (small) changes, but my rough mission is
to land this largely as-is so we can use it as a model of the
functionality we need going forward for lazy-fetching of file contents
from a server.
# no-check-commit because of a few foo_bar functions
Differential Revision: https://phab.mercurial-scm.org/D4782
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 17 Oct 2018 17:42:32 +0300] rev 40544
store: introduce _matchtrackedpath() and use it to filter store files
This patch introduces a function to filter store files on the basis of the path
which they are tracking.
The function assumes that the entries can be of two types, 'meta/*' and 'data/*'
which means it will just work on revlog based storage and not with another
storage ways.
For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the
beginning and the end then pass that to matcher.
For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from
beginning and end then call matcher.visitdir() with it to make sure all the
parent directories are also downloaded.
Since the storage filtering for narrow stream clones is implemented with this
patch, we remove the un-implemented error message, add some more tests and add
the treemanifest case to tests too.
The tests demonstrate that it works correctly.
After this patch, we have now narrow stream clones working. Narrow stream clones
are a very important feature for large repositories who have good internet
connection because they use streamclones for cloning and if they do normal
narrow clone, that takes more time then a full streamclone. Also narrow-stream
clone will drastically speed up clone timings.
Differential Revision: https://phab.mercurial-scm.org/D5139
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 17 Oct 2018 17:24:55 +0300] rev 40543
narrow: only send includepats and excludepats if they are not empty
If we send an empty includepats or excludepats argument to getbundle, it's
translated to `['']` on the server which causes problems because even though
it's empty, bool of that value if True and we end up creating differencematcher
with narrowspec.match() which results in unexpected behavior.
Differential Revision: https://phab.mercurial-scm.org/D5138
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 16 Oct 2018 17:53:26 +0300] rev 40542
exchange: pass includepats and excludepats as arguments to getbundle()
This will help in implementing narrow stream clones. Also narrow extension used
to add these arguments, now we add them by default if they are not empty.
Since reading includepats and excludepats on the server only works when narrow
is enabled, we check if narrow if enabled or not before passing them.
Differential Revision: https://phab.mercurial-scm.org/D5119
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:51:21 -0700] rev 40541
catapult: add a bit more documentation on how to use catapult tracing
Differential Revision: https://phab.mercurial-scm.org/D5217
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:44:02 -0700] rev 40540
catapult: introduce HGTESTCATAPULTSERVERPIPE to control run-tests' tracing
If unset, it will inherit its value from HGCATAPULTSERVERPIPE (which hg itself
also respects). By setting only HGTESTCATAPULTSERVERPIPE, we can get per-command
breakdowns of the test runtimes for the whole test suite without overloading the
trace file with the contents of the tracing from hg (such as demandimport, etc.)
Differential Revision: https://phab.mercurial-scm.org/D5216
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:44:37 -0700] rev 40539
catapult: rename 'active' to 'activetrace'; this isn't storing a boolean state
(This is an attempt to make it more obvious that the indentation is correct, to
prevent a repeat of 97f52862b1bd)
Differential Revision: https://phab.mercurial-scm.org/D5215
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:44:32 -0700] rev 40538
catapult: fix broken run-tests catapult tracing
Commit 97f52862b1bd thought this block was misindented, but it was not; the
indentation after that commit means we never log a catapult trace for individual
commands. Also, we've never emitted the "end" trace line when the test was
finished, so the trace viewer claims that these "did not finish", fixing that as
well.
Differential Revision: https://phab.mercurial-scm.org/D5214
Sangeet Kumar Mishra <mail2sangeetmishra@gmail.com> [Sat, 03 Nov 2018 23:24:15 +0530] rev 40537
test: fix self._testdir to use the right mercurial library during testing
Currently if you run tests from some other directory other than the
`../tests/`, you will get a warning stating
```
warning: Testing with unexpected mercurial lib: mercurial
(expected /tmp/hgtests.xxxxxx/install/lib/python/mercurial)
```
This is because the current directory being added to the 'PATH', if the
`self._testdir != runtestdir`, owing to this line
```
if self._testdir != runtestdir:
path = [self._testdir] + path
```
Also say you ran the tests from the hg base directory,
because directory is being added in the PATH (see the above snippet, at
that stage the `self._testdir` has the value as `cwd`, owing to a faulty
initialization). And since the current directory already has the 'hg',
that is used in place of the hg that is installed for the testing purposes
in `/tmp/hgtests.xxxxxx/...`.
Differential Revision: https://phab.mercurial-scm.org/D5199
Matt Harbison <matt_harbison@yahoo.com> [Sat, 03 Nov 2018 20:50:04 -0400] rev 40536
tests: conditonalize a difference in test-merge-tools.t on Windows
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 17:12:22 -0400] rev 40535
statprof: clean up unicode/bytes a little
I'm not really sure how this worked before, but something perturbed it
and what I've got in this change I believe is a little tidier. This
fixes test-profile.t on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5210