Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:10:59 -0700] rev 37634
treediscovery: switch to command executor interface
We now have a new interface for requesting that commands run.
Switch to it.
Differential Revision: https://phab.mercurial-scm.org/D3272
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 16:18:26 -0700] rev 37633
wireproto: remove iterbatch() from peer interface (API)
Good riddance.
Some tests have been ported to the new API. This probably should
have been done in earlier commits. But duplicating the test coverage
would have been difficult. It was easier this way.
.. api::
The wire protocol peer's ``iterbatch()`` for bulk executing commands
has been remove.d Use ``peer.commandexecutor()`` instead.
Differential Revision: https://phab.mercurial-scm.org/D3271
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:08:46 -0700] rev 37632
largefiles: use command executor for batch operation
This is the only other user of iterbatch() in core.
Tests changed because the new command executor is smart enough
to not send a "batch" command over the wire if only 1 command
was requested.
There is still coverage for the "batch" command in this test
though.
Differential Revision: https://phab.mercurial-scm.org/D3270
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 11:02:34 -0700] rev 37631
wireproto: implement batching on peer executor interface
This is a bit more complicated than non-batch requests because we
need to buffer sends until the last request arrives *and* we need
to support resolving futures as data arrives from the remote.
In a classical concurrent.futures executor model, the future
"starts" as soon as it is submitted. However, we have nothing to
start until the last command is submitted.
If we did nothing, calling result() would deadlock, since the future
hasn't "started." So in the case where we queue the command, we return
a special future type whose result() will trigger sendcommands().
This eliminates the deadlock potential. It also serves as a check
against callers who may be calling result() prematurely, as it will
prevent any subsequent callcommands() from working. This behavior
is slightly annoying and a bit restrictive. But it's the world
that half duplex connections forces on us.
In order to support streaming responses, we were previously using
a generator. But with a futures-based API, we're using futures
and not generators. So in order to get streaming, we need a
background thread to read data from the server.
The approach taken in this patch is to leverage the ThreadPoolExecutor
from concurrent.futures for managing a background thread. We create
an executor and future that resolves when all response data is
processed (or an error occurs). When exiting the context manager,
we wait on that background reading before returning.
I was hoping we could manually spin up a threading.Thread and this
would be simple. But I ran into a few deadlocks when implementing.
After looking at the source code to concurrent.futures, I figured
it would just be easier to use a ThreadPoolExecutor than implement
all the code needed to manually manage a thread.
To prove this works, a use of the batch API in discovery has been
updated.
Differential Revision: https://phab.mercurial-scm.org/D3269
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 10:51:23 -0700] rev 37630
wireproto: implement command executor interface for version 1 peers
Now that we've defined our new interface for issuing commands,
let's implement it.
We add the interface to the base peer interface. This means all
peer types must implement it.
The only peer types that we have are the local peer in localrepo
and a shared wire peer for version 1 of the wire protocol.
The local peer implementation is pretty straightforward. We
don't do anything fancy and just return a resolved future with
the result of a method call. This is similar to what
localiterbatcher does.
The wire protocol version 1 implementation is a bit more complicated
and is a more robust implementation.
The wire executor queues commands by default. And because the new
executor interface always allows multiple commands but not all version
1 commands are @batchable, it has to check that the requested commands
are batchable if multiple commands are being requested.
The wire executor currently only supports executing a single command.
This is for simplicity reasons. Support for multiple commands will
be added in a separate commit.
To prove the new interface works, a call to the "known" command
during discovery has been updated to use the new API.
It's worth noting that both implementations require a method having
the command name to exist on the peer. There is at least one caller
in core that don't have a method calls peer._call() directly. We
may need to shore up the requirements later...
Differential Revision: https://phab.mercurial-scm.org/D3268
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Apr 2018 10:23:05 -0700] rev 37629
repository: define new interface for running commands
Today, the peer interface exposes methods for each command that can
be executed. In addition, there is an iterbatch() API that allows
commands to be issued in batches and provides an iterator over the
results. This is a glorified wrapper around the "batch" wire command.
Wire protocol version 2 supports nicer things (such as batching
any command and out-of-order replies). It will require a more
flexible API for executing commands.
This commit introduces a new peer interface for making command
requests. In the new world, you can't simply call a method on the
peer to execute a command: you need to obtain an object to be used
for executing commands. That object can be used to issue a single
command or it can batch multiple requests. In the case of full duplex
peers, the command may even be sent out over the wire immediately.
There are no per-command methods. Instead, there is a generic
method to call a command. The implementation can then perform domain
specific processing for specific commands. This includes passing
data via a specially named argument.
Arguments are also passed as a dictionary instead of using **kwargs.
While **kwargs is nicer to use, we've historically gotten into
trouble using it because there will inevitably be a conflict between
the name of an argument to a wire protocol command and an argument
we want to pass into a function.
Instead of a command returning a value, it returns a future which
will resolve to a value. This opens the door for out-of-order
response handling and concurrent response handling in the version
2 protocol.
Differential Revision: https://phab.mercurial-scm.org/D3267
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:28:57 -0700] rev 37628
pycompat: export a handle on concurrent.futures
On Python 3, we use the built-in version in the standard library. Else
we use our vendored backport.
Differential Revision: https://phab.mercurial-scm.org/D3266
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:27:52 -0700] rev 37627
setup: add packages for concurrent.futures
We conceivably don't need to distribute this package on Python 3
since we will use the version in the standard library. However,
we want installs to be usable of multiple versions of Python. So
it is best to always have it.
Differential Revision: https://phab.mercurial-scm.org/D3265
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:23:48 -0700] rev 37626
futures: switch to absolute and relative imports
This makes the package conform with our importing policy,
silencing a number of warnings. It also makes the package usable
when it isn't named "concurrent.futures."
Differential Revision: https://phab.mercurial-scm.org/D3264
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:22:31 -0700] rev 37625
tests: silence pyflakes for thirdparty/concurrent
It is complaining about unused imports.
Differential Revision: https://phab.mercurial-scm.org/D3263
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 12:19:37 -0700] rev 37624
futures: get rid of extend_path
This is used so mutliple directories can provide a package. We don't
need it when vendoring.
Differential Revision: https://phab.mercurial-scm.org/D3262
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 14:48:24 -0700] rev 37623
thirdparty: vendor futures 3.2.0
Python 3 has a concurrent.futures package in the standard library
for representing futures. The "futures" package on PyPI is a backport
of this package to work with Python 2.
The wire protocol code today has its own future concept for handling
of "batch" requests. The frame-based protocol will also want to
use futures.
I've heavily used the "futures" package on Python 2 in other projects
and it is pretty nice. It even has a built-in thread and process pool
for running functions in parallel. I've used this heavily for concurrent
I/O and other GIL-less activities.
The existing futures API in the wire protocol code is not as nice as
concurrent.futures. Since concurrent.futures is in the Python standard
library and will presumably be the long-term future for futures in our
code base, let's vendor the backport so we can use proper futures today.
# no-check-commit because of style violations
Differential Revision: https://phab.mercurial-scm.org/D3261
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 15:05:49 +0530] rev 37622
py3: make sure decode() first argument is str
Uses pycompat.sysstr() to make sure we uses bytes on Python 2 and unicodes on
Python 3.
Differential Revision: https://phab.mercurial-scm.org/D3279
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 23:14:38 -0700] rev 37621
patch: make extract() a context manager (API)
Previously, this function was creating a temporary file and relying
on callers to unlink it. Yuck.
We convert the function to a context manager and tie the lifetime of
the temporary file to that of the context manager. This changed
indentation not only from the context manager, but also from the
elination of try blocks. It was just easier to split the heart of
extract() into its own function.
The single consumer of this function has been refactored to use it as
a context manager. Code for cleaning up the file in tryimportone()
has also been removed.
.. api::
``patch.extract()`` is now a context manager. Callers no longer have
to worry about deleting the temporary file it creates, as the file is
tied to the lifetime of the context manager.
Differential Revision: https://phab.mercurial-scm.org/D3306
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 23:06:27 -0700] rev 37620
cmdutil: pass in parsed patch to tryimportone() (API)
Previously, we parsed the patch in tryimportone(). This assumes the
input is in a patch format that needs to be parsed. We want to support
feeding in data from other formats. So let's let the caller handle the
parsing.
One wonky thing about patch parsing is that patch.extract() creates
a temp file to hold the diffs and it is up to tryimportone() to
unlink that temp file. I'll improve this in a subsequent commit.
Differential Revision: https://phab.mercurial-scm.org/D3305
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 12 Apr 2018 20:42:42 -0700] rev 37619
stringutil: support more types with pprint()
bytearray wasn't working. Integers and floats were not being
formatted.
I /think/ %f is portable across both Python 2 and 3, as it should
default to 6 decimal points on each.
Differential Revision: https://phab.mercurial-scm.org/D3302
Augie Fackler <augie@google.com> [Thu, 12 Apr 2018 14:27:13 -0400] rev 37618
fix: port most of the way to python 3
Only most of the way because we now have to decide: if we want to keep
the current .format() interface for the config in hgrc, we have to use
unicodes to do formatting in Python 3, rather than bytes. I'm
basically fine with that, so a follow-up patch will do so.
Differential Revision: https://phab.mercurial-scm.org/D3300
Matt Harbison <matt_harbison@yahoo.com> [Thu, 12 Apr 2018 17:24:55 -0700] rev 37617
lfs: teach the blob server to handle --prefix
Matt Harbison <matt_harbison@yahoo.com> [Thu, 05 Apr 2018 15:42:40 -0400] rev 37616
hgweb: fallback to checking wsgireq.env for REPO_NAME for 3rd party hosting
Starting with d7fd203e36cc, SCM Manager began to 404 any repository access.
What's happening is that it is generating a python script that creates an hgweb
application (not hgwebdir), and launches hgweb via wsgicgi. It must be setting
REPO_NAME in the process environment before launching this script, which gets
picked up and put into wsgireq.env when wsgicgi launches the hgweb application.
>From there, other variables (notably 'apppath' and 'dispatchpath') are
constructed differently.
d7fd203e36cc^ (working):
apppath: /hg/eng/devsetup
dispatchpath:
pathinfo: /eng/devsetup
reponame: eng/devsetup
d7fd203e36cc:
apppath: /hg
dispatchpath: eng/devsetup
pathinfo: /eng/devsetup
reponame: None
REPO_NAME: eng/devsetup
Rather than having an existing installation break when Mercurial is upgraded,
just resume checking the environment. I have no idea how many other hosting
solutions would break without restoring this.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 12:51:09 -0700] rev 37615
peer: scatter module to the wind (API)
peer.py hardly contained any code. The code it did contain was
generic to the version 1 peer interface or specific to the
local repository peer.
So code has been moved to wireprotov1peer and localrepo, as
appropriate.
Differential Revision: https://phab.mercurial-scm.org/D3260
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 12:49:08 -0700] rev 37614
wireproto: move version 1 peer functionality to standalone module (API)
wireproto.py contains code for both the client and the server. There
*should* be a somewhat strong separation between the two.
This commit extracts the client-side code from wireproto.py into a new
module - wireprotov1peer.
Differential Revision: https://phab.mercurial-scm.org/D3259
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 10:51:38 -0700] rev 37613
wireproto: move gboptsmap to wireprototypes and rename (API)
This is also shared between client and server and will need to
exist in a shared module when that code is split into different
modules.
Differential Revision: https://phab.mercurial-scm.org/D3258
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 10:50:58 -0700] rev 37612
wireproto: move value encoding functions to wireprototypes (API)
These functions should live in the same place. I plan to separate
client from server code in upcoming commits. wireprototypes is
where we are putting shared code like this.
Differential Revision: https://phab.mercurial-scm.org/D3257
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 19:09:35 -0700] rev 37611
httppeer: basic implementation of capabilities interface
This is a bit crude. The capabilities mechanism for version 2 of
the wire protocol is a bit different from version 1. And code
in core is relying on strings passed to capable() matching strings
advertised by the "capabilities" wire protocol command. I may
refactor the internal checking mechanism to be a bit more
abstract or based on interfaces. Time will tell...
Differential Revision: https://phab.mercurial-scm.org/D3256
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 18:47:09 -0700] rev 37610
repository: split capabilities methods into separate interface
So we can implement them without having to implement support for
every command.
Differential Revision: https://phab.mercurial-scm.org/D3255
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 11 Apr 2018 11:03:45 -0700] rev 37609
httppeer: implement ipeerconnection
This is low hanging fruit. We might as well start somewhere.
Differential Revision: https://phab.mercurial-scm.org/D3254
Augie Fackler <augie@google.com> [Thu, 12 Apr 2018 13:25:54 -0400] rev 37608
py3: whitelist another six passing tests
Differential Revision: https://phab.mercurial-scm.org/D3286
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 17:43:00 -0400] rev 37607
py3: whitelist another nine passing tests
Differential Revision: https://phab.mercurial-scm.org/D3253
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 14:01:37 -0400] rev 37606
hgweb: use our forked wsgiheaders module instead of stdlib one
Now we use bytes for headers, rather than native strings.
Differential Revision: https://phab.mercurial-scm.org/D2854
Augie Fackler <augie@google.com> [Thu, 12 Apr 2018 10:00:09 -0700] rev 37605
wsgiheaders: import a bytes-ified fork of wsgiref.headers from cpython@46f5072
This will let us restore Python 3 compatibility for tests that do http things.
Differential Revision: https://phab.mercurial-scm.org/D3245
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 23:13:55 +0900] rev 37604
export: enable formatter support (API)
This change is basically the same as "hg cat". A formatter object is created
by caller.
.. api::
``cmdutil.export()`` takes a formatter as an argument.
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 22:59:49 +0900] rev 37603
export: extract function to write patch to file object (API)
This is common use case of cmdutil.export(), and we wouldn't want to handle
formatter thingy everywhere.
.. api::
The ``fp`` argument is removed from ``cmdutil.export()``. Use
``cmdutil.exportfile()`` instead.
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 22:39:43 +0900] rev 37602
export: port _exportsingle() to formatter
Pass 'fm' instead of 'write', and use fm.plain(), fm.write(), etc. instead.
The callers will be updated later.
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 22:08:55 +0900] rev 37601
export: serialize revisions to be exported per destination file
Prepares for porting to the formatter API, where we can't simply append
to existing files because JSON can't be streamed for example.
The modemap hack is removed since cmdutil.export() was the only user.
I also made the destination filename printed only once.
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 21:56:02 +0900] rev 37600
export: split cmdutil.export() to single-file and maybe-multiple-files cases
Porting "hg export" to formatter is a bit hard because cmdutil.export() may
append to files if the fntemplate is specified. This patch splits the hard
part from the trivial case.
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 21:47:17 +0900] rev 37599
export: remove unused argument 'rev' from _exportsingle()
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 21:33:47 +0900] rev 37598
export: do not start pager if output will be written to file
A copy of 3b569745af6c.
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 23:25:58 +0900] rev 37597
formatter: carry opts to file-based formatters by basefm
This makes it slightly easier to port "hg export" to formatter.
Yuya Nishihara <yuya@tcha.org> [Thu, 12 Apr 2018 23:24:33 +0900] rev 37596
formatter: remove unused private attributes from baseformatter
No idea what they were for.
Danny Hooper <hooper@google.com> [Fri, 30 Mar 2018 16:40:25 -0700] rev 37595
fix: add --all flag to fix non-public non-obsolete revisions
Differential Revision: https://phab.mercurial-scm.org/D3213
Augie Fackler <augie@google.com> [Sat, 24 Mar 2018 14:28:24 -0400] rev 37594
fsmonitor: layer on another hack in bser.c for os.stat() compat (issue5811)
It's unclear to me how these `bserobj_tuple` objects are used, other
than as stat objects. This should fix fsmonitor in the wake of
ffa3026d4196 and similar changes. I regret the hack here, but the code
already has plenty of hg-specific hacks. :(
It feels like we should be able to use int(result.st_mtime) globally,
but that doesn't work. See issue4836 for a bug that was hard to track
down relating to rounding behavior causing very subtle dirstate
problems.
Differential Revision: https://phab.mercurial-scm.org/D2939
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 17:24:38 -0400] rev 37593
tests: use `f --newer` instead of `stat -c` in test-fix.t
Also increase sleep to two seconds so this test will likely pass on FAT32.
Differential Revision: https://phab.mercurial-scm.org/D3252
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 17:07:07 -0400] rev 37592
tests: glob away fqdn wherever we print it
These localhost instances are actually from a getfqdn call, which
means on some of my test systems it comes out as localhost.localdomain
or
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa. I'm
tired of this, so let's glob it away.
Differential Revision: https://phab.mercurial-scm.org/D3251
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 16:26:40 -0400] rev 37591
hgweb: use native strings when interfacing with stdlib headers
We're still parsing the stdlib-provided headers here, so we need to
tread carefully and use native strings. Yuck.
Differential Revision: https://phab.mercurial-scm.org/D3250
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 16:10:02 -0400] rev 37590
wireprotoserver: headers are bytes for us internally, use bytes
This re-fixes test-pull-http.t on Python 3. Probably many others as well.
Differential Revision: https://phab.mercurial-scm.org/D3249
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 16:09:23 -0400] rev 37589
hgweb: put response headers back into str for Python 3
This fixes a lot of hanging tests on Python 3, because
"Content-Length" was getting sent as "b'Content-Length'" (yes, really)
and then clients would expect a close-is-end body instead of counting
off a certain number of bytes.
Differential Revision: https://phab.mercurial-scm.org/D3248
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 14:57:11 -0400] rev 37588
tests: load showstack in test-pull-http.t so network hangs are easier to find
This also gives us some minimal "it loads" coverage on showstack,
which I rather like. showstack doesn't work on Windows per mbarbison,
so it's disabled there.
I added this in service of debugging a hang introduced on Python 3 by
revision a88d68dc3ee8. I'm still not sure what the problem there is,
but this at least gives us a little bit of a chance to figure out
what's going on.
Differential Revision: https://phab.mercurial-scm.org/D3247
Augie Fackler <augie@google.com> [Wed, 11 Apr 2018 14:39:49 -0400] rev 37587
keepalive: rewrite readinto() to not use read()
It turns out http.client on Python 3 sometimes uses readinto() in the
implementation of read(). Unfortunately, Python 2 doesn't have a
readinto() in httplib's client, so we have to support both codepaths.
Subclassing is bad, folks.
Differential Revision: https://phab.mercurial-scm.org/D3246
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 17:22:59 +0530] rev 37586
py3: use bytes() instead of str() on util.url()
We internally deal with bytes and anything as string breaks things.
Differential Revision: https://phab.mercurial-scm.org/D3285
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 17:21:56 +0530] rev 37585
py3: use stringutil.forcebytestr() to convert error messages to bytes
Differential Revision: https://phab.mercurial-scm.org/D3284
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 08:06:39 -0700] rev 37584
py3: suppress the return value of write() in tests/test-subrepo-missing.t
write() on Python 3 returns a value whereas does not return anything on Python 2.
So we need to supress the value.
Differential Revision: https://phab.mercurial-scm.org/D3283
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 16:52:32 +0530] rev 37583
py3: add b'' prefixes in tests/test-linerange.py
This makes the test pass on Python 3.
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3282
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 15:23:36 +0530] rev 37582
py3: add b'' prefix to make the regex bytes
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D3281
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 15:22:56 +0530] rev 37581
py3: use b'%d' to convert int to bytes instead of str()
Differential Revision: https://phab.mercurial-scm.org/D3280
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 15:03:15 +0530] rev 37580
py3: use print as a function in test-convert-git.t
Differential Revision: https://phab.mercurial-scm.org/D3278
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 14:30:37 +0530] rev 37579
py3: use '%d' for integers instead of '%s'
Differential Revision: https://phab.mercurial-scm.org/D3277
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 14:28:08 +0530] rev 37578
py3: prevent transformer from adding b'' by adding r'' prefix
These are cases where we need to use str, therefore we add r'' prefix.
# skip-blame because just r'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3276
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 12 Apr 2018 14:26:31 +0530] rev 37577
py3: use pycompat.{strkwargs|byteskwargs} in infinitepush
Differential Revision: https://phab.mercurial-scm.org/D3275
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 11 Apr 2018 22:36:16 +0530] rev 37576
py3: make sure we open file in bytes mode
Differential Revision: https://phab.mercurial-scm.org/D3274
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 01:37:25 +0900] rev 37575
diffhelpers: be more tolerant for stripped empty lines of CRLF ending
Exchange. It appears to trim lines containing only whitespace as well as
converting LF to CRLF.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 21:08:52 +0900] rev 37574
diffhelpers: make return value of testhunk() more Pythonic
It's no longer C.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 21:06:46 +0900] rev 37573
patch: error out if reached to EOF while reading hunk
This was where out-of-bounds read occurred in old C extension.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 20:55:05 +0900] rev 37572
diffhelpers: remove unused return value from fixnewline() and addlines()
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 20:54:00 +0900] rev 37571
diffhelpers: move out of pure package
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 20:52:54 +0900] rev 37570
diffhelpers: naming and whitespace cleanup
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 20:51:23 +0900] rev 37569
diffhelpers: remove C implementation in favor of pure Python version
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 20:49:39 +0900] rev 37568
patch: stop using cext.diffhelpers
The C implementation has a couple of memory bugs, and lacks error handling
which could lead to SEGV. I could fix them one by one (and I mostly finished
that), but the performance gain provided by cext.diffhelper is quite low.
Besides, diffhelpers.addlines() calls back Python, linereader.readline(),
from the innermost loop.
$ hg export -R mozilla-central 0:100 > patch
$ ls -lh patch
-rw-r--r-- 184M patch
$ hg init repo && hg -R repo import patch --time --bypass
(cext) time: real 34.970 secs (user 32.720+0.000 sys 2.230+0.000)
(pure) time: real 35.950 secs (user 33.600+0.000 sys 2.330+0.000)
So, let's simply use the pure Python implementation.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 20:47:43 +0900] rev 37567
diffhelpers: port docstrings from cext to pure
I'll remove the C implementation.
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Apr 2018 20:44:41 +0900] rev 37566
py3: get rid of character access from pure.diffhelpers
's' is a result of readline(), so 'c == "\n"' means 's == "\n"'.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Apr 2018 18:23:29 -0400] rev 37565
lfs: handle paths that don't end with '/' when inferring the blob store
While here, I also checked the lfs.url config directly instead of testing the
scheme, as requested by Yuya.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Apr 2018 14:22:12 -0400] rev 37564
lfs: infer the blob store URL from an explicit push dest or default-push
Unlike pull, the blobs are uploaded within the exchange.push() window, so simply
wrap it and swap in a properly configured remote store. The '_subtoppath' field
shouldn't be available during this window, but give the passed path priority for
clarity.
At one point I hit an AttributeError in one of the convert tests when trying to
save the original remote blobstore when the swap was run unconditionally. I
wrapped it in a util.safehasattr(), but then today I wasn't able to reproduce
it. But now the whole thing is tucked under the requirement guard because
without the requirement, there are no blobs in the repo, even if the extension
is loaded.
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Apr 2018 01:23:39 -0400] rev 37563
lfs: infer the blob store URL from an explicit pull source
I don't see any easier way to do this because the update part of `hg pull -u`
happens outside exchange.pull(), and commands.postincoming() doesn't take a
path. So (ab)use the mechanism used by subrepos to redirect where subrepos are
pulled from when an explicit path is given. As a bonus, this should allow lfs
blobs to be pulled into a subrepo when it is checked out.
An explicit push path can be handled within exchange.push(). That can be done
next, outside of this dirty hack.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Apr 2018 17:29:55 -0400] rev 37562
lfs: special case the null:// usercache instead of treating it as a url
The previous code worked on Windows, but not on Unix, and a pending patch's test
failed. The url being used was something like "/tmp/.../client1/null://",
courtesy of ui.configpath(). Looking at the doc comment, this seems like it's
maybe not the right function to call (why should a relative cache path be
expanded relative to the repo root or config file?), but largefiles has been
using it since 8b8dd13295db (Oct 2011). It was introduced in 1b591f9b7fd2 (Jan
2011) without comment or callers. A grep over the whole history shows that only
largefiles used it until lfs and infinitepush came along recently.
It looks like if the `if not os.path.isabs(v) or "://" not in v` in configpath()
is changed to an 'and', both Linux and Windows are happy. I'm guessing that
"://" is to pick off URLs, so that seems reasonable. But I'm not sure why it
isn't explicitly "file://", and I thought that "file://foo" is relative anyway.
(At least, there are doctests for file:///tmp in util.url.) There is no mention
of this setting in the help, but it is referenced on the wiki page for
largefiles. (There's no mention that this is intended to be a URL, and the
example uses an absolute path.)
I don't want this blocking the rest of the lfs server discovery stuff. It was
also wrong to allow a file:// URL here, but not in largefiles.
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 04 Apr 2018 17:37:35 +0530] rev 37561
tests: add tests showing pulling from infinitepush works over wire
The current tests in test-infinitepush-ci.t showed that `hg pull -r <rev>` does
not work. Digging in code, I found that we have logic for pulling from
bundlestore without having client side logic. This patch adds test demonstrating
that pulling from bundlestore works when working over wire.
Pulling from bundlestore when the peer is a localpeer still does not works.
Differential Revision: https://phab.mercurial-scm.org/D3072
Danny Hooper <hooper@google.com> [Fri, 30 Mar 2018 17:01:12 -0700] rev 37560
fix: use a portable python script instead of sed in test
Differential Revision: https://phab.mercurial-scm.org/D2988
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 11 Apr 2018 14:35:37 +0530] rev 37559
py3: use pycompat.bytestr() where repr in involved
Differential Revision: https://phab.mercurial-scm.org/D3244
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 18:16:47 -0700] rev 37558
httppeer: support protocol upgrade
With the new handshake defined and in place on the server, we can
now implement it on the client.
The HTTP handshake mechanism has been taught to add headers advertising
its support for the new capabilities response. Response handling
has been adjusted to allow CBOR responses through. And makepeer()
has been taught to instantiate a mutually supported peer.
The HTTPv2 peer class doesn't implement the full peer interface. So
HTTPv2 is not yet usable as a peer.
Like the server side, we support registering handlers for
different API services. This allows extensions to easily implement
API services and peers. A practical use case for this is to
provide a previous implementation of the experimental version 2
wire protocol to a future version of Mercurial. We know there will
be BC breaks after 4.6 ships. But someone could take the peer and
server code from 4.6, drop it in an extension, and allow its use
indefinitely.
Differential Revision: https://phab.mercurial-scm.org/D3243
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 14:29:15 -0700] rev 37557
wireproto: define and implement HTTP handshake to upgrade protocol
When clients connect to repositories over HTTP, they issue a request
to the well-known URL "?cmd=capabilities" to fetch the repository
capabilities. This is the handshake portion of the HTTP protocol.
This commit defines a mechanism to use that HTTP request to return
information about modern server features.
If a client sends an X-HgUpgrade-* header containing a list of
client-supported API names, the server responds with a response
containing information about available services. This includes
the normal capabilities string. So if the server doesn't support
any newer services, the client can easily fall back.
By advertising supported services from clients, server operators
can see and log what client support exists in the wild. This will
also help with debugging.
The response contains the base path to API services. We know there
are potential issues with the <repo>/api/ URL space conflicting with
hgwebdir and subrepos. By making the API URL dynamic from the
perspective of the client, the URL for APIs is not subject to backwards
compatibility concerns - at least as long as a ?cmd=capabilities request
is made.
We've also defined the ``cbor`` client capability for the X-HgProto-*
header. This MUST be sent in order to get the modern response from
"?cmd=capabilities". During implementation, I initially always sent
an application/mercurial-cbor response. However, the handshake
mechanism will be more future compatible if the client is in charge
of which formats to request. We already perform content negotiation
from X-HgProto-*, so keying off this for the capabilities response
feels appropriate.
In addition, I initially used application/cbor. However, it is
conceivable that a non-Mercurial server could serve application/cbor.
To rule out this possibility, I've invented a new media type that
is Mercurial specific and can't be confused for generic CBOR.
Differential Revision: https://phab.mercurial-scm.org/D3242
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 18:13:28 -0700] rev 37556
httppeer: only advertise partial-pull if capabilities are known
We don't need to be advertising client protocol parameters as part
of the capabilities request during the handshake because nothing in
version 1 of the wire protocol will use this data. i.e. the
advertisement is wasteful.
Differential Revision: https://phab.mercurial-scm.org/D3241
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 16:53:44 -0700] rev 37555
httppeer: always add x-hg* headers to Vary header
Before, we manually updated the Vary header value for each
header contributing to it.
All X-Hg* headers are reserved for the Mercurial protocol and
could have caching implications. So it makes sense to always add
these headers to Vary.
A test revealed that X-HgArgs-Post wasn't being added to Vary.
This is only sent on POST requests. POST requests generally
aren't cacheable. However, it is possible if the server sends
the appropriate headers. Mercurial shouldn't be sending those
headers. But let's not take any chances.
Differential Revision: https://phab.mercurial-scm.org/D3240
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 13:41:21 -0700] rev 37554
httppeer: don't accept very old media types (BC)
Versions of Mercurial older than 1.0 emitted the text/plain
and application/hg-changegroup media types in response to wire
protocol commands.
Way back in 8760d0c83b9b in 2005, the code validating these media
types was added, presumably for backwards compatibility. 0b245edec124
a short time before that commit changed things from text/plain and
application/hg-changegroup to application/mercurial-0.1 and
application/hg-0.1. 8760d0c83b9b seemed to indicate ("for now") that
the BC compatibility was temporary. But that code has lived until
this day.
It has been more than 10 years and nobody should be running pre 1.0
servers.
Pretty much the only risk to this is if there's a server somewhere
advertising the old media types or server software is interfering
and not letting Mercurial send the proper Content-Type header. I
think the chances are rare.
The wire protocol docs were created (by me) from reading existing
code. So the deletions don't constitute a spec change as much as
reflecting the reality of how things have been for years.
.. bc::
The HTTP client no longer accepts text/plain and
application/hg-changegroup Content-Type values as a valid Mercurial
command response. These should only be encountered on pre 1.0
Mercurial servers.
Differential Revision: https://phab.mercurial-scm.org/D3239
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 13:07:13 -0700] rev 37553
httppeer: allow opener to be passed to makepeer()
This allows us to use makepeer() in `hg debugwireproto`.
Differential Revision: https://phab.mercurial-scm.org/D3238
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 13:11:40 -0700] rev 37552
httppeer: perform capabilities request in makepeer()
Previously, we constructed an httppeer then always ran _fetchcaps()
to issue the capabilities command.
We want to issue the capabilities command before constructing a
peer instance so we can construct an appropriate peer instance
depending on the capabilities result.
With the code for making and sending requests moved out of httppeer,
it is now possible to send command requests without an httppeer.
This commit creates a new function for making the capabilities
request and calls it as part of makepeer().
This code should be functionality equivalent to what existed before.
Differential Revision: https://phab.mercurial-scm.org/D3237
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 12:52:29 -0700] rev 37551
httppeer: extract common response handling into own function
This allows the common redirect detection, content type
validation, and decompression wrapping to be usable outside of
httppeer instances.
Differential Revision: https://phab.mercurial-scm.org/D3236
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 12:12:07 -0700] rev 37550
httppeer: move error handling and response wrapping into sendrequest
This is common for all HTTP requests. It should be part of
sendrequest().
Differential Revision: https://phab.mercurial-scm.org/D3235
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:51:12 -0700] rev 37549
httppeer: extract code for creating a request into own function
Some of this feels awkward, such as having to pass in a function
to evaluate a capability. And this code is generally pretty difficult
to read. I didn't want to perform too much refactoring as part of
the code move since it would make review more difficult.
Differential Revision: https://phab.mercurial-scm.org/D3234
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:27:49 -0700] rev 37548
httppeer: extract code for performing an HTTP request
This is generic and doesn't need to live as a method of httppeer.
Differential Revision: https://phab.mercurial-scm.org/D3233
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Apr 2018 10:22:26 -0700] rev 37547
httppeer: move requestbuilder defaults into makepeer() argument
Upcoming commits will move the initial ?cmd=capabilities handshake
request out of httppeer so the handshake can be performed before a
peer instance is constructed. In order to do this, we'll need to
refactor code for making HTTP requests.
The type used to construct HTTP requests is configurable. If we'll
be making HTTP requests outside of httppeer, we should be able to
use a custom request builder. So move the definition of that type
into makepeer().
Extensions can monkeypatch the function and override the argument
value.
Differential Revision: https://phab.mercurial-scm.org/D3232
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 19:35:39 -0700] rev 37546
wireproto: move version 2 command handlers to wireprotov2server
This is relatively straightforward.
As part of this, we introduced a local @wireprotocommand that
wraps the main one and defines a v2 only policy by default.
Because the hacky HTTPv2 peer isn't using capabilities response
yet, we had to move some code around to force import of
wireprotov2server so commands are registered. This is super
hacky. But this code will go away once the HTTPv2 peer is using
the capabilities response to derive permissions.
Differential Revision: https://phab.mercurial-scm.org/D3231
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 19:35:04 -0700] rev 37545
wireproto: extract HTTP version 2 code to own module
wireprotoserver has generic and version 1 server code. The wireproto
module also has both version 1 and version 2 command implementations.
Upcoming work I want to do will make it difficult for this code to
live in the current locations. Plus, it kind of makes sense for the
version 2 code to live in an isolated module.
This commit copies the HTTPv2 bits from wireprotoserver into a new
module. We do it as a file copy to preserve history. A future
commit will be copying wire protocol commands into this module
as well. But there is little history of that code, so it makes
sense to take history for wireprotoserver.
Differential Revision: https://phab.mercurial-scm.org/D3230
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 16:54:20 -0700] rev 37544
wireproto: client reactor support for receiving frames
We can now feed received frames into the client reactor and it will
validate their sanity, dispatch them appropriately.
The hacky HTTP peer has been updated to use the new code. No
existing tests changed, somewhat proving the code works as
expected.
Rudimentary unit tests for the new functionality have been
implemented.
Differential Revision: https://phab.mercurial-scm.org/D3224
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 15:32:01 -0700] rev 37543
wireproto: introduce a reactor for client-side state
We have a nice state machine of sorts for reacting to server-side
events. Now it is time to implement the client equivalent.
We introduce a "clientreactor." It allows callers to request
that commands be issued. It has multiple modes of operation to
reflect what the underlying transport supports. e.g. for SSH,
we can perform wire sends immediately but for HTTP we need to
buffer sends until all command requests are received. In addition,
SSH allows sending multiple requests as long as the connection is
open. But HTTP/1.1 only allows sending request data once.
For SSH, we'll have one reactor per connection. For HTTP, we'll
have one reactor per HTTP request. But because code that calls
wire protocol commands should not be aware of how the underlying
transport works, this will all be abstracted away by the peer
interface.
Our crude HTTP peer has been updated to use the reactor instead
of formulating frames directly. No behavior should have changed
here and tests seem to confirm that.
Basic unit tests for the reactor behavior have been added.
Differential Revision: https://phab.mercurial-scm.org/D3223
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 14:17:57 -0700] rev 37542
tests: extract wire protocol framing tests to own file
I was lazy when I put these in test-wireproto-serverreactor.py. Let's
do it properly.
Differential Revision: https://phab.mercurial-scm.org/D3222
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:33:38 -0700] rev 37541
wireproto: disallow commands handlers for multiple transport versions
I think it will be more trouble than it is worth to code version 1
and version 2 command handlers to the same interface. It will feel
awkward to shoehorn functionality into e.g. the version 1 protocol
handler interface. This would likely constrain the ability for version
2 to evolve.
Previous commits introduced a clean separation between command handlers
for version 1 and version 2 transports. This commit reinforces that
separation by dropping support for having a single command handler
service both version 1 and version 2 transports.
Differential Revision: https://phab.mercurial-scm.org/D3208
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:57:12 -0700] rev 37540
wireproto: make @wireprotocommand version 1 only by default
For backwards compatibility reasons. We want extension provided
commands to opt in to version 2 rather than get inherited
automatically. This will facilitate a clean break between the
protocols.
As part of this, we duplicate some commands used in tests so
there are different command handlers per transport.
Differential Revision: https://phab.mercurial-scm.org/D3207
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:54:31 -0700] rev 37539
wireproto: only expose "getbundle" and "unbundle" to v1 transports
These are the most complicated wire protocol commands. I don't want
to deal with porting them just yet. Let's disable both of them on
version 2 transports so we drive the final wedge between command
handlers and start to evolve version 2 command handlers more.
Differential Revision: https://phab.mercurial-scm.org/D3206
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:48:07 -0700] rev 37538
wireproto: port lookup to wire protocol v2
This is pretty straightforward. We don't yet handle errors because we
don't have an error handling mechanism in place yet.
I'm also tempted to fold this into `known`. We'll come back to this
later.
Differential Revision: https://phab.mercurial-scm.org/D3205
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:39:40 -0700] rev 37537
wireproto: port pushkey command to wire protocol version 2
It doesn't do output redirection yet. And I'd love to generally overhaul
the pushkey protocol for wire protocol version 2. But this will be a bit
of effort. Let's do it as a follow-up.
Differential Revision: https://phab.mercurial-scm.org/D3204
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:21:16 -0700] rev 37536
wireproto: only expose "clonebundles" to version 1 transports
This may make a comeback in wire protocol version 2. The feature
definitely needs to be carried forward. But at this juncture, I'm
flirting with the idea of implementing this via a "redirect"
mechanism at the command response level itself rather than something
that requires one-off client support for querying and handling.
i.e. I want to make it so servers can say "fetch this first and
then come back" and clients handle that automatically. This would
not only support clone bundles, but would also support piece-meal
"pull bundles." Whatever happens, we can deal with it down the
road.
Differential Revision: https://phab.mercurial-scm.org/D3203
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 17:14:06 -0700] rev 37535
wireproto: define and expose types of wire command arguments
Exposing the set of argument names is cool. But with wire protocol
version 2, we're using CBOR to transport arguments and this allows us
to have typing for arguments.
Typed arguments are much nicer because they will cut down on transfer
overhead and processing overhead for decoding values.
This commit teaches @wireprotocommand to accept a dictionary for
arguments. The arguments registered for version 2 transports are
canonically stored as dictionaries rather than a space-delimited string.
It is an error to defined arguments with a dictionary for commands using
version 1 transports. This reinforces my intent to fully decouple command
handlers for version 2 transports.
Differential Revision: https://phab.mercurial-scm.org/D3202
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 16:49:57 -0700] rev 37534
wireproto: only expose "stream_out" to version 1 transports
I have plans to implement stream clone using a better mechanism than
this existing command. Let's not carry it forward to wire protocol
version 2.
Differential Revision: https://phab.mercurial-scm.org/D3201
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 11:52:31 -0700] rev 37533
wireproto: implement capabilities for wire protocol v2
The capabilities mechanism for wire protocol version 2 represents a
clean break from version 1.
Instead of effectively exchanging a set of capabilities, we're
exchanging a rich data structure.
This data structure currently contains information about
every available command, including its accepted arguments. It also
contains information about supported compression formats.
Exposing information about supported commands will allow clients
to automatically generate bindings to the server. Clients will be
able to do things like detect when they are attempting to run a
command that isn't known to the server. Exposing the required
permissions to run a command can be used by clients to determine if
they have privileges to call a command before actually calling it.
We could potentially even have clients send credentials
preemptively without waiting for the server to deny the command
request. Lots of potential here.
The data returned by this command will likely evolve heavily. So we
shouldn't bikeshed the implementation just yet.
Differential Revision: https://phab.mercurial-scm.org/D3200
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:45:45 -0700] rev 37532
context: add deprecation warnings for deprecated types of changeids
It's close to code freeze, and dropping support for repo['123'] and
repo ['my-bookmark'] and repo['deadbeef'] is pretty dispruptive, so
this just adds deprecation warnings so extensions can easily find the
places they need to fix.
Differential Revision: https://phab.mercurial-scm.org/D3197
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:28:08 -0700] rev 37531
revsymbol: stop delegating to repo.__getitem__ for unhandled symbols (API)
The only remaining cases where we were delegating unhandled symbols to
repo.__getitem__ should now be when the symbol could not be found. In
that case we just delegated to repo.__getitem__ for the error
message. Let's just copy the error message instead.
If there were any cases where we got e.g. a binary nodeid or an
integer revnum into revsymbol() (e.g. via repo.lookup()), we'd now
start raising an exception instead. That is why this is marked (API).
This affects one test case, but the new behavior seems better to me. I
can't tell if the old behavior was desired or if the test was just
there to document how it happened to work.
Differential Revision: https://phab.mercurial-scm.org/D3196
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:28:49 -0700] rev 37530
context: handle partial nodeids in revsymbol()
Similar reasoning as previous patches.
Differential Revision: https://phab.mercurial-scm.org/D3195
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:46:17 -0700] rev 37529
context: handle namespaces in revsymbol()
Similar reasoning as previous patches.
Differential Revision: https://phab.mercurial-scm.org/D3194
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:43:52 -0700] rev 37528
scmutil: handle full hex nodeids in revsymbol()
This is a bit unfortunate, but it enables moving other pieces out of
changectx's constructor without affecting the order in which we
look up things (e.g. hex nodeid before bookmark).
We convert nodeid to revnum before calling repo.__getitem__, even
though that will result in converting back to nodeid later. This is so
we can handle the LookupError and attempt to interpret the string as
something else (e.g. a bookmark).
We also need to start handling WdirUnsupported now, since the full hex
nodeid "ffff..." represents the working directory. The exception is
raised by the revlog layer.
Differential Revision: https://phab.mercurial-scm.org/D3193
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 23:39:40 -0700] rev 37527
context: handle stringified ints in revsymbol()
This patch copies the handling of stringified ints from changectx's
constructor. It then calls repo.__getitem__ with the int. Since that
method only interprets integers as revnums the first thing it does,
this will not be redoing any of the work already done. We leave the
old code in place so we can later deprecate it instead of breaking
extensions.
Differential Revision: https://phab.mercurial-scm.org/D3146
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 19:32:08 +0530] rev 37526
py3: make sure we write bytes to file
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D3220
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 19:21:16 +0530] rev 37525
py3: workaround comparing NoneType and integers
Comparing None with integers was fine in Python 2 but returns error in Python 3
which is nice. This patch replaces None with -1 where sorting is done and some
related logic.
Differential Revision: https://phab.mercurial-scm.org/D3219
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:57:56 +0530] rev 37524
py3: use sys.stdout instead of print in test-mq-qpush-fail.t
Differential Revision: https://phab.mercurial-scm.org/D3218
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:57:32 +0530] rev 37523
py3: use '//' for integer division in tests/test-mq-qimport.t
Differential Revision: https://phab.mercurial-scm.org/D3217
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 11 Apr 2018 14:35:28 +0530] rev 37522
py3: use pycompat.byterepr() instead of repr()
Differential Revision: https://phab.mercurial-scm.org/D3216
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:56:19 +0530] rev 37521
py3: use bytes instead of str in isinstance()
We deal with bytes internally and things should be bytes
Differential Revision: https://phab.mercurial-scm.org/D3215
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 18:55:35 +0530] rev 37520
py3: use .startswith() instead of bytes[0]
Differential Revision: https://phab.mercurial-scm.org/D3214
Pulkit Goyal <7895pulkit@gmail.com> [Tue, 10 Apr 2018 20:14:12 +0530] rev 37519
py3: add b'' prefixes in tests/test-ui-config.py
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D3221
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 22:22:20 -0400] rev 37518
lfs: infer the blob store URL from paths.default
If `lfs.url` is specified, it takes precedence. However, now that we support
serving blobs via hgweb, we shouldn't *require* this setting. Less
configuration is better (things will work out of the box once this is sorted
out), and git has similar functionality.
This is not a complete solution- it isn't able to infer the blob store from an
explicitly supplied path, and it should consider `paths.default-push` for push.
The pull solution for that is a bit hacky, and this alone is an improvement for
the vast majority of cases.
Even though there are only a handful of references to the saved remote store,
the location of them makes things complicated.
1) downloading files on demand in the revlog flag processor
2) copying to readonlyvfs with bundlerepo
3) downloading in the file prefetch hook
4) the canupload()/skipdownload() checks
5) uploading blobs
Since revlog doesn't have a repo or ui reference, we can't avoid creating a
remote store when the extension is loaded. While the long term goal is to make
sure the prefetch hook is invoked early for every command for efficiency, this
handling in the flag processor is needed as a last ditch fetch.
In order to support the clone command, the remote store needs to be created
later than when the extension loads, since `paths.default` isn't set until just
before the files are checked out. Therefore, this patch changes the prefetch
hook to ignore the saved reference, and build a new one.
The canupload()/skipdownload() checks simply check if the stored instance is a
`_nullremote`. Since this can only be set via `lfs.url` (which is reflected in
the saved reference), checking only the instance created when the extension
loaded is fine.
The blob uploading function is called from several places:
1) a prepush hook
2) when writing a new bundle
3) from infinitepush
The prepush hook gets an exchange.pushop, so it has a path to where the push is
going. The bundle writer and infinitepush don't. Further, bundle creation for
things like strip and amend are causing blobs to be uploaded. This seems wrong,
but I don't want to side track this sorting that out, so punt on trying to
handle explicit push paths or `paths.default-push`.
I also think that sending blobs to a remote store when pushing to a local repo
is wrong. This functionality predates the usercache, so perhaps that's the
reason for it. I've got some patches floating around to stop sending blobs
remotely in this case, and instead write directly to the other repo's blob
store. But the tests for corruption handling weren't happy with this change,
and I don't have time to rewrite them. So exclude filesystem based paths from
this for now.
I don't think there's much of a chance to implement `paths.remote:lfsurl` style
configs, given how early these are resolved vs how late the remote store is
created. But git has it, so I threw a TODO in there, in case anyone has ideas.
I have no idea why this is now doing http auth twice when it wasn't before. I
don't think the original blobstore's url is ever being used in these cases.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 22:40:11 -0400] rev 37517
lfs: add the ability to disable the usercache
While the usercache is important for real world uses, I've been tripped up more
than a couple of times by it in tests- thinking a file was being downloaded, but
it was simply linked from the local cache. The syntax for setting it is the
same as for setting a null remote endpoint, and like that endpoint, is left
undocumented.
This may or may not be a useful feature in the real world (I'd expect any sane
filesystem to support hardlinks at this point).
Matt Harbison <matt_harbison@yahoo.com> [Tue, 10 Apr 2018 22:57:55 -0400] rev 37516
tests: stabilize test-pull-bundle.t for Windows
See 594dd384803c.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:41:49 +0900] rev 37515
hgweb: wrap {archives} with mappinglist
No bare list of mappings should be put in a template mapping.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:40:15 +0900] rev 37514
hgweb: forward archivelist() of hgweb to webutil
self.configlist() is ui.configlist(untrusted=True), and url=None in templater
effectively means 'url' is undefined.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:37:03 +0900] rev 37513
hgweb: move archivelist() of hgwebdir to webutil
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:33:55 +0900] rev 37512
hgweb: drop archivespecs from requestcontext
It's a constant.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:32:34 +0900] rev 37511
hgweb: move archivespecs to webutil
Move one step further from e38e7ea21987. Since 'archivespecs' is a constant
shared with hgweb and hgwebdir, webutil is a better place to define it.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:20:44 +0900] rev 37510
hgweb: wrap {labels} by hybridlist()
This one is a simple list of strings, which can support map operation.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:14:36 +0900] rev 37509
hgweb: wrap {pathdef} with mappinglist
No bare list of mappings should be put in a template mapping.
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 22:11:58 +0900] rev 37508
hgwebdir: wrap {entries} with mappinggenerator
No bare generator should be put in a template mapping.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 09:53:17 -0700] rev 37507
eol: look up partial nodeid as partial nodeid
Similar reasoning as the previous patch. For some reason the hook
gives us a partial nodeid, so we need to resolve that to a full
nodeid.
Differential Revision: https://phab.mercurial-scm.org/D3159
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 09:43:17 -0700] rev 37506
histedit: look up partial nodeid as partial nodeid
I'm about to remove support for repo[<partial hex nodeid>]. In the
verify() method, we know that self.node is always a partial or full
binary nodeid, so the most correct way to look up the revision is by
using changelog._partialmatch(), so let's do that. (It's closer to the
current code to do scmutil.revsymbol(), but that's less correct
because it will match a bookmark or tag that happens to have the same
prefix as the node.)
Differential Revision: https://phab.mercurial-scm.org/D3158
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 09:41:25 -0700] rev 37505
histedit: drop unnecessary check for "self.node is not None"
We are doing hex(self.node) just a few lines up, so it shouldn't be
None. The only way it could be none is if it was reassigned in
between. The only way that can happen is if the user had put a
"ffff..." wdirhex revision in the histedit script. This code is much
older than the "ffff..." identifier, so I'm confident it's not there
to handle that case. I'll let someone else add proper checks for
"ffff..." if they care enough.
Differential Revision: https://phab.mercurial-scm.org/D3157
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 08:06:34 -0700] rev 37504
context: extract partial nodeid lookup method to scmutil
We will add another caller soon, and there's a non-obvious reason to
use the unfiltered repo that we don't want to copy across the code
base.
Differential Revision: https://phab.mercurial-scm.org/D3189
Yuya Nishihara <yuya@tcha.org> [Mon, 19 Mar 2018 20:23:27 +0900] rev 37503
templatekw: fix return type of {succsandmarkers} (BC)
A hybrid object represents a list/dict of values, but {succsandmarkers}
returns a list of template mappings.
This change means old-style list templates (e.g. "start_succsandmarkers")
are no longer supported, but that should be okay since {succsandmarkers}
is still experimental and relatively new.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 23:34:38 +0900] rev 37502
formatter: remove template resources from nested items before generating JSON
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 21:09:37 +0900] rev 37501
templater: add public interface returning a set of resource keys
The next patch depends on knownresourcekeys(), and there's no reason to
keep availableresourcekeys() private.
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 22:27:16 +0900] rev 37500
formatter: make nested items somewhat readable in template output
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 23:36:52 +0900] rev 37499
templater: wrap result of '%' operation so it never looks like a thunk
This fixes min/max()/json() of map result. Before, it was taken as a lazy
byte string and stringified by evalfuncarg().
Joerg Sonnenberger <joerg@bec.de> [Thu, 18 Jan 2018 12:54:01 +0100] rev 37498
wireproto: support for pullbundles
Pullbundles are similar to clonebundles, but served as normal inline
bundle streams. They are almost transparent to the client -- the only
visible effect is that the client might get less changes than what it
asked for, i.e. not all requested head revisions are provided.
The client announces support for the necessary retries with the
partial-pull capability. After receiving a partial bundle, it updates
the set of revisions shared with the server and drops all now-known
heads from the request list. It will then rerun getbundle until
no changes are received or all remote heads are present.
Extend badserverext to support per-socket limit, i.e. don't assume that
the same limits should be applied to all sockets.
Differential Revision: https://phab.mercurial-scm.org/D1856
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 06 Apr 2018 22:39:58 -0700] rev 37497
filelog: wrap revlog instead of inheriting it (API)
The revlog base class exposes a ton of methods. Inheriting the
revlog class for filelog will make it difficult to expose a
clean interface. There will be abstraction violations.
This commit breaks the inheritance of revlog by the filelog
class. Filelog instances now contain a reference to a revlog
instance. Various properties and methods are now proxied to
that instance.
There is precedence for doing this: manifestlog does something
similar. Although, manifestlog has a cleaner interface than
filelog. We'll get there with filelog...
The new filelog class exposes a handful of extra properties and
methods that aren't part of the declared filelog interface.
Every extra item was added in order to get a test to pass. The
set of tests that failed without these extra proxies has
significant overlap with the set of tests that don't work with
the simple store repo. There should be no surprise there.
Hopefully the hardest part about this commit to review are the
changes to bundlerepo and unionrepo. Both repository types
define a custom revlog or revlog-like class and then have a
custom filelog that inherits from both filelog and their custom
revlog. This code has been changed so the filelog types don't
inherit from revlog. Instead, they replace the revlog instance
on the created filelog. This is super hacky. I plan to fix this
in a future commit by parameterizing filelog.__init__.
Because Python function call overhead is a thing, this change
could impact performance by introducing a nearly empty proxy
function for various methods and properties. I would gladly
measure the performance impact of it, but I'm not sure what
operations have tight loops over filelog attribute lookups
or function calls. I know some of the DAG traversal code can
be sensitive about the performance of e.g. parentrevs(). However,
many of these functions are implemented on the revlog class and
therefore have direct access to self.parentrevs() and aren't
going through a proxy.
.. api::
filelog.filelog is now a standalone class and doesn't inherit
from revlog. Instead, it wraps a revlog instance at self._revlog.
This change was made in an attempt to formalize storage APIs and
prevent revlog implementation details leaking through to callers.
Differential Revision: https://phab.mercurial-scm.org/D3154
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 10:18:10 -0700] rev 37496
util: drop write_content_size=True
This is now the default in python-zstandard 0.9. While we're here,
also add a comment about the ability to drop frame magic to save
space.
Differential Revision: https://phab.mercurial-scm.org/D3199
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 09 Apr 2018 10:13:29 -0700] rev 37495
zstandard: vendor python-zstandard 0.9.0
This was just released. It features a number of goodies. More info at
https://gregoryszorc.com/blog/2018/04/09/release-of-python-zstandard-0.9/.
The clang-format ignore list was updated to reflect the new source
of files.
The project contains a vendored copy of zstandard 1.3.4. The old
version was 1.1.3. One of the changes between those versions is that
zstandard is now dual licensed BSD + GPLv2 and the patent rights grant
has been removed. Good riddance.
The API should be backwards compatible. So no changes in core
should be needed. However, there were a number of changes in the
library that we'll want to adapt to. Those will be addressed in
subsequent commits.
Differential Revision: https://phab.mercurial-scm.org/D3198
Joerg Sonnenberger <joerg@bec.de> [Sun, 08 Apr 2018 01:08:43 +0200] rev 37494
revlog: reset _nodepos after strip
When using the pure revlog parser, _nodepos is used to keep track of the
position during index scanning in the non-cached cache. If it is out of
bounds, BaseIndexObject._fix_index will assert. Since strip can actually
remove the position scanned last, make sure to reset it. Add an
assertion in the place where the invariance is clearer.
Differential Revision: https://phab.mercurial-scm.org/D3188
Anton Shestakov <av6@dwimlabs.net> [Wed, 04 Apr 2018 13:14:48 +0800] rev 37493
paper: make all source lines have the same minimum height
Empty source lines in paper and coal themes used to have smaller height than
every other line (because of the way line numbers are shown and because they
are using smaller font). This wasn't very noticeable before the follow lines
functionality was added, but after that just using the follow-lines button to
select a block of code with empty lines would demonstrate the fact that empty
lines didn't have enough height - there were white "gaps" in the selection
block.
Since this problem occurs when lines don't have any content inside, let's
create a pseudo-element (it's unselectable because of that) which still doesn't
have any content, but fills up empty lines to 100% of their height because of
display: inline-block. This is the most natural way to solve this annoyance
that I've found so far.
Hardcoding height isn't useful because we can have wrapped lines, in which case
multiple lines of text need to fit into a single <span>.
Setting min-height or line-height doesn't remove the gaps when viewed in
Chromium.
Anton Shestakov <av6@dwimlabs.net> [Sun, 08 Apr 2018 20:53:07 +0800] rev 37492
hgweb: make followlines button absolutely positioned
It used to have position: absolute only on annotate page, but it makes sense to
have it everywhere, because the button shouldn't affect other elements at all.
Especially since the button has a set height, which meant that for certain
smaller fonts source lines were changing their height on hover.
Note that the button doesn't set any of the usual properties that accompany
absolute position (top, right, bottom or left). These properties would position
the button without any account for source line padding. Instead, margins are
used (the button already has all margins defined, they do the job).
Anton Shestakov <av6@dwimlabs.net> [Mon, 09 Apr 2018 22:00:11 +0800] rev 37491
hgweb: insert followlines buttons before any children, including text nodes
This way the buttons come before any other content, including text nodes.
Because highlight extension replaces every line of text with some <span>
elements that have CSS classes for highlighting, the placement of followlines
buttons used to depend on if that extension was enabled or not. Let's make the
placement more consistent, it'll help the next patch in this series.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 16:28:54 -0700] rev 37490
wireproto: only expose "debugwireargs" to version 1 transports
I'm not even sure this command should be enabled for version 1
transports. It is just a reflection endpoint for argument data.
We definitely don't need to support it in version 2.
Differential Revision: https://phab.mercurial-scm.org/D3184
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 16:14:42 -0700] rev 37489
wireproto: only expose "hello" command to version 1 transports
This command is only ever used for the handshake in the SSH protocol.
We probably don't even need for it to be a proper command. Let's not
carry it forward to version 2 because I don't see a use for it there.
Differential Revision: https://phab.mercurial-scm.org/D3183
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 16:10:41 -0700] rev 37488
wireproto: port branchmap to wire protocol v2
Differential Revision: https://phab.mercurial-scm.org/D3182
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 16:00:02 -0700] rev 37487
wireproto: port listkeys commands to wire protocol v2
Differential Revision: https://phab.mercurial-scm.org/D3181
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 15:44:47 -0700] rev 37486
wireproto: port keep command to wire protocol v2
This is pretty straightforward.
Differential Revision: https://phab.mercurial-scm.org/D3180
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 14:55:13 -0700] rev 37485
wireproto: port heads command to wire protocol v2
After much thought and consideration, wire protocol version 2's
commands will be defined in different functions from the existing
commands. This will make it easier to implement these commands
because it won't require shoehorning things like response formatting
and argument declaration into the same APIs.
For example, wire protocol version 1 requires that commands declare
a fixed and ordered list of argument names. It isn't really possible
to insert new arguments or have optional arguments without
breaking backwards compatibility. Wire protocol version 2, however,
uses CBOR maps for passing arguments. So arguments a) can be
optional b) can be added without BC c) can be strongly typed.
This commit starts our trek towards reimplementing the wire protocol
for version 2 with the heads command. It is pretty similar to the
existing heads command. One added feature is it can be told to
operate on only public phase changesets. This is useful for
making discovery faster when a repo has tens of thousands of
draft phase heads (such as Mozilla's "try" repository).
The HTTPv2 server-side protocol has had its `getargs()` implementation
updated to reflect that arguments are a map and not a list.
Differential Revision: https://phab.mercurial-scm.org/D3179
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 10:52:40 -0700] rev 37484
largefiles: wrap heads command handler more directly
extensions.wrapfunction() is a more robust method for wrapping a
function, since it allows multiple wrappers.
While we're here, wrap the function registered with the command instead
of installing a new command handler.
Differential Revision: https://phab.mercurial-scm.org/D3178
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 15:09:34 -0700] rev 37483
wireproto: crude support for version 2 HTTP peer
As part of implementing the server-side bits of the wire protocol
command handlers for version 2, we want a way to easily test those
commands. Currently, we use the "httprequest" action of `hg
debugwireproto`. But this requires explicitly specifying the HTTP
request headers, low-level frame details, and the data structure
to encode with CBOR. That's a lot of boilerplate and a lot of it can
change as the wire protocol evolves.
`hg debugwireproto` has a mechanism to issue commands via the peer
interface. That is *much* easier to use and we prefer to test with
that going forward.
This commit implements enough parts of the peer API to send basic
requests via the HTTP version 2 transport.
The peer code is super hacky. Again, the goal is to facilitate
server testing, not robustly implement a client. The client code
will receive love at a later time.
Differential Revision: https://phab.mercurial-scm.org/D3177
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Mar 2018 15:34:52 -0700] rev 37482
tests: extract wire protocol shell helpers to standalone file
This will make it easier for other tests to get up and running without
the boilerplate.
Differential Revision: https://phab.mercurial-scm.org/D3176
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 09:30:35 -0700] rev 37481
addbranchrevs: no longer accept revset as "revs" (API)
The only caller was removed in 0b4692b9646d (bundle: avoid
repo.lookup() for converting revnum to nodeid, 2018-04-02).
Differential Revision: https://phab.mercurial-scm.org/D3192
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 08:53:43 -0700] rev 37480
clone: avoid using repo.lookup() with binary nodeid
The code in hg.clone() is a bit of a mess, but it seems like the
"checkout" variable is always a binary nodeid (tests pass when run
with "assert len(checkout) == 20" before the
repo.lookup()). repo.lookup() will soon work only with string inputs,
so we need remove this use.
Differential Revision: https://phab.mercurial-scm.org/D3191
Martin von Zweigbergk <martinvonz@google.com> [Sun, 08 Apr 2018 08:41:58 -0700] rev 37479
merge: avoid unnecessary conversion from binary nodeid to binary nodeid
"node" is already a binary nodeid here, so there's no need to convert
it. repo.lookup() will soon lose support for binary nodeids as input.
Differential Revision: https://phab.mercurial-scm.org/D3190
Matt Harbison <matt_harbison@yahoo.com> [Sun, 08 Apr 2018 22:56:16 -0400] rev 37478
tests: stabilize test-push-http.t for Windows
See 594dd384803c.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Apr 2018 13:10:20 -0400] rev 37477
tests: add a substitution pattern for HTTP error log dates
This will make tests with error log content less of a nuisance. I didn't roll
it into to $LOGDATE$ because the regex got long and line wrapped. It didn't
seem worth the obscurity.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 11:55:46 +0900] rev 37476
wireproto: convert python literal to object without using unsafe eval()
Follows up cc5a040fe150.
At this point, I don't think we need a real eval(). If we want to support
a set literal, maybe we can vendor ast.literal_eval(), which is relatively
simple function.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 12:30:59 +0900] rev 37475
tests: quote variable passed to shell test command
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 11:23:55 +0900] rev 37474
py3: system-stringify repr(frame)
That's the Py3 requirement.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 11:21:58 +0900] rev 37473
wireproto: show unknown id and flags in repr(frame)
Perhaps we'll want it for debugging.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 11:14:47 +0900] rev 37472
wireproto: fix repr(frame) to not crash by unknown type id
Follows up 5ef2da00e935.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 15:39:08 +0900] rev 37471
py3: use s.startswith() instead of s[n] while parsing patches
I know 'bytes[n] in bytes' magically works, but I'm tired of finding which
one breaks the tests.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 15:32:09 +0900] rev 37470
py3: do not try to byte-stringify None in cmdutil.tryimportone()
It's a debug message, so just use '' instead.
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 15:22:30 +0900] rev 37469
py3: work around weird handling of bytes/unicode in decode_header()
Basically decode_header() works as follows, and on Python 3, email headers
ARE UNICODE.
def decode_header(header):
if not ecre.search(header): # ecre is unicode regexp
return [(header, None)] # so header is unicode string
... decode header into [(bytes_data, unicode_charset_name)]
return collapsed
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 15:03:00 +0900] rev 37468
py3: use system string to access email headers
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 14:59:12 +0900] rev 37467
py3: fix string issues of email message in test-import.t
- payload can be bytes
- headers must be unicode on Python 3
- need to call msg.as_bytes() on Python 3, but msg.as_string() on Python 2,
where bytes(msg) magic works
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 14:46:24 +0900] rev 37466
py3: use lower-cased module 'email.message' in test-import.t
Yuya Nishihara <yuya@tcha.org> [Sun, 08 Apr 2018 15:41:40 +0900] rev 37465
py3: drop b'' from error message of fancyopts
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 21:26:37 +0900] rev 37464
procutil: drop unused 'newlines' option from popen*() (API)
It's unlikely for us to use the universal_newlines option.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 21:23:42 +0900] rev 37463
procutil: make explainexit() simply return a message (API)
Almost all callers want it.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 21:21:03 +0900] rev 37462
procutil: do not convert return code of signal exit to positive number (API)
The docstring states that "codes from kill are negative", and it doesn't
make sense to make exit/signal code ambiguous.
.. api::
``hook.hook()`` and ``hook.runhooks()`` may return a negative integer
to denote that the process was killed by signal.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 21:17:50 +0900] rev 37461
procutil: fix error message of tempfile filter
First, we need to use procutil.system() to get an exit code compatible with
explainexit(). Second, explainexit() returns (msg, code) tuple.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 21:14:16 +0900] rev 37460
procutil: unify platform.explainexit()
Since 4368f582c806 "use subprocess instead of os.system", posix.explainexit()
is the superset of Windows implementation.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 21:09:21 +0900] rev 37459
procutil: rewrite popen() as a subprocess.Popen wrapper (issue4746) (API)
os.popen() of Python 3 is not the popen() we want. First, it doesn't accept
command in bytes. Second, a returned stream is always wrapped by TextIO.
So we have to reimplement our popen(). Fortunately, this fixes the bug 4746
since ours returns an exit code compatible with explainexit().
.. api::
``procutil.popen()`` no longer supports text mode I/O.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 20:50:38 +0900] rev 37458
procutil: always popen() in binary mode
On Python 3, non-binary stream is useless. Let's convert line ending by
ourselves.
Note that we don't need fromnativeeol() in patch._externalpatch() since
any whitespace characters are rstrip()-ed.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 13:46:32 +0900] rev 37457
py3: byte-stringify test-import.t
Still the test doesn't pass.
# skip-blame because just adding some b''
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 13:42:37 +0900] rev 37456
py3: convert parsed message items to bytes in patch.extract()
Appears that BytesParser() parses bytes into unicode, sigh.
Yuya Nishihara <yuya@tcha.org> [Sat, 07 Apr 2018 13:21:59 +0900] rev 37455
py3: silence warning about deprecation of imp module
Well, we could fix that, but we aren't yet to reach the state caring about
deprecation on the Python 3 line. So let's silence it for now to fix tons
of "minor" Py2/3 incompatibilities by relying on our test suite.
Martin von Zweigbergk <martinvonz@google.com> [Sat, 07 Apr 2018 00:00:33 -0700] rev 37454
tests: add another full hex node in plain text
Should have been part of D3168.
Differential Revision: https://phab.mercurial-scm.org/D3186
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 16:50:19 -0700] rev 37453
infinitepush: look up bookmarks only among bookmarks
Differential Revision: https://phab.mercurial-scm.org/D3165
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 10:48:11 -0700] rev 37452
destutil: look up bookmarks only among bookmarks
Differential Revision: https://phab.mercurial-scm.org/D3164
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 10:49:43 -0700] rev 37451
discovery: look up bookmarks only among bookmarks
Differential Revision: https://phab.mercurial-scm.org/D3163
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 11:29:30 -0700] rev 37450
bookmarks: introduce a repo._bookmarks.changectx(mark) method and use it
Many places were doing repo[mark], which usually works, but it's
slightly incorrect: if the bookmark has a name that matches a full hex
nodeid of another node, then the context for the other node will be
returned instead. Also, I'm about to remove support for
repo[<namespace thing>] :)
Differential Revision: https://phab.mercurial-scm.org/D3162
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Mar 2018 18:34:07 +0900] rev 37449
revlog: detect pseudo file nodeids to raise WdirUnsupported exception
Again, I'm not sure if this is the right thing, but adding a few more pseudo
hashes wouldn't be any worse than the current state.
Differential Revision: https://phab.mercurial-scm.org/D2942
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Mar 2018 18:31:22 +0900] rev 37448
node: rename wdirnodes to clarify they are for manifest/filelogs
Differential Revision: https://phab.mercurial-scm.org/D2941
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Mar 2018 18:27:43 +0900] rev 37447
workingctx: build _manifest on filenode() or flags() request
I'm not sure if this is the best workaround, but this fixes the following
exception:
AttributeError: 'workingctx' object has no attribute '_manifestdelta'
The short hash '303030303030' seen in the test is node.modifiednodeid.
Differential Revision: https://phab.mercurial-scm.org/D2940
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 13:11:40 -0700] rev 37446
tests: enter full hex hash in plain text in bundle part
We were looking it up be prefix by repo.__getitem__, which I'm about
to drop support for. It's easiest to just include the full hash in
plain text.
Differential Revision: https://phab.mercurial-scm.org/D3168
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 05 Apr 2018 17:44:18 -0700] rev 37445
tests: call rawsize() directly
rawsize() is not reimplemented outside of revlog. I'm not sure why
this code was insisting it call a specific implementation. Changing
it to call rawsize() on the repo.file(f) result seems to work just
fine.
Differential Revision: https://phab.mercurial-scm.org/D3153
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 05 Apr 2018 17:40:51 -0700] rev 37444
upgrade: sniff for filelog type
The upgrade code should never encounter a vanilla revlog
instance: only changelog, manifestrevlog, and filelog should
be seen.
The previous code assumed !changelog & !manifestrevlog meant
file data. So this change feels pretty safe. If nothing else, it
will help tease out typing issues.
Differential Revision: https://phab.mercurial-scm.org/D3152
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 05 Apr 2018 16:31:45 -0700] rev 37443
revlog: move censor logic into main revlog class
Previously, the revlog class implemented dummy methods for
various censor-related functionality. Revision censoring was
(and will continue to be) only possible on filelog instances.
So filelog implemented these methods to perform something
reasonable.
A problem with implementing censoring on filelog is that
it assumes filelog is a revlog. Upcoming work to formalize
the filelog interface will make this not true.
Furthermore, the censoring logic is security-sensitive. I
think action-at-a-distance with custom implementation of core
revlog APIs in derived classes is a bit dangerous. I think at
a minimum the censor logic should live in revlog.py.
I was tempted to created a "censored revlog" class that
basically pulled these methods out of filelog. But, I wasn't
a huge fan of overriding core methods in child classes. A
reason to do that would be performance. However, the censoring
code only comes into play when:
* hash verification fails
* delta generation
* applying deltas from changegroups
The new code is conditional on an instance attribute. So the
overhead for running the censored code when the revlog isn't
censorable is an attribute lookup. All of these operations are
at least a magnitude slower than a Python attribute lookup. So
there shouldn't be a performance concern.
Differential Revision: https://phab.mercurial-scm.org/D3151
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 05 Apr 2018 18:22:35 -0700] rev 37442
revlog: move parsemeta() and packmeta() from filelog (API)
filelog.parsemeta() and filelog.packmeta() are used to decode
and encode metadata for file copies and censor.
An upcoming commit will move the core logic for censoring revlogs
into revlog.py. This would create a cycle between revlog.py and
filelog.py. So we move these metadata functions to revlog.py.
.. api::
filelog.parsemeta() and filelog.packmeta() have been moved to
the revlog module.
Differential Revision: https://phab.mercurial-scm.org/D3150
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 05 Apr 2018 15:18:23 -0700] rev 37441
filelog: declare that filelog implements a storage interface
Now that we have a declared interface, let's declare that filelog
implements it.
Tests have been added that confirm the object conforms to the
interface.
The existing interface checks verify there are no extra public
attributes outside the declared interface. filelog has several
extra attributes. So we added a mechanism to suppress this check.
The goal is to modify the filelog class so we can drop this check.
Differential Revision: https://phab.mercurial-scm.org/D3149
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 05 Apr 2018 15:09:41 -0700] rev 37440
repository: define existing interface for file storage
Now that we have mostly successfully implemented an alternate
storage backend for files data, let's start to define the
interface for it!
This commit takes the mostly-working interface as defined by the
simple store repo and codifies it as the file storage interface.
The interface has been split into its logical components:
* index metadata
* fulltext data
* mutation
* everything else
I don't consider the existing interface to be great. But it will
help to have it more formally defined so we can start chipping away
at refactoring it.
Differential Revision: https://phab.mercurial-scm.org/D3148
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 05 Apr 2018 11:16:54 -0700] rev 37439
tests: run some largefiles and lfs tests with simple store
Now that the simple store handles flags properly, a handful of
the largefiles and lfs tests pass!
Differential Revision: https://phab.mercurial-scm.org/D3147
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 21:27:02 -0700] rev 37438
commands: don't violate storage abstractions in `manifest --all`
Previously, we asked the store to emit its data files. For modern
repos, this would use fncache to resolve the set of files then would
stat() each file. For my copy of the mozilla-unified repository, this
took 3.3-10s depending on the state of my filesystem cache to render
449,790 items.
The previous behavior was a massive layering violation because it
assumed tracked files would have specific filenames in specific
directories. Alternate storage backends would violate this assumption.
The new behavior scans the changelog entries for the set of files
changed by each commit. It aggregates them into a set and then
sorts and prints the result. This reliably takes ~16.3s on my
machine. ~80% of the time is spent in zlib decompression.
The performance regression is unfortunate. If we want to claw it
back, we can create a proper storage API to query for the set of
tracked files. I'm not opposed to doing that. But I'm in no hurry
because I suspect ~0 people care about the performance of
`hg manifest --all`.
.. perf::
`hg manifest --all` is likely slower due to changing its
implementation to respect storage interface boundaries. If you
are impacted by this regression in a meaningful way, please make
noise on the development mailing list and it can be dealt with.
Differential Revision: https://phab.mercurial-scm.org/D3119
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 21:09:47 -0700] rev 37437
commands: document the layering violation in `manifest --all`
This commit fixes the last test failures when using the simple
store extension!
It turns out that `hg manifest --all` locks the repo and scans for
revlogs. This feature was added by 71938479eff9 in 2011. I am
debating changing the behavior. But that can occur in another
commit.
As part of debugging this, I realized that test-manifest.t is the
only meaningful tester of `hg manifest --all` and that test was
improperly disabled when bundlerepos aren't supported. The test is
testing manifest behavior, not whether you can `hg pull` from a
bundle. So I changed the test to `hg unbundle` instead.
FWIW, I wasted a non-trivial amount of time tracking down this
failure. I thought the issue involved Git, which is why I refactored
the test to be more deterministic. Never in my mind would I have
guessed that code in `hg manifest` would scan revlogs. I should have
looked there to begin with. Doh.
Differential Revision: https://phab.mercurial-scm.org/D3118
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 19:17:22 -0700] rev 37436
simplestore: correctly implement flag processors
There were a couple of bugs around the implementation of
flags processing with the simple store. After these changes,
test-flagprocessor.t now passes!
test-flagprocessor.t was also updated to include explicit test
coverage that pushed data is as expected on the server.
The test extension used by test-flagprocessor.t has been updated
so it monkeypatches the object returned from repo.file() instead
of monkeypatching filelog.filelog. This allows it to work with
extensions that return custom types from repo.file().
The monkeypatching is rather hacky and probably is performance
prohibitive for real repos. We should probably come up with a
better mechanism for registering flag processors so monkeypatching
isn't needed.
Differential Revision: https://phab.mercurial-scm.org/D3116
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 17:40:09 -0700] rev 37435
tests: `hg init` after resetting HGRCPATH
Otherwise extensions loaded via --extra-config-opt could prevent
access to the repo by introducing requirements file. This does mean
that custom extensions loaded in this way won't impact this test.
I'm fine with that.
Differential Revision: https://phab.mercurial-scm.org/D3115
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 17:33:59 -0700] rev 37434
tests: work around potential repo incompatibility
test-run-tests.t invokes run-tests.py. But custom extensions providing
new repo requirements may be in play and may not get inherited by the
new run-tests.py. We ensure our repo is created with a vanilla config
to mitigate extension-caused badness.
Differential Revision: https://phab.mercurial-scm.org/D3114
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 17:29:02 -0700] rev 37433
tests: disable test-keyword.t with simple store
The keyword extension is hooking into repo.file() and defining its
own filelog class. It will likely require a more formal storage
interface before keywords are usable with alternate storage backends.
Differential Revision: https://phab.mercurial-scm.org/D3113
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 17:12:00 -0700] rev 37432
tests: conditionalize test-treemanifest.t
Parts of the test were assuming the use of revlogs with fnstore
path encoding.
Other parts of the test assumed we could create repos with different
store encodings and that stream clone bundles worked.
Make all of this conditional on running a revlog repo.
Differential Revision: https://phab.mercurial-scm.org/D3112
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 17:02:54 -0700] rev 37431
tests: use unbundle in test-symlink-os-yes-fs-no.py
The test (which should probably be rewritten as a .t test - the
test was initially authored in 2009 and this may have predated
some test harness features allowing us to implement it as a .t
test) is verifying symlink behavior with regards to working
directory operations. How it pulls bundle data into a repo is
not relevant. So we can switch from pull to unbundle so we can
support environments where bundlerepos don't work.
Differential Revision: https://phab.mercurial-scm.org/D3111
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 16:49:22 -0700] rev 37430
tests: disable `hg clone --stream` test with simple store
We mass disabled stream clone tests in a previous commit. Looks like
one was missed.
Differential Revision: https://phab.mercurial-scm.org/D3110
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 16:47:06 -0700] rev 37429
tests: use `hg unbundle` in test-setdiscovery.t
This is testing how discovery between 2 well-defined repos works,
not that `hg pull` works with bundles. Switch to `hg unbundle` so it
works in environments where bundlerepos aren't supported.
Differential Revision: https://phab.mercurial-scm.org/D3109
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 16:41:43 -0700] rev 37428
tests: require revlog store for test-verify.t
This tests is doing tons of revlog-y things. It should be
possible to make verification work across multiple stores. But it
will likely have to wait until we have better abstractions
in place.
Differential Revision: https://phab.mercurial-scm.org/D3108
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 16:32:51 -0700] rev 37427
tests: conditionalize test-commandserver.t based on extra extensions
If running with extra extensions we get an output difference.
Differential Revision: https://phab.mercurial-scm.org/D3107
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 16:29:19 -0700] rev 37426
tests: conditionalize test-bundle.t
This test is massive and could probably be split up.
This change essentially requires the revlog store for stream clone
tests and support for bundlerepos for various tests operating
on bundle files.
Differential Revision: https://phab.mercurial-scm.org/D3106
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 16:16:42 -0700] rev 37425
tests: require revlog store with test-repair-strip.t
This test is doing a number of low-level things with revlogs.
Mark it as requiring the revlog store.
Differential Revision: https://phab.mercurial-scm.org/D3105
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 17:53:45 -0700] rev 37424
simplestore: back up index when adding a revision
This ensures that transaction rollback reverts the index to its
previous state. This fixed a few test failures due to `hg verify`
complaining about a reference to an undefined changeset revision.
Differential Revision: https://phab.mercurial-scm.org/D3104
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 16:00:50 -0700] rev 37423
tests: disable shallow narrow tests with simple store
I think these are failing because of issues with flags processing
in the simple store. Let's revisit this later.
Differential Revision: https://phab.mercurial-scm.org/D3103
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 15:57:49 -0700] rev 37422
tests: skip test-hgweb-bundle.t if we don't support bundlerepos
Differential Revision: https://phab.mercurial-scm.org/D3102
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 15:31:25 -0700] rev 37421
tests: disable test-audit-path.t with simple store
The simple store is using a primitive filename encoding scheme.
Many tests in this file fail. So let's disable it for now.
At some point, we should support a mode where we can run tests
without fnstore or dotencode and have all tests assuming use of
these filename encoding mechanisms be conditional on their usage.
Differential Revision: https://phab.mercurial-scm.org/D3101
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 14:43:19 -0700] rev 37420
tests: port test-convert-filemap.t to simple store
Differential Revision: https://phab.mercurial-scm.org/D3100
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 14:37:53 -0700] rev 37419
tests: disable test-static-http.t with simple store
It /should/ be possible to support repos statically hosted on HTTP
servers with alternate stores. But it's more trouble than it is worth
right now. Let's just disable the test.
Differential Revision: https://phab.mercurial-scm.org/D3099
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 14:25:20 -0700] rev 37418
tests: don't drop global hgrc
Various tests are failing when the simple store extension is loaded
because the test overrides HGRCPATH and preempts loading of extensions
that were injected via --extra-config-opt.
In most cases, it is acceptable to always load the global HGRCPATH.
So this commit changes a couple of tests so the global HGRCPATH
is still pulled in.
Differential Revision: https://phab.mercurial-scm.org/D3098
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 14:11:43 -0700] rev 37417
verify: allow suppressing warnings about extra files
The verifier issues warnings when the set of files in .hg/store
doesn't align with the set of files that are advertised via
repo.file(f).files() for all files seen in ctx.files() changelog
traversal.
This logic is reasonable for a default implementation. But some
stores may have extra files whose presence is harmless. Or those
stores may not have the same transaction rollback semantics that
unlink files as other stores.
This commit adds support for disabling the warning for orphaned
files.
The simple store extension has been taught to set this flag.
Differential Revision: https://phab.mercurial-scm.org/D3097
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 14:04:18 -0700] rev 37416
tests: conditionalize tests for various repo features
Working my down the long tail of test failures due to the simple
store.
We're now down to 38 failures with the simple store.
Differential Revision: https://phab.mercurial-scm.org/D3096
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 14:09:02 -0700] rev 37415
simplestore: use a custom store for the simple store repo
Before, we used the default store, which was based on fncache
and dotencode. After attempting to port tests to work with the
simple store, I realized that fncache was more trouble than it is
worth.
This commit implements a proper store type for the simple repo -
one that isn't based off fncache.
This causes a number of new test failures because of tests
expecting the full fncache store filename encoding. I may
extend the store format in a subsequent commit to take the
filename encoding parts of fncache that we can take
(basically everything except hash encoding, since that isn't
reversible). But for now, let's use encoded store.
As part of this, we implement proper requirements support for
repos created with the simple store. This should have been
done from the beginning, as a requirement is needed to lock
out clients that don't understand a storage format.
A new hghave feature advertising the presence of fncache in repos
has been added. Most tests touching the fncache are now conditional
on that feature.
Other tests have added the optional repo requirement to output.
Differential Revision: https://phab.mercurial-scm.org/D3095
Joerg Sonnenberger <joerg@bec.de> [Tue, 27 Feb 2018 02:37:31 +0100] rev 37414
wireproto: allow direct stream processing for unbundle
Introduce a new option server.streamunbundle which starts a transaction
immediately to apply a bundle instead of writing it to a temporary file
first. This side steps the need for a large tmp directory at the cost of
preventing concurrent pushes. This is a reasonable trade-off for many
setups as concurrent pushes for the main branch at least are disallowed
anyway. The option defaults to off to preserve existing behavior.
Change the wireproto interface to provide a generator for reading the
payload and make callers responsible for consuming all data.
Differential Revision: https://phab.mercurial-scm.org/D2470
Joerg Sonnenberger <joerg@bec.de> [Fri, 06 Apr 2018 22:22:19 +0200] rev 37413
wireproto: send server capabilities in canonical order
Differential Revision: https://phab.mercurial-scm.org/D3171
Joerg Sonnenberger <joerg@bec.de> [Fri, 06 Apr 2018 22:16:11 +0200] rev 37412
wireproto: don't special case bundlecaps, but sort all scsv arguments
Differential Revision: https://phab.mercurial-scm.org/D3170
Joerg Sonnenberger <joerg@bec.de> [Fri, 06 Apr 2018 21:50:01 +0200] rev 37411
wireproto: turn client capabilities into sets, sorted on the wire
Differential Revision: https://phab.mercurial-scm.org/D3169
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 13:43:52 -0700] rev 37410
verify: drop "revlog" from warning message
Not all stores may be backed by revlogs. Switch to a more generic
error message.
Differential Revision: https://phab.mercurial-scm.org/D3094
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 10:16:08 -0700] rev 37409
store: make file filtering during walk configurable
Previously, the walking mechanism assumed the use of revlogs for
storage.
Making the file filter configurable will enable custom stores
to override _walk() so it recognizes additional files.
Differential Revision: https://phab.mercurial-scm.org/D3093
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 09:31:19 -0700] rev 37408
simplestore: shore up lookup errors
When revisions or nodes can't be resolved, we're expected to raise
an error.LookupError. When I ported code from revlog.py, I failed
to realize that "LookupError" in that module is aliased to
error.LookupError. I thought we were using the builtin LookupError
instead.
This commit switches us to error.LookupError. It also fixes
rev() to raise error.LookupError instead of KeyError.
Differential Revision: https://phab.mercurial-scm.org/D3092
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 09:57:21 -0700] rev 37407
tests: extract dumprevlog tests to own file
And mark the test as requiring the revlog store.
Differential Revision: https://phab.mercurial-scm.org/D3091
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 09:21:11 -0700] rev 37406
tests: skip test-censor.t when using simple store
Censor assumes the use of revlogs.
Proper censor support for non-revlog storage will require a better
abstraction in the storage layer to remove data. Let's skip the
censor tests until we have something in place.
Differential Revision: https://phab.mercurial-scm.org/D3090
Yuya Nishihara <yuya@tcha.org> [Mon, 02 Apr 2018 21:54:12 +0900] rev 37405
templater: deduplicate iterator of overlay mappings
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 21:01:23 +0900] rev 37404
templater: complain about invalid application of '%' operator (BC)
Before, '{x % y % z ...}' was silently evaluated as '{x % y}'. We no longer
need this hack since the web template bugs was fixed by earlier patches.
At this point, the error message may contain '<generator *>', which will
be fixed later.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 21:18:57 +0900] rev 37403
hgweb: fix type of {nav} keyword when linerange filter is active
Spotted by making '%' operator check the operand type more strictly. Before,
"{nav}" would be evaluated to a lazy "" (empty string) and {"" % x} would
yield one empty string.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 21:46:20 +0900] rev 37402
hgweb: fix {diff} expansion in JSON template
The same sort of bug as the previous patch. In this case, I decided to fix
the JSON template since we aren't ready to fix the {diff} keyword without BC.
I'll rework it later.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 21:35:43 +0900] rev 37401
hgweb: fix summary {tags} and {shortlog} to not forcibly expand template
The same sort of bug as the previous patch. In this case, JSON template was
wrong.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 20:51:39 +0900] rev 37400
hgweb: fix search {entries} to not return results of template expansion
"{entries%changelogentry}" in raw/search.tmpl was utterly wrong because
"{entries}" here was a generator yielding results of template expansion.
That's why we have a weird hack in runmap(), which I'm going to get rid of.
https://www.mercurial-scm.org/repo/hg/file/4.5.2/mercurial/templater.py#l469
We have two choices:
a) drop "%changelogentry" from raw/search.tmpl
b) fix "{entries}" to yield mappings
I take (b) because that's what the other log-like "{entries}" do. The
"entries" keyword is wrapped by mappinggenerator so "{entries}" without
"%searchentry" still works.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 22:47:02 +0900] rev 37399
templater: add class representing a nested mappings
The mappinggenerator class is necessary to fix hgweb bugs without BC. The
mappinglist is for nested formatter items. They are similar, so factored
out the base class. The mappinglist could be implemented by using the
mappinggenerator, but we'll probably need a direct access to the raw list,
so they are implemented as separate classes.
Note that tovalue() isn't conforming to the spec yet in that it may return
a list of dicts containing unprintable resources. This problem will be
fixed later.
Tests will be added by subsequent patches.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 22:56:49 +0900] rev 37398
templater: add function that expands internal literal templates
This will be used when rendering nested formatter items with the default
template, e.g.
fm.nested('parents', tmpl='{rev}:{node|formatnode}', sep=' ')
^^^^^^^^^^^^^^^^^^^^^^^
the default item template
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 11:28:26 -0700] rev 37397
bookmarks: use isrevsymbol() for detecting collision with existing symbol
Differential Revision: https://phab.mercurial-scm.org/D3167
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 11:26:50 -0700] rev 37396
debugwhyunstable: add support for revsets
Differential Revision: https://phab.mercurial-scm.org/D3166
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 10:46:24 -0700] rev 37395
convert: look up branch only among branches
repo[<branch name>] can find something that's not a branch, which is
not good.
Differential Revision: https://phab.mercurial-scm.org/D3161
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 16:10:52 -0700] rev 37394
convert: remove unused/unnecessary variable "parentctx"
Differential Revision: https://phab.mercurial-scm.org/D3160
Joerg Sonnenberger <joerg@bec.de> [Sat, 24 Mar 2018 17:57:22 +0100] rev 37393
wireproto: provide accessors for client capabilities
For HTTP, this refactors the existing logic, including the parsing of
the compression engine capability.
For SSH, this adds a ssh-only capability "protocaps" and a command for
informing the server on what the client supports. Since SSH is stateful,
keep track of the capabilities in the server instance.
Differential Revision: https://phab.mercurial-scm.org/D1944
Gábor Stefanik <gabor.stefanik@nng.com> [Thu, 05 Apr 2018 17:51:10 +0200] rev 37392
copies: clean up _related logic
The limit parameter was never actually used, since the only way the 4th case
could be reached was if f1r and f2r converged. The new code makes this clear,
and additionally reduces the conditional block to just 3 cases.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 06 Apr 2018 09:34:44 -0700] rev 37391
context: stop catching RepoLookupError from namespace.singlenode()
As pointed out by Yuya, the RepoLookupError was there for catching
errors from repo.branchtip(). However, since 885c0290f7d5 (localrepo:
add ignoremissing parameter to branchtip, 2014-10-16), that should no
longer happen. I think it should now be an error if a namespace raises
a RepoLookupError, so we propagate the exception up and and make it
easy to fix, rather than trying to interpret the changeid as nodeid
prefix and raise a general "unknown revision '...'" error.
I also don't think we should catch FilteredLookupError and LookupError
from the changelog.rev() call, for the same reason as above: If a
namespace returns a node that doesn't exist, we should provide a more
helpful exception than "unknown revision '...'".
Differential Revision: https://phab.mercurial-scm.org/D3145
Boris Feld <boris.feld@octobus.net> [Thu, 05 Apr 2018 18:02:42 +0200] rev 37390
histedit: simplify desthistedit
Instead of sorting the revset and take the first one, take the minimum
revision.
Differential Revision: https://phab.mercurial-scm.org/D3137
Boris Feld <boris.feld@octobus.net> [Thu, 05 Apr 2018 17:58:58 +0200] rev 37389
stack: follow-up on the stack revset
Follow good-practice for defining the stack revset.
Differential Revision: https://phab.mercurial-scm.org/D3136
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 23:23:48 -0700] rev 37388
githelp: use revsymbol() for looking up symbol
I don't know if we should be using revsingle() here, so I was
conservative and switched to revsymbol().
Differential Revision: https://phab.mercurial-scm.org/D3156
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 21:32:59 -0700] rev 37387
hgweb: use revsymbol() to determine if query is a revision
repo.__getitem__ is about to get dumber.
Differential Revision: https://phab.mercurial-scm.org/D3155
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 14:03:33 -0700] rev 37386
context: make repo[<filtered binary nodeid>] match node
If you pass in a binary nodeid of a filtered node to repo.__getitem__,
it would run through this code:
try:
self._node = changeid
self._rev = repo.changelog.rev(changeid)
return
except error.FilteredLookupError:
raise
except LookupError:
pass
However, repo.changelog.rev() would raise a FilteredLookupError, not
FilteredRepoLookupError. Instead, we would hit the "except
LookupError" and continue, trying to interpret the nodeid as a
bookmark etc. The end result would be an error like this:
abort: unknown revision 'ddadbd7c40ef8b8ad6d0d01a7a842092fc431798'!
After this patch, it would instead be:
abort: 00changelog.i@ddadbd7c40ef8b8ad6d0d01a7a842092fc431798: filtered node!
This only happens when we get a binary nodeid, which means it's not
string directly from the user, so it would be a programming error if
it happened. It's therefore a little hard to test (I checked
test-context.py, but it doesn't use obsmarkers).
It looks like this has been wrong ever since dc25ed84bee8 (changectx:
issue a FilteredRepoLookupError when applicable, 2014-10-15).
Differential Revision: https://phab.mercurial-scm.org/D3144
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 00:04:09 -0700] rev 37385
context: move handling of filtering error to revsymbol() (API)
When changectx's constructor runs into various Filtered*Error, it
creates an exception with a hint about using --hidden. This only makes
sense when the revision was provided by the user (if we get e.g. a
FilteredLookupError from repo[p1], then it's instead a programming
error). Thus, I'm moving the handling into revsymbol(). Also changed
"unfilteredrepo[changeid]" to "revsymbol(unfilteredrepo, changeid)" as
part of the move.
Differential Revision: https://phab.mercurial-scm.org/D3143
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 17:56:24 +0530] rev 37384
py3: whitelist 13 new passing tests
Some tests were passing before the series too.
Differential Revision: https://phab.mercurial-scm.org/D3133
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 16:47:44 +0530] rev 37383
py3: convert user value to bytes by b'' prefix
This makes test-journal* pass on Python 3.5.
# skip-blame beacuse just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D3125
Augie Fackler <augie@google.com> [Thu, 05 Apr 2018 10:13:01 -0400] rev 37382
util: whitelist apfs for hardlink support
Seems to work fine for me.
Differential Revision: https://phab.mercurial-scm.org/D3134
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 11:33:36 -0700] rev 37381
mq: avoid a silly conversion from binary nodeid to hex
We generally deal with binary nodeids in code. In this case the hex
nodeid was passed to strip.checksubstate() where it was passed to
repo.__getitem__, which of course accepts a binary nodeid.
Differential Revision: https://phab.mercurial-scm.org/D3141
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 12:36:44 -0700] rev 37380
narrow: remove unused "cacheprop" stuff
This is unused since D3046. I didn't even notice it then, but tests
still pass so I hope it's still handled safely.
Differential Revision: https://phab.mercurial-scm.org/D3142
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 14:11:43 -0700] rev 37379
tests: disable infinitepush tests for simple store
Infinitepush relies heavily on bundles and bundlerepo. As such,
it won't be easy to make compatible with alternate storage
backends at this time.
Let's disable the tests when running with our simple store.
Differential Revision: https://phab.mercurial-scm.org/D3062
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 11:01:42 -0700] rev 37378
rebase: remove unnecessary and incorrect handling of nullid
We used to write a nullid as hex to the rebase state file and
interpret it as the "todo" state (value -1). However, when reading it,
we compared the string value to (binary) nullid, which would of course
not match. AFAICT, it still worked because when the read nodeid did
not match nullid (which, again, it didn't), we'd use the normal path
which did repo[<hex nullid>].rev(), and that also happens to return
-1. It seems to have been this way ever since 9972758ab4c5 (rebase:
handle revtodo as a special value when storing/restoring state,
2014-12-02).
Differential Revision: https://phab.mercurial-scm.org/D3140
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 10:13:45 -0700] rev 37377
rebase: convert "oldrev" to revnum earlier
It was done in 3 places before, now just 1.
Differential Revision: https://phab.mercurial-scm.org/D3139
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 10:12:10 -0700] rev 37376
rebase: make "destnode" consistently a revnum and rename it to "destrev"
Differential Revision: https://phab.mercurial-scm.org/D3138
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 08:28:12 -0700] rev 37375
bookmarks: drop always-None argument from calculateupdate()
Thanks to Yuya for noticing.
Differential Revision: https://phab.mercurial-scm.org/D3135
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 09:58:16 -0700] rev 37374
narrow: move manifestlog overrides to core
With this and the previous patch, I couldn't measure any significant
difference from `hg files -r .` in a FireFox repo with 65k files. I
tried with both a flat-manifest and a tree-manifest version of
it. Neither had the narrow extension enabled.
Differential Revision: https://phab.mercurial-scm.org/D3046
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 23:46:04 -0700] rev 37373
narrow: move manifestrevlog overrides to core
Differential Revision: https://phab.mercurial-scm.org/D3045
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 00:13:02 -0700] rev 37372
narrow: move excludeddir and related classes to core
Differential Revision: https://phab.mercurial-scm.org/D3044
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 17:29:32 +0530] rev 37371
py3: add missing b'' prefix in mdiff.py
Transformer won't add b'' prefix because that value is already have a r''
prefix.
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D3132
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 17:15:52 +0530] rev 37370
py3: fix error string with bytestr() on repr()d value
Differential Revision: https://phab.mercurial-scm.org/D3131
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 17:14:11 +0530] rev 37369
py3: suppress output from f.write() function class
We need to suppress them because the output is not present on Python 2
Differential Revision: https://phab.mercurial-scm.org/D3130
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 17:13:09 +0530] rev 37368
py3: use pycompat.byteskwargs() in tests/autodiff.py
Differential Revision: https://phab.mercurial-scm.org/D3129
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 17:00:15 +0530] rev 37367
py3: suppress the output of open() using `and None`
This patch suppresses the output of open() on Python 3 as it does not return any
output on Python 2.
This makes test-diffstat.t pass on Python 3.5
Differential Revision: https://phab.mercurial-scm.org/D3128
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 16:56:34 +0530] rev 37366
py3: return bytes from util.removeauth()
util.hidepassword() also returns bytes and we should deal in bytes as much as
possible.
This makes test-logexchange.t pass on Python 3.5
Differential Revision: https://phab.mercurial-scm.org/D3127
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 16:54:56 +0530] rev 37365
py3: use bytes instead of str in instance()
We deal internally with bytes, so we should check whether the remote is a bytes
or not.
Differential Revision: https://phab.mercurial-scm.org/D3126
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 05 Apr 2018 16:46:34 +0530] rev 37364
py3: use pycompat.bytestr in test-journal.t
Differential Revision: https://phab.mercurial-scm.org/D3124
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 23:55:47 -0700] rev 37363
hgweb: don't include hidden revisions in /filelog/ view
This is a very crude way of doing it, but it seems to be working well
enough. The number of entries on the page won't be the usual maximum
number per page, but this is good enough for me.
Differential Revision: https://phab.mercurial-scm.org/D3122
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 00:00:48 -0700] rev 37362
tests: show that hgweb contains hidden revisions in /filelog/ view
Note that these entries contain the summary line of the hidden commit,
which is a bit a privacy issue to display. Also note that the links
from the entries take you to a page that says:
An error occurred while processing your request:
filtered revision '2d32257e1109' (not in 'served' subset)
Differential Revision: https://phab.mercurial-scm.org/D3121
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 09:03:17 -0700] rev 37361
convert: use repo.lookup() for converting to nodeid
This is a list of revs that come from the CLI (opts['rev']). Perhaps
we should allow any revset, but I'll leave that for someone else to
improve if they care.
Differential Revision: https://phab.mercurial-scm.org/D3089
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 22:36:21 -0700] rev 37360
extdatasource: use revsymbol() for converting to node
It's unclear what we want to support here. Perhaps it should just be
nodeids? It doesn't make much sense to share revnums between repos,
and bookmarks are probably too unstable. I used scmutil.revsymbol() to
preserve the current behavior. We can change later if we want to.
Differential Revision: https://phab.mercurial-scm.org/D3088
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 15:13:32 -0700] rev 37359
bookmarks: calculateupdate() returns a bookmark, not a rev
This changes the inaccurate/unclear documentation and also changes the
code so "node" now contains a binary nodeid.
Differential Revision: https://phab.mercurial-scm.org/D3087
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 10:13:08 -0700] rev 37358
pull: pass rev to check out as integer to postincoming()
I don't know if there's ever been a need for it to be a string (it's
been like that since 02f40b2ece3f (commands: use rev from remote repo
when updating as part of a pull, 2009-10-21)). I'm soon going to
require it to be an integer, so let's fix this first.
Differential Revision: https://phab.mercurial-scm.org/D3086
Martin von Zweigbergk <martinvonz@google.com> [Thu, 05 Apr 2018 14:21:37 +0530] rev 37357
children: support specifying revision by revset
Same reason as the previous patch.
Differential Revision: https://phab.mercurial-scm.org/D3085
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 15:08:26 -0700] rev 37356
heads: add support for specifying branches by revset
Before this commit, e.g. "hg heads .^" would fail with:
abort: unknown revision '.^'!
Like the previous patch, I don't care about the command itself (I
don't think I had ever used it before), I'm just cleaning up uses of
repo[<string>].
Differential Revision: https://phab.mercurial-scm.org/D3084
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 15:06:32 -0700] rev 37355
perf: make perfmanifest and perfnodelookup work with revsets
They were using repo[rev], which only works with a single symbol
(e.g. "." or "my-bookmark"), not general revsets. Switch them to
scmutil.revsingle() so they can also be used with e.g. ".^".
I don't actually care about these commands, but I want to remove uses
of repo[<string>].
Differential Revision: https://phab.mercurial-scm.org/D3083
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 23:47:01 -0700] rev 37354
context: use revsymbol() in "merge.preferancestor" code
Differential Revision: https://phab.mercurial-scm.org/D3082
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 15:11:43 -0700] rev 37353
wireproto: use repo.lookup() for lookup command
I'm trying to reduce use of repo[<string>] and this seems like an
obvious place to use repo.lookup().
Differential Revision: https://phab.mercurial-scm.org/D3081
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 14:57:58 -0700] rev 37352
localrepo: use revsymbol in lookupbranch() too
lookupbranch() takes a string that comes from the CLI, so
scmutil.revsymbol() is appropriate for looking it up.
Differential Revision: https://phab.mercurial-scm.org/D3080
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 14:31:09 -0700] rev 37351
localrepo: drop "remote" argument from lookupbranch() (API)
According to `hg grep --all lookupbranch`, the "remote" argument has
never been used ever since it was introduced in ca739acf1a98
(commands: add more robust support for 'hg log -b' (issue2078),
2010-04-12).
Differential Revision: https://phab.mercurial-scm.org/D3079
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 22:48:32 -0700] rev 37350
revset: use revsymbol() for checking if a symbol is valid
Differential Revision: https://phab.mercurial-scm.org/D3078
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 15:08:14 -0700] rev 37349
tests: disable tests for advanced clone features with simple store
There are a handful of test failures in the simple store with
regards to stream clones. Fixing them will require a lot of
conditional output. Disabling the tests wholesale is easier
at this juncture.
Differential Revision: https://phab.mercurial-scm.org/D3064
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 14:24:14 -0700] rev 37348
tests: skip largefiles and lfs tests when using simple store
Getting these tests to pass is more work than it is worth right
now. Let's punt on it.
Differential Revision: https://phab.mercurial-scm.org/D3063
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 18:23:33 -0700] rev 37347
tests: conditionalize test output for simple store
Differential Revision: https://phab.mercurial-scm.org/D3061
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 11:44:38 -0700] rev 37346
tests: disallow using simple store repo with bundlerepo
bundlerepo is... going to be difficult to port to an alternate
store because it assumes revlogs for storage and essentially
overlays the contents of a bundle onto a fake revlog-like
primitive. It will be a good test case for our eventual new
storage interface.
Refactoring bundlerepo to make it work with non-revlog storage is
going to be a bit of work. So for now, let's refuse to use the
simple store repo when a bundlerepo is in play.
A new test requirement advertising support for treating bundle
files as repo instances has been added. Some tests have been
made conditional on this feature. Additional tests will be
annotated in subsequent commits.
Having positive opt-in to repo features will be simpler in the
long run because it will allow multiple storage backends to
declare feature support and we won't have to annotate each test
with the set of repo backends that are supported. Again, we'll
probably want better integration between repo features and
tests. But this is the easiest we can do at the moment.
Differential Revision: https://phab.mercurial-scm.org/D3060
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 11:31:35 -0700] rev 37345
tests: disable test-revlog-v2 when using simple store
Because the simple store has nothing to do with revlogs.
Differential Revision: https://phab.mercurial-scm.org/D3056
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 18:16:04 -0700] rev 37344
tests: skip filelog damage tests when not using revlogs
Differential Revision: https://phab.mercurial-scm.org/D3042
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 10:04:30 -0700] rev 37343
tests: skip some tests when using simple store
generaldelta is a revlog implementation detail. We don't need to
test it with the simple store.
union repos are heavily revlog based. It should be possible to
run them with alternate stores. But it's not worth the trouble
at this juncture.
Differential Revision: https://phab.mercurial-scm.org/D3040
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 13:21:34 -0700] rev 37342
tests: conditionalize tests based on presence of custom extensions
The test harness supports injecting extensions via --extra-config-opt.
However, if you do this, various tests that print state about loaded
extensions fail.
This commit teaches the test harness to recognize when custom
extensions are loaded so that tests can use feature sniffing to
conditionalize tests based on that.
Differential Revision: https://phab.mercurial-scm.org/D3039
Augie Fackler <augie@google.com> [Wed, 04 Apr 2018 12:16:50 -0700] rev 37341
simplestorerepo: avoid shadowing dict in list comprehension over dict
Caught by pyflakes.
Differential Revision: https://phab.mercurial-scm.org/D3077
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 09:54:20 -0700] rev 37340
narrow: pass node into revlog.revision()
This is one of the few (possibly only) places where we pass an int in
for filelogs. Other revlogs (notably changelog) are very heavy on int
usage. But filelogs are surprisingly node centric. I'd like to
formalize the interface around the use of nodes (at least for
filelogs). So let's switch to a node.
We can't inline revlog.node() because of a check-code rule. I think
that rule is suspect. But it may be for performance reasons with
changelog code. I'd rather not touch it at this time.
Differential Revision: https://phab.mercurial-scm.org/D3043
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 10:15:21 -0700] rev 37339
changegroup: remove "revlog" from error message
The previous message leaked an implementation detail.
Differential Revision: https://phab.mercurial-scm.org/D3041
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 18:15:24 -0700] rev 37338
tests: conditionalize tests based on presence of revlogs for files
~85 tests don't like our non-revlog file store for various reasons.
This commit introduces hghave functionality for declaring and querying
repository features. By default, we assume repositories have
revlog-based file storage. But if the HGREPOFEATURES environment
variable is set, we can override the default set of repository
features. If you run the test harness with our simplestorerepo
extension and an environment variable set to the proper value, you
can override the hghave defaults to agree with simplestorerepo's
version of reality.
Various tests have been modified so behavior dependent on revlog-based
file storage is marked as such.
This fixes a handful of test failures with our custom file storage
extension. But dozens remain. The point of this commit is to demonstrate
how tests will need to be modified to account for custom storage
implementations.
TBH, I'm not convinced hghave is the proper layer for repository
feature detection. I /think/ we'll eventually want something in
run-tests.py itself. But that would require inventing a new primitive
in the test harness. This is all very alpha at the moment. So I think
hghave is an acceptable place to hang this feature detection. I think
the right time to be thinking about integrating this into run-tests.py
is *after* we have a stable alternate storage implementation in core.
For now, let's try to make progress towards the idea of an alternate
storage backend.
Differential Revision: https://phab.mercurial-scm.org/D3030
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 04 Apr 2018 11:37:07 -0700] rev 37337
tests: add test extension implementing custom filelog storage
In order to better support partial clones, we'll need alternate
repository storage mechanisms that aren't based on revlogs.
Today, the interface for repository storage isn't very well defined.
And there are various layering violations and assumptions made
throughout the code that storage is backed by revlogs.
In order to support alternate storage mechanisms, we'll need to
formally declare and adhere to interfaces for storage. This will
be a long, arduous process.
This commit creates an extension that implements non-revlog storage
for files. It defines a custom type that quacks like the existing
revlog/filelog API but isn't backed by a revlog. The backing storage
is - for simplicity reasons - a CBOR index and per-node files
representing fulltext data.
The localrepository class is modified so file(f) returns instances of
this class instead of filelog instances.
The purpose of this extension is to tease out what the actual filelog
interface is - based on running the test harness - so we can formalize
that interface and then implement a *real* alternate storage backend.
Using `run-tests.py --extra-config-opt` to run the test harness
with this extension enabled yields 83 failures out of 634 ran
tests.
The most common test failures are due to:
* Issues with `hg verify`
* LFS and largefiles (probably flags processing related)
* Narrow.
* Any test touching or inspecting individual filelog paths.
* help and error output that is confused by the presence of an
extension.
* `hg debug*` commands doing low-level, revlog-y things.
An 88% pass rate is pretty good for an initial implementation if you
ask me!
There is a bit of duplicate code in the new extension. That's by
design: a point of this code is to tease out dependencies on revlog.
That being said, there is opportunity to consolidate code by moving
things out of the revlog API. For example, DAG traversal operations
don't necessarily need to be implemented at the storage level. (Although
for performance reasons they probably do.) Once we have a more
well-defined interface, we could probably define the default
implementations in terms of the base interface, pull those in via
class inheritance, and have implementations override with faster
versions if they so choose. (Or something like that.) But for now,
the duplicate code should be acceptable.
Differential Revision: https://phab.mercurial-scm.org/D3029
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 13:56:09 -0700] rev 37336
tests: use `hg unbundle` instead of `hg pull` in some tests
`hg pull <bundle>` uses the special "bundlerepo" repository. The
bundlerepo code makes many assumptions about the storage of
repositories. It will be difficult to teach bundlerepo to use
non-revlog storage before a better storage interface is established.
Many test failures using our "simple store" are related to
bundlerepo: the simple store just isn't compatible with bundlerepo
because of storage assumptions in bundlerepo.
In order to mitigate the impact of bundlerepo on our code base,
this commit changes various tests to use `hg unbundle` instead
of `hg pull`. This bypasses the bundlerepo code.
Tests exercising exchange functionality have not been altered, as
they should be using `hg pull` and going through the bundlerepo
code paths.
Differential Revision: https://phab.mercurial-scm.org/D3059
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 09:41:18 -0700] rev 37335
hgweb: use revsymbol() for creating context from changeid
These seem to be for looking up a revision that can come from the
user, so revsymbol() is the right method to call (0194dac7 has more
information about my plans for repo[x]).
Differential Revision: https://phab.mercurial-scm.org/D3075
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 23:00:41 -0700] rev 37334
hgweb: inline changeidctx()
Differential Revision: https://phab.mercurial-scm.org/D3074
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 21:50:42 -0700] rev 37333
hgweb: drop support for "manifest" parameter
AFAICT (but note that this is the first time I look at hgweb code),
the "mercurial" query parameter was removed from rendered pages in
36fa5db79dd5 (hgweb: convert gitweb to NWI, 2006-10-05). Search for
"manifest=" in that diff to see why I think it was removed. It's about
time we stop looking for the parameter in requests.
Differential Revision: https://phab.mercurial-scm.org/D3073
Martin von Zweigbergk <martinvonz@google.com> [Wed, 04 Apr 2018 10:32:48 -0700] rev 37332
rebase: use single transaction when running in memory
rebase.singletransaction make rebase noticeably faster (~20% in a test
I just ran). It is not enabled by default because it risks losing
information if it aborts (see `hg help rebase`). When running rebase
with the experimental in-memory option on, rebase is first attempted
in memory, and if any conflicts occur, it restarts, this time writing
to disk. Thus, it should be safe to turn on single-transaction mode
for the in-memory phase.
Differential Revision: https://phab.mercurial-scm.org/D3076
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 13:16:12 -0700] rev 37331
bundlerepo: use super() when calling file()
We should be calling the default method, not reimplementing it.
Differential Revision: https://phab.mercurial-scm.org/D3058
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 03 Apr 2018 18:20:10 -0700] rev 37330
tests: remove superfluous config setting
format.usegeneraldelta defaults to true.
Differential Revision: https://phab.mercurial-scm.org/D3057
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 22:30:25 +0900] rev 37329
templater: mark .joinfmt as a private attribute
Yuya Nishihara <yuya@tcha.org> [Mon, 19 Mar 2018 20:32:06 +0900] rev 37328
obsutil: make obsfateprinter() less dependent on templater
joinfmt() is defined as 'lambda x: scmutil.formatchangeid(repo[x])' in
showsuccsandmarkers().
Function arguments are reordered so they look more normal.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 22:06:31 +0900] rev 37327
templater: abstract away from joinfmt
Future patches will add a wrapper for a list of template mappings, which
will implement a custom join() something like {join(mappings % template)}.
The original join() function is broken down as follows:
if hasattr(joinset, 'joinfmt'):
# hybrid.join() where values must be a list or a dict
joinitems((joinfmt(x) for x in values), sep)
elif isinstance(joinset, templateutil.wrapped):
# mappable.join()
show()
else:
# a plain list, a generator, or a byte string; joinfmt was identity()
joinset = templateutil.unwrapvalue(context, joinset)
joinitems(pycompat.maybebytestr(joinset), joiner)
Yuya Nishihara <yuya@tcha.org> [Tue, 20 Mar 2018 23:16:28 +0900] rev 37326
templater: micro-optimize join() with empty separator
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 21:42:27 +0900] rev 37325
templater: factor out generator of join()-ed items
Prepares for defining join() behavior per wrapped types and getting rid
of the public joinfmt attribute.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 23:24:50 +0900] rev 37324
templater: pass context to itermaps() for future extension
Unlike show() and tovalue(), a base mapping isn't passed to itermaps()
since it is the function to generate a partial mapping.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 21:21:50 +0900] rev 37323
templater: define interface for objects which act as iterator of mappings
Yuya Nishihara <yuya@tcha.org> [Wed, 04 Apr 2018 23:26:49 +0900] rev 37322
stringutil: drop escapedata() in favor of escapestr()
They are quite similar. Let's choose one that uses standard Python escape.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 30 Mar 2018 18:57:13 -0700] rev 37321
peer: make ui an attribute
With abc interfaces, instance attributes could not satisfy
@abc.abstractproperty requirements because interface conformance
was tested at type creation time. When we created the abc
peer interfaces, we had to make "ui" a @property to satisfy
abc.
Now that peer interfaces are using zope.interface and there is no
import time validation (but there are tests validating instances
conform to the interface), we can go back to using regular object
attributes.
Differential Revision: https://phab.mercurial-scm.org/D3069
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 30 Mar 2018 18:53:17 -0700] rev 37320
repository: port peer interfaces to zope.interface
zope.interface is superior. Let's switch to it.
Unlike abc, which defines interfaces through a base class,
zope.interface uses different types for interfaces and for
implementations. So, we had to invent some new types to hold the
interfaces in order to separate the interface from its default
implementation.
The names here could probably be better. I've been wanting to
overhaul the peer interface for a while. And wire protocol version
2 will force that work. So anticipate a refactoring of these
interfaces in later commits.
With this commit, we no longer test abc interfaces in
test-check-interfaces.py, so code for that has been removed.
Differential Revision: https://phab.mercurial-scm.org/D3068
# no-check-commit because of stream_out()
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 30 Mar 2018 14:52:32 -0700] rev 37319
wireproto: convert human output frames to CBOR
This is easier than rolling our own encoding format.
As a bonus, some of our artificial limits around lengths of
things went away because we are no longer using fixed length
fields to hold sizes.
Differential Revision: https://phab.mercurial-scm.org/D3067
Pulkit Goyal <7895pulkit@gmail.com> [Mon, 02 Apr 2018 17:06:42 +0530] rev 37318
py3: use pycompat.bytestr() intsead of str
Differential Revision: https://phab.mercurial-scm.org/D3071
Pulkit Goyal <7895pulkit@gmail.com> [Mon, 19 Mar 2018 12:46:56 +0530] rev 37317
py3: use print as a function in tests/test-walk.t
Differential Revision: https://phab.mercurial-scm.org/D3070
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 13:19:35 -0700] rev 37316
repo: remove now-unused changectx() method (API)
repo.changectx(x) was just a synonym for repo[x], so any extensions
that fail due to this commit should switch over to that form.
Differential Revision: https://phab.mercurial-scm.org/D3037
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 15:08:09 -0700] rev 37315
localrepo: use revsymbol() in lookup()
lookup() seems to be about looking up a revision based on a symbol
that may come from the user (via the wire protocol), so revsymbol() is
appropriate here.
Differential Revision: https://phab.mercurial-scm.org/D3055
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 23:29:51 -0700] rev 37314
histedit: avoid repo.lookup() for converting revnum to nodeid
Differential Revision: https://phab.mercurial-scm.org/D3054
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 23:27:50 -0700] rev 37313
outgoing: avoid repo.lookup() for converting revnum to nodeid
Differential Revision: https://phab.mercurial-scm.org/D3053
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 23:21:17 -0700] rev 37312
bisect: avoid repo.lookup() for converting revnum to nodeid
Differential Revision: https://phab.mercurial-scm.org/D3052
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 23:19:37 -0700] rev 37311
transplant: avoid repo.lookup() for converting revnum to nodeid
Differential Revision: https://phab.mercurial-scm.org/D3051
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 23:10:25 -0700] rev 37310
tests: avoid repo.lookup() for converting revnum to nodeid
Differential Revision: https://phab.mercurial-scm.org/D3050
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 15:14:31 -0700] rev 37309
bundle: consistently keep a list of stringified revisions in "revs"
Before this patch, "revs", in the "not base" branch, would be a list
of mixed integral revnums, hex nodeids, and branch names. After this
patch, they're all strings. They can still be a mix of hex nodeids and
branch names, but the important thing for my future patches is that
they're consistently in string form.
Differential Revision: https://phab.mercurial-scm.org/D3049
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 15:10:41 -0700] rev 37308
bundle: avoid repo.lookup() for converting revnum to nodeid
Differential Revision: https://phab.mercurial-scm.org/D3048
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 14:39:21 -0700] rev 37307
push: avoid using repo.lookup() for converting to nodeid
repo.lookup(x) currently simply does repo[x].node(), which supports
various types of inputs. As I explained in 0194dac77c93 (scmutil: add
method for looking up a context given a revision symbol, 2018-04-02),
I'd like to split that up so we use the new scmutil.revsymbol() for
string inputs repo[x] for integer revnums and binary nodeids. Since
repo.lookup() seems to exist in order to serve peer.lookup(), I think
it should be calling revsymbol. However, we have several callers that
use repo.lookup() with something that's not a string, so we need to
remove those first. This patch starts doing that. Many more will
follow.
Differential Revision: https://phab.mercurial-scm.org/D3047
Yuya Nishihara <yuya@tcha.org> [Tue, 03 Apr 2018 22:24:50 +0900] rev 37306
addremove: pass command-level similarity value down to scmutil.addremove()
Since we've changed to carry a similarity value by opts dict, it makes sense
to leave a string '0'-'100' value unmodified.
Augie Fackler <augie@google.com> [Tue, 03 Apr 2018 13:37:57 -0400] rev 37305
setup: add overlooked hgext.infinitepush package declaration
Will fix infinitepush tests that have been failing when run without --local.
Differential Revision: https://phab.mercurial-scm.org/D3038
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 09:12:15 -0700] rev 37304
tests: remove dependence on repo.changectx()
This was one of few remaining uses of repo.changectx() in core.
Differential Revision: https://phab.mercurial-scm.org/D3036
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 08:55:49 -0700] rev 37303
log: remove dependence on repo.changectx()
This was one of few remaining uses of repo.changectx() in core.
Differential Revision: https://phab.mercurial-scm.org/D3035
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 08:55:16 -0700] rev 37302
verify: remove dependence on repo.changectx()
This was one of few remaining uses of repo.changectx() in core.
Differential Revision: https://phab.mercurial-scm.org/D3034
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 08:46:58 -0700] rev 37301
bookmarks: switch from repo.changectx('.') to repo['.']
The two forms are synonymous and the new form is by far the more
common form.
Differential Revision: https://phab.mercurial-scm.org/D3033
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 14:52:57 -0700] rev 37300
stringutil: add function to pretty print an object
This is inspired by the pprint() module/function (which we can't
use because the output is different on Python 2 and 3 - namely the
use of b'' literals).
We hook it up to `hg debugwireproto` for printing the response to
a wire protocol command.
This foreshadows future peer work, which will support decoding
CBOR responses into rich data structures.
Differential Revision: https://phab.mercurial-scm.org/D2987
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 13:30:24 -0700] rev 37299
wireproto: add frame flag to denote payloads as CBOR
We may eventually want a separate frame type for this. But for
now this is the easiest to implement.
Differential Revision: https://phab.mercurial-scm.org/D2986
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 13:01:28 -0700] rev 37298
wireproto: implement custom __repr__ for frame
This version won't print the full payload (which could be large).
It also prints human friendly values for types and flags.
Differential Revision: https://phab.mercurial-scm.org/D2985
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 12:44:35 -0700] rev 37297
keepalive: implement readinto()
This is part of the standard I/O interface. It is used by the framing
protocol. So we need to implement it so frames can be decoded.
Differential Revision: https://phab.mercurial-scm.org/D2984
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 23 Mar 2018 16:24:53 -0700] rev 37296
wireproto: port protocol handler to zope.interface
zope.interface is superior to the abc module. Let's port to it.
As part of this, we add tests for interface conformance for
classes implementing the interface.
Differential Revision: https://phab.mercurial-scm.org/D2983
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 10:40:41 -0700] rev 37295
wireproto: separate commands tables for version 1 and 2 commands
We can't easily reuse existing command handlers for version 2
commands because the response types will be different. e.g. many
commands return nodes encoded as hex. Our new wire protocol is
binary safe, so we'll wish to encode nodes as binary.
We /could/ teach each command handler to look at the protocol
handler and change behavior based on the version in use. However,
this would make logic a bit unwieldy over time and would make
it harder to design a unified protocol handler interface. I think
it's better to create a clean break between version 1 and version 2
of commands on the server.
What I imagine happening is we will have separate @wireprotocommand
functions for each protocol generation. Those functions will parse the
request, dispatch to a common function to process it, then generate
the response in its own, transport-specific manner.
This commit establishes a separate table for tracking version 1
commands from version 2 commands. The HTTP server pieces have been
updated to use this new table.
Most commands are marked as both version 1 and version 2, so there is
little practical impact to this change.
A side-effect of this change is we now rely on transport registration
in wireprototypes.TRANSPORTS and certain properties of the protocol
interface. So a test had to be updated to conform.
Differential Revision: https://phab.mercurial-scm.org/D2982
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 10:12:02 -0700] rev 37294
wireproto: mark SSHv2 as a version 1 transport
The version component is used for filtering/routing wire protocol
commands to their proper handler. The actual version 2 of the wire
protocol commands will use a different encoding of responses. We
already have tests using the version 2 SSH transport and version 2
of the wire protocol commands won't be implemented atomically.
This commit marks the SSHv2 transport as version 1 so it will
still invoke the version 1 commands. Once the commands are all
implemented in version 2, we can restore its proper behavior.
Some tests had to be disabled as a result of this change.
Differential Revision: https://phab.mercurial-scm.org/D2981
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 14:05:29 -0700] rev 37293
wireproto: stop aliasing wire protocol types (API)
We generally shy away from aliasing module symbols. I think I
was keeping this around for API compatibility. We've already made
tons of other API breaks in the wire protocol code this release.
What's one more?
.. api::
``wireproto`` module no longer re-exports various types used to
define responses to wire protocol commands. Access these types
from the ``wireprototypes`` module.
Differential Revision: https://phab.mercurial-scm.org/D2979
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Mar 2018 14:34:32 -0700] rev 37292
wireproto: use CBOR for command requests
Now that we're using CBOR in the new wire protocol, let's convert
command requests to it.
Before I wrote this patch and was even thinking about CBOR, I was
thinking about how commands should be issued and came to the
conclusion that we didn't need separate frames to represent the
command name from its arguments. I already had a partially
completed patch prepared to merge the frames.
But with CBOR, it makes the implementation a bit simpler because
we don't need to roll our own serialization.
The changes here are a bit invasive. I tried to split this into
multiple commits to make it easier to review. But it was just too
hard.
* "command name" and "command argument" frames have been collapsed
into a "command request" frame.
* The flags for this new frame are totally different.
* Frame processing has been overhauled to reflect the new order
of things.
* Test fallout was significant. A handful of tests were removed.
Altogether, I think the new code is simpler. We don't have
complicated state around receiving commands. We're either receiving
command request frames or command data frames. We /could/
potentially collapse command data frames into command request
frames. Although I'd have to think a bit more about this before
I do it.
Differential Revision: https://phab.mercurial-scm.org/D2951
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Mar 2018 10:50:36 -0700] rev 37291
wireproto: define frame to represent progress updates
Today, a long-running operation on a server may run without any sign
of progress on the client. This can lead to the conclusion that the
server has hung or the connection has dropped. In fact, connections
can and do time out due to inactivity. And a long-running server
operation can result in the connection dropping prematurely because
no data is being sent!
While we're inventing the new wire protocol, let's provide a mechanism
for communicating progress on potentially expensive server-side events.
We introduce a new frame type that conveys "progress" updates. This
frame type essentially holds the data required to formulate a
``ui.progress()`` call.
We only define the frame right now. Implementing it will be a bit of
work since there is no analog to progress frames in the existing
wire protocol. We'll need to teach the ui object to write to the
wire protocol, etc.
The use of a CBOR map may seem wasteful, as this will encode key
names in every frame. This *is* wasteful. However, maps are
extensible. And the intent is to always use compression via
streams. Compression will make the overhead negligible since repeated
strings will be mostly eliminated over the wire.
Differential Revision: https://phab.mercurial-scm.org/D2902
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 28 Mar 2018 15:05:39 -0700] rev 37290
wireproto: syntax for encoding CBOR into frames
We just vendored a library for encoding and decoding the CBOR
data format. While the intent of that vendor was to support state
files, CBOR is really a nice data format. It is extensible and
compact.
I've been feeling dirty inventing my own data formats for
frame payloads. While custom formats can always beat out a generic
format, there is a cost to be paid in terms of implementation,
comprehension, etc. CBOR is compact enough that I'm not too
worried about efficiency loss. I think the benefits of using
a standardized format outweigh rolling our own formats. So
I plan to make heavy use of CBOR in the wire protocol going
forward.
This commit introduces support for encoding CBOR data in frame
payloads to our function to make a frame from a human string.
We do need to employ some low-level Python code in order to
evaluate a string as a Python expression. But other than that,
this should hopefully be pretty straightforward.
Unit tests for this function have been added.
Differential Revision: https://phab.mercurial-scm.org/D2948
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Mar 2018 13:59:56 -0700] rev 37289
wireproto: explicit API to create outgoing streams
It is better to create outgoing streams through the reactor so the
reactor knows about what streams are active and can track them
accordingly.
Test output changes slightly because frames from subsequent responses
no longer have the "stream begin" stream flag set because the stream
is now used across all responses.
Differential Revision: https://phab.mercurial-scm.org/D2947
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Mar 2018 11:00:16 -0700] rev 37288
wireproto: add streams to frame-based protocol
Previously, the frame-based protocol was just a series of frames,
with each frame associated with a request ID.
In order to scale the protocol, we'll want to enable the use of
compression. While it is possible to enable compression at the
socket/pipe level, this has its disadvantages. The big one is it
undermines the point of frames being standalone, atomic units that
can be read and written: if you add compression above the framing
protocol, you are back to having a stream-based protocol as opposed
to something frame-based.
So in order to preserve frames, compression needs to occur at
the frame payload level.
Compressing each frame's payload individually will limit compression
ratios because the window size of the compressor will be limited
by the max frame size, which is 32-64kb as currently defined. It
will also add CPU overhead, as it is more efficient for compressors
to operate on fewer, larger blocks of data than more, smaller blocks.
So compressing each frame independently is out.
This means we need to compress each frame's payload as if it is part
of a larger stream.
The simplest approach is to have 1 stream per connection. This
could certainly work. However, it has disadvantages (documented below).
We could also have 1 stream per RPC/command invocation. (This is the
model HTTP/2 goes with.) This also has disadvantages.
The main disadvantage to one global stream is that it has the very
real potential to create CPU bottlenecks doing compression. Networks
are only getting faster and the performance of single CPU cores has
been relatively flat. Newer compression formats like zstandard offer
better CPU cycle efficiency than predecessors like zlib. But it still
all too common to saturate your CPU with compression overhead long
before you saturate the network pipe.
The main disadvantage with streams per request is that you can't
reap the benefits of the compression context for multiple requests.
For example, if you send 1000 RPC requests (or HTTP/2 requests for
that matter), the response to each would have its own compression
context. The overall size of the raw responses would be larger because
compression contexts wouldn't be able to reference data from another
request or response.
The approach for streams as implemented in this commit is to support
N streams per connection and for streams to potentially span requests
and responses. As explained by the added internals docs, this
facilitates servers and clients delegating independent streams and
compression to independent threads / CPU cores. This helps alleviate
the CPU bottleneck of compression. This design also allows compression
contexts to be reused across requests/responses. This can result in
improved compression ratios and less overhead for compressors and
decompressors having to build new contexts.
Another feature that was defined was the ability for individual frames
within a stream to declare whether that individual frame's payload
uses the content encoding (read: compression) defined by the stream.
The idea here is that some servers may serve data from a combination
of caches and dynamic resolution. Data coming from caches may be
pre-compressed. We want to facilitate servers being able to essentially
stream bytes from caches to the wire with minimal overhead. Being
able to mix and match with frames are compressed within a stream
enables these types of advanced server functionality.
This commit defines the new streams mechanism. Basic code for
supporting streams in frames has been added. But that code is
seriously lacking and doesn't fully conform to the defined protocol.
For example, we don't close any streams. And support for content
encoding within streams is not yet implemented. The change was
rather invasive and I didn't think it would be reasonable to implement
the entire feature in a single commit.
For the record, I would have loved to reuse an existing multiplexing
protocol to build the new wire protocol on top of. However, I couldn't
find a protocol that offers the performance and scaling characteristics
that I desired. Namely, it should support multiple compression
contexts to facilitate scaling out to multiple CPU cores and
compression contexts should be able to live longer than single RPC
requests. HTTP/2 *almost* fits the bill. But the semantics of HTTP
message exchange state that streams can only live for a single
request-response. We /could/ tunnel on top of HTTP/2 streams and
frames with HEADER and DATA frames. But there's no guarantee that
HTTP/2 libraries and proxies would allow us to use HTTP/2 streams
and frames without the HTTP message exchange semantics defined in
RFC 7540 Section 8. Other RPC protocols like gRPC tunnel are built
on top of HTTP/2 and thus preserve its semantics of stream per
RPC invocation. Even QUIC does this. We could attempt to invent a
higher-level stream that spans HTTP/2 streams. But this would be
violating HTTP/2 because there is no guarantee that HTTP/2 streams
are routed to the same server. The best we can do - which is what
this protocol does - is shoehorn all request and response data into
a single HTTP message and create streams within. At that point, we've
defined a Content-Type in HTTP parlance. It just so happens our
media type can also work as a standalone, stream-based protocol,
without leaning on HTTP or similar protocol.
Differential Revision: https://phab.mercurial-scm.org/D2907
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Mar 2018 13:57:22 -0700] rev 37287
wireproto: start to associate frame generation with a stream
An upcoming commit will introduce "streams" into the frame-based wire
protocol. In preparation for this invasive change, we introduce a basic
"stream" class and have all operations that create frames also operate
alongside a stream instance.
Differential Revision: https://phab.mercurial-scm.org/D2906
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 26 Mar 2018 13:51:22 -0700] rev 37286
tests: fix duplicate and failing test
There were two "testconflictingrequestid" methods. Naturally this isn't
an error in Python. And by our luck, the test was failing.
So we rename the test and fix it to pass.
As part of this, _sendsingleframe() now takes a frame, not a string
describing the frame. This is better because action at a distance can
be confusing.
Differential Revision: https://phab.mercurial-scm.org/D2950
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Apr 2018 16:47:53 -0700] rev 37285
debugcommands: drop offset and length from debugindex by default
These fields are an implementation detail of revlog storage. As
such, they are not part of the generic storage "index" interface
and shouldn't be displayed by default.
Because we don't have another way to display these fields, we've
retained support for printing these fields via --verbose.
Yes, I know we should probably be doing all this formatting using
modern formatting/templater APIs. I didn't feel like scope
bloating this patch.
Differential Revision: https://phab.mercurial-scm.org/D3028
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Apr 2018 16:28:20 -0700] rev 37284
debugcommands: drop base revision from debugindex
Revlog index data consists of generic index metadata that will
likely be implemented across all storage engines and revlog-specifc
metadata.
Most tests printing index data only care about the generic fields.
This commit drops the printing of the base revision from
`hg debugindex`. This value is an implementation detail of
revlogs / delta chains. If tests are interested in verifying this
implementation detail, `hg debugdeltachain` is a better command.
Most tests were skipping over this field anyway. Tests that weren't
looked like they were newer. So my guess is we forgot to make them
skip the field to match the style of the older tests. This reinforces
my belief that the base revision is not worth having in
`hg debugindex`.
Differential Revision: https://phab.mercurial-scm.org/D3027
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Apr 2018 16:24:57 -0700] rev 37283
tests: use debugdeltachain where appropriate
Some tests are verifying delta chain type things. This metadata
has more to do with a revlog implementation details than index
data, which is theoretically generic.
This commit ports some tests to `hg debugdeltachain`, as it is the
more appropriate debug command for looking at delta metadata.
Differential Revision: https://phab.mercurial-scm.org/D3026
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Apr 2018 15:55:50 -0700] rev 37282
tests: don't use revlog paths in tests
Debug commands operating on revlogs don't need the full revlog
path: they can accept the relative path to a tracked file or use
-c/-m to specify a changelog or manifest.
Not using the revlog path makes tests more resilient to cases
where revlogs aren't being used for storage.
Differential Revision: https://phab.mercurial-scm.org/D3025
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 21:03:16 +0900] rev 37281
templater: define interface for objects requiring unwrapvalue()
unwrapvalue() is changed to not return a lazy bytes generator for "wrapped"
types because I want to define the tovalue() interface as such. It's a baby
step to unify unwrapvalue() and _unwrapvalue().
Yuya Nishihara <yuya@tcha.org> [Fri, 23 Mar 2018 21:40:16 +0900] rev 37280
templater: extract private function to evaluate generator to byte string
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 23:14:21 +0900] rev 37279
templater: pass (context, mapping) down to unwrapvalue()
The same reason as why I made unwraphybrid() take a (context, mapping) pair.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 20:58:28 +0900] rev 37278
templater: drop unneeded generator from mappable object
Per the definition of the show() interface, it can return a bytes.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 20:56:42 +0900] rev 37277
templater: mark .gen as a private attribute
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 00:11:36 +0900] rev 37276
templatekw: do not directly call .gen
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 20:52:50 +0900] rev 37275
templater: define interface for objects requiring unwraphybrid()
Prepares for introducing another hybrid-like data type. show() takes context
as an argument so a wrapper class may render its items by pre-configured
template:
def show(self, context, mapping):
return (context.expand(self._tmpl, mapping + lm) for lm in self._mappings)
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Mar 2018 20:09:05 +0900] rev 37274
templater: pass (context, mapping) down to unwraphybrid()
See the subsequent patches for why.
I initially thought it would be wrong to pass a mapping to flatten() and
stringify() since these functions may be applied to a tree of generators,
where each node should be bound to the mapping when it was evaluated. But,
actually that isn't a problem. If an intermediate node has to override a
mapping dict, it can do on unwraphybrid() and yield "unwrapped" generator
of byte strings:
"{f(g(v))}" # literal template example.
^^^^ # g() want to override a mapping, so it returns a wrapped
# object 'G{V}' with partial mapping 'lm' attached.
^^^^^^^ # f() stringifies 'G{V}', starting from a mapping 'm'.
# when unwrapping 'G{}', it updates 'm' with 'lm', and
# passes it to 'V'.
This structure is important for the formatter (and the hgweb) to build a
static template keyword, which can't access a mapping dict until evaluation
phase.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 16:18:33 -0700] rev 37273
scmutil: add method for looking up a context given a revision symbol
changectx's constructor currently supports a mix if inputs:
* integer revnums
* binary nodeids
* '.', 'tip', 'null'
* stringified revnums
* namespaced identifiers (e.g. bookmarks and tags)
* hex nodeids
* partial hex nodeids
The first two are always internal [1]. The other five can be specified
by the user. The third type ('.', 'tip', 'null') often comes from
either the user or internal callers. We probably have some internal
callers that pass hex nodeids too, perhaps even partial ones
(histedit?). There are only a few callers that pass user-supplied
strings: revsets.stringset, peer.lookup, webutil.changeidctx, and
maybe one or two more.
Supporting this mix of things in the constructor is convenient, but a
bit strange, IMO. For example, if repo[node] is given a node that's
not in the repo, it will first check if it's bookmark etc before
raising an exception. Of course, the risk of it being a bookmark is
extremely small, but it just feels ugly.
Also, a problem with having this code in the constructor (whether it
supports a mix of types or not) is that it's harder to override (I'd
like to override it, and that's how this series started).
This patch starts moving out the handling of user-supplied strings by
introducing scmutil.revsymbol(). So far, that just checks that the
input is indeed a string, and then delegates to repo[symbol]. The
patch also calls it from revsets.stringset to prove that it works.
[1] Well, you probably can enter a 20-byte binary nodeid on the
command line, but I don't think we should care to preserve
support for that.
Differential Revision: https://phab.mercurial-scm.org/D3024
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 23:52:43 -0700] rev 37272
narrow: add trailing slash to dir earlier for debug{revlog,index,data}
The treemanifest code internally uses trailing slashes on directories
(except for the root directory, which is an empty string). We should
make sure we pass in directories with trailing slashes when we work
with the treemanifest code. For some reason, I seem to have decided to
be nice to the callers instead in 49c583ca48c4 (treemanifest: add
--dir option to debug{revlog,data,index}, 2015-04-12). Let's fix that
and pay the cost of fixing up the directory name close close to where
we get it from the user.
Differential Revision: https://phab.mercurial-scm.org/D3032
Sushil khanchi <sushilkhanchi97@gmail.com> [Sat, 31 Mar 2018 23:49:58 +0530] rev 37271
addremove: remove dry_run, similarity from scmutil.addremove (API)
Differential Revision: https://phab.mercurial-scm.org/D3000
Sangeet Kumar Mishra <mail2sangeetmishra@gmail.com> [Tue, 03 Apr 2018 12:16:19 +0530] rev 37270
histedit: make errror message translatable
This is a follow up patch to https://phab.mercurial-scm.org/D2394
As suggested by Yuya, this patch makes the error message translatable
Differential Revision: https://phab.mercurial-scm.org/D3031
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 09:18:01 -0700] rev 37269
context: drop support for changeid='' (API)
Since the previous commit, there seem to be no users who pass '' to
repo.__getitem__, so let's drop support for it.
It may seem like a small cost to keep support for it, but I've spent
time being confused by it twice already.
Differential Revision: https://phab.mercurial-scm.org/D3021
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 08:43:08 -0700] rev 37268
subrepo: use repo['.'] instead of repo['']
The "state" value (a revision) passed to abstractsubrepo.phase() can
be '' to represent the currently checked out revisions. Let's convert
that to the more common '.'.
I think this is the last of use of repo['.'] in core.
Differential Revision: https://phab.mercurial-scm.org/D3019
Martin von Zweigbergk <martinvonz@google.com> [Tue, 03 Apr 2018 00:12:29 +0530] rev 37267
children: use repo['.'] instead of repo['']
Differential Revision: https://phab.mercurial-scm.org/D3020
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 09:16:52 -0700] rev 37266
revset: drop support for '' as alias for '.'
Not marked BC because I think support for using '' on the CLI was
there by accident, and we don't seem to have documented it.
Differential Revision: https://phab.mercurial-scm.org/D3018
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 09:06:24 -0700] rev 37265
tests: add test showing current parse of empty string symbol in revset
We support e.g. parents(""), but I think that's by accident and I'll
change it soon.
Differential Revision: https://phab.mercurial-scm.org/D3017
Martin von Zweigbergk <martinvonz@google.com> [Mon, 02 Apr 2018 08:46:07 -0700] rev 37264
clone: rename "rev" to "revs" since there can be many
It was a little tricky in hg.clone(), since there was a local "revs"
variable defined there, but "rev" was never used after "revs", so I
just overwrote it.
Note that clonewithshare() should also have its "rev" argument renamed
to "revs", but I'll leave that to someone else.
Differential Revision: https://phab.mercurial-scm.org/D3016
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 15:41:16 -0700] rev 37263
parseurl: consistently call second output "branches"
Differential Revision: https://phab.mercurial-scm.org/D3015
Yuya Nishihara <yuya@tcha.org> [Sun, 01 Apr 2018 11:06:29 +0900] rev 37262
templatefuncs: do not crash because of invalid value fed to mailmap()
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:34:15 -0700] rev 37261
scmutil: deprecate revpairnodes()
Differential Revision: https://phab.mercurial-scm.org/D3012
Martin von Zweigbergk <martinvonz@google.com> [Sun, 01 Apr 2018 09:30:44 -0700] rev 37260
tests: use context-return revpair() in autodiff
Differential Revision: https://phab.mercurial-scm.org/D3013
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:26:07 -0700] rev 37259
fileset: use context-returning revpair()
Differential Revision: https://phab.mercurial-scm.org/D3011
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:31:28 -0700] rev 37258
status: use context-returning revpair()
Differential Revision: https://phab.mercurial-scm.org/D3010
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:49:44 -0700] rev 37257
diff: simplify by converting contexts to nodeids a little later
Differential Revision: https://phab.mercurial-scm.org/D3009
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:25:02 -0700] rev 37256
diff: use context-returning revpair()
Differential Revision: https://phab.mercurial-scm.org/D3008
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:38:53 -0700] rev 37255
extdiff: use context-returning revpair()
Differential Revision: https://phab.mercurial-scm.org/D3007
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:10:46 -0700] rev 37254
scmutil: make revpair() return context objects (API)
Differential Revision: https://phab.mercurial-scm.org/D3006
Martin von Zweigbergk <martinvonz@google.com> [Sat, 31 Mar 2018 23:37:25 -0700] rev 37253
scmutil: introduce deprecated alias for revpair()
revsingle() returns a context object, revpair() returns nodeids,
revrange() returns integer revisions (in a revset). I'm going to
reduce this inconsistency by making revpair() return context
objects. Changing the return type is not nice to extensions, so this
patch introduces a nodeid-returning version of revpair() that they can
detect and use. Update callers to the new function so we can change
revpair() itself and then migrate them back one by one.
Differential Revision: https://phab.mercurial-scm.org/D3005
Matt Harbison <matt_harbison@yahoo.com> [Sat, 31 Mar 2018 23:58:08 -0400] rev 37252
lfs: ensure the transfer request is for a known URI
Since the dispatching code only checks the beginning of the string, this
enforces that there's only one more path component.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 31 Mar 2018 23:47:56 -0400] rev 37251
lfs: avoid an improper usage of os.path.basename() to parse a URI