Kyle Lippincott <spectral@google.com> [Thu, 08 Mar 2018 11:44:03 -0800] rev 37091
copyfile: preserve stat info (mtime, etc.) when doing copies/renames
Differential Revision: https://phab.mercurial-scm.org/D2729
Matt Harbison <matt_harbison@yahoo.com> [Thu, 22 Mar 2018 22:58:31 -0400] rev 37090
merge: add 'isknown=True' to a dirstate.normalize() in _unknowndirschecker
Per the docstring for dirstate.normalize().
Matt Harbison <matt_harbison@yahoo.com> [Thu, 22 Mar 2018 22:56:29 -0400] rev 37089
merge: pconvert paths in _unknowndirschecker before dirstate-normalizing
This fixes the failure in test-pathconflicts-basic.t on Windows. The test was
passing in 'a\b', which was getting normalized to 'A\B', which isn't in
dirstate. (The filesystem path is all lowercase anyway.)
This isn't the only case of calling dirstate.normalize(), but other methods here
(util.finddirs()) seem to assume the input paths are already using '/'. I think
the backslash comes from wvfs.reljoin() (in this case), but could also come from
wvfs.walk(), so this is the only case that needs it.
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 22:39:43 +0900] rev 37088
util: enable deprecation warning for stringutil proxy (API)
.. api::
Several generic string helper functions have been moved to utils.stringutil
module.
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 21:56:20 +0900] rev 37087
stringutil: bulk-replace call sites to point to new module
This might conflict with other patches floating around, sorry.
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 21:19:31 +0900] rev 37086
stringutil: move generic string helpers to new module
Per https://phab.mercurial-scm.org/D2903#46738
URL and file paths functions are left since they are big enough to make
separate modules.
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 21:32:19 +0900] rev 37085
util: remove unused private constant '_hextochr'
The only user, _urlunquote(), was removed by 81d38478fced.
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 21:20:47 +0900] rev 37084
util: mark internal constants of escapedata() as private
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 21:14:12 +0900] rev 37083
util: adjust indent level in wrap()
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 21:13:31 +0900] rev 37082
util: mark MBTextWrapper as private
Makes porting slightly easier.
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Mar 2018 21:10:42 +0900] rev 37081
util: add helper to define proxy functions to utils.*
Kyle Lippincott <spectral@google.com> [Wed, 21 Mar 2018 12:36:29 -0700] rev 37080
filemerge: make the 'local' path match the format that 'base' and 'other' use
If we pass a separate '$output' arg to the merge tool, we produce four files:
local, base, other, and output. In this situation, 'output' will be the
original filename, 'base' and 'other' are temporary files, and previously
'local' would be the backup file (so if 'output' was foo.txt, 'local' would be
foo.txt.orig).
This change makes it so that 'local' follows the same pattern as 'base' and
'other' - it will be a temporary file either in the
`experimental.mergetempdirprefix`-controlled directory with a name like
foo~local.txt, or in the normal system-wide temp dir with a name like
foo~local.RaNd0m.txt.
For the cases where the merge tool does not use an '$output' arg, 'local' is
still the destination filename, and 'base' and 'other' are unchanged.
The hope is that this is much easier for people to reason about; rather than
having a tool like Meld pop up with three panes, one of them with the filename
"foo.txt.orig", one with the filename "foo.txt", and one with
"foo~other.StuFf2.txt", we can (when the merge temp dir stuff is enabled) make
it show up as "foo~local.txt", "foo.txt" and "foo~other.txt", respectively.
This also opens the door to future customization, such as getting the
operation-provided labels and a hash prefix into the filenames (so we see
something like "foo~dest.abc123", "foo.txt", and "foo~src.d4e5f6").
Differential Revision: https://phab.mercurial-scm.org/D2889
Matt Harbison <matt_harbison@yahoo.com> [Wed, 21 Mar 2018 22:36:26 -0400] rev 37079
test-strip-narrow: adjust bundle removal for Windows test stability
MSYS was mangling $TESTTMP to C:\\Users\\...\\test-narrow-strip.t-flat/, which
caused `rm` to fail. The -f was suppressing -ENOENT, so the only clue something
was wrong was when 2 bundles were applied via `hg unbundle` on line 91, instead
of just 1. This changed the text output of `hg unbundle`.
The first `rm` wasn't causing an issue, but is changed for consistency with the
rest of the file.
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 21:38:57 +0900] rev 37078
templater: drop symbols which should be overridden by new 'ctx' (issue5612)
This problem is caused by impedance mismatch between the templater and the
formatter interface, which is that the template keywords are generally
evaluated dynamically, but the formatter puts static values into a template
mapping.
This patch avoids the problem by removing conflicting values from a mapping
dict when a 'ctx' is switched.
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 21:22:52 +0900] rev 37077
templater: factor out function to create mapping dict for nested evaluation
overlaymap() is the hook point to drop mapping items conflicting with the
default keywords which have to be re-evaluated with new 'ctx' resource.
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 20:43:39 +0900] rev 37076
templater: introduce resourcemapper class
A couple more functions will be added later to work around nested mapping
bugs such as the issue 5612.
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Mar 2018 20:27:38 +0900] rev 37075
log: do no expect templateresources() returning a dict
The resources dict will be replaced with new resource mapper object, which
won't implement __getitem__(key). Share the whole resources object with
_graphnodeformater() to make porting easier.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 23:11:55 +0900] rev 37074
templatekw: mark _showlist() as deprecated (API)
.. api::
``templatekw._showlist()`` is deprecated in favor of
``templateutil._showcompatlist()``, which takes ``context`` in place of
``templ``.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 23:09:21 +0900] rev 37073
templater: drop 'templ' from resources dict
Partially resolves cycle, templ -> context -> templ. This will make it easier
to replace the resources dict with new immutable resource mapper interface.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 23:01:51 +0900] rev 37072
templatekw: stop using _showlist() which is about to be deprecated
Use the new context-based API instead.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 22:47:15 +0900] rev 37071
templater: use template context to render old-style list template
Prepares for dropping the 'templ' resource.
This means old-style list templates are processed by the same engine class
as the one for the list node. I think that's fine since templates for the
same list should be tightly coupled, and I believe the extension point for
the engine classes isn't actually used.
Now templatekw._showlist() is a compatibility wrapper for _showcompatlist(),
and will be deprecated soon. The function is still marked as private since
I plan to change the interface to get rid of closures capturing context and
mapping.
Yuya Nishihara <yuya@tcha.org> [Fri, 16 Mar 2018 22:36:40 +0900] rev 37070
templater: add context.preload(t) to test if the specified template exists
I'm going to remove 'templ' from the resources dict because it is the only
resource that the caller can't provide. This also implies that putting
'templ' into the resources dict creates a reference cycle.
context.preload(t) will be used in place of templater.__contains__().
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Mar 2018 12:28:19 +0900] rev 37069
annotate: pack line content into annotateline object (API)
Just for code readability. We can do that since the annotateline type is
no longer used while computing the history.
Yuya Nishihara <yuya@tcha.org> [Tue, 13 Mar 2018 22:18:06 +0900] rev 37068
annotate: drop linenumber flag from fctx.annotate() (API)
Now linenumber=True is fast enough to be enabled by default.
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Mar 2018 20:45:10 +0900] rev 37067
annotate: do not construct attr.s object per line while computing history
Unfortunately, good abstraction has a cost. It's way slower to construct
an annotateline() object than creating a plain tuple or a list. This patch
changes the internal data structure from row-based to columnar, so the
decorate() function can be instant (i.e. no Python in hot loop.)
For code readability, the outermost tuple is switched to an attr.s object
instead.
(original, row-based attr.s)
$ hg annot mercurial/commands.py --time > /dev/null
time: real 11.470 secs (user 11.400+0.000 sys 0.070+0.000)
$ hg annot mercurial/commands.py --time --line-number > /dev/null
time: real 39.590 secs (user 39.500+0.000 sys 0.080+0.000)
(this patch, columnar)
$ hg annot mercurial/commands.py --time > /dev/null
time: real 11.780 secs (user 11.710+0.000 sys 0.070+0.000)
$ hg annot mercurial/commands.py --time --line-number > /dev/null
time: real 12.240 secs (user 12.170+0.000 sys 0.090+0.000)
(cf. 4.3.3, row-based tuple)
$ hg annot mercurial/commands.py --time --line-number > /dev/null
time: real 19.540 secs (user 19.460+0.000 sys 0.080+0.000)
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 15 Mar 2018 18:05:49 -0700] rev 37066
wireproto: explicitly track which requests are active
We previously only tracked which requests are receiving. A
misbehaving client could accidentally have multiple requests with
the same ID in flight.
We now explicitly track which request IDs are currently active.
We make it illegal to receive a frame associated with a request
ID that has already been dispatched.
Differential Revision: https://phab.mercurial-scm.org/D2901
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 15 Mar 2018 16:09:58 -0700] rev 37065
wireproto: use named arguments when passing around frame data
Named arguments is easier to reason about compared to positional
arguments. Especially when you have many positional arguments.
Differential Revision: https://phab.mercurial-scm.org/D2900
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 15 Mar 2018 16:03:14 -0700] rev 37064
wireproto: define attr-based classes for representing frames
When frames only had 3 attributes, it was reasonable to
represent them as a tuple. With them growing more attributes,
it will be easier to pass them around as a more formal type.
So let's define attr-based classes to represent frame headers and
full frames.
Differential Revision: https://phab.mercurial-scm.org/D2899
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 22:19:00 -0700] rev 37063
wireproto: define human output side channel frame
Currently, the SSH protocol delivers output tailored for people over
the stderr file descriptor. The HTTP protocol doesn't have this
file descriptor (because it only has an input and output pipe). So
it encodes textual output intended for humans within the protocol
responses. So response types have a facility for capturing output
to be printed to users. Some don't. And sometimes the implementation
of how that output is conveyed is super hacky.
On top of that, bundle2 has an "output" part that is used to store
output that should be printed when this part is encountered.
bundle2 also has the concept of "interrupt" chunks, which can be
used to signal that the regular bundle2 stream is to be
preempted by an out-of-band part that should be processed immediately.
This "interrupt" part can be an "output" part and can be used to
print data on the receiver.
The status quo is inconsistent and insane. We can do better.
This commit introduces a dedicated frame type on the frame-based
protocol for denoting textual data that should be printed on the
receiver. This frame type effectively constitutes a side-channel
by which textual data can be printed on the receiver without
interfering with other in-progress transmissions, such as the
transmission of command responses.
But wait - there's more! Previous implementations that transferred
textual data basically instructed the client to "print these bytes."
This suffered from a few problems.
First, the text data that was transmitted and eventually printed
originated from a server with a specic i18n configuration. This
meant that clients would see text using whatever the i18n settings
were on the server. Someone in France could connect to a server in
Japan and see unlegible Japanese glyphs - or maybe even mojibake.
Second, the normalization of all text data originated on servers
resulted in the loss of the ability to apply formatting to that
data. Local Mercurial clients can apply specific formatting
settings to individual atoms of text. For example, a revision can
be colored differently from a commit message. With data over the
wire, the potential for this rich formatting was lost. The best you
could do (without parsing the text to be printed), was apply a
universal label to it and e.g. color it specially.
The new mechanism for instructing the peer to print data does
not have these limitations.
Frames instructing the peer to print text are composed of a
formatting string plus arguments. In other words, receivers can
plug the formatting string into the i18n database to see if a local
translation is available. In addition, each atom being instructed
to print has a series of "labels" associated with it. These labels
can be mapped to the Mercurial UI's labels so locally configured
coloring, styling, etc settings can be applied.
What this all means is that textual messages originating on servers
can be localized on the client and richly formatted, all while
respecting the client's settings. This is slightly more complicated
than "print these bytes." But it is vastly more user friendly.
FWIW, I'm not aware of other protocols that attempt to encode
i18n and textual styling in this manner. You could lobby the
claim that this feature is over-engineered. However, if I were to
sit in the shoes of a non-English speaker learning how to use
version control, I think I would *love* this feature because
it would enable me to see richly formatted text in my chosen
locale.
Anyway, we only implement support for encoding frames of this
type and basic tests for that encoding. We'll still need to
hook up the server and its ui instance to emit these frames.
I recognize this feature may be a bit more controversial than
other aspects of the wire protocol because it is a bit
"radical." So I'd figured I'd start small to test the waters and
see if others feel this feature is worthwhile.
Differential Revision: https://phab.mercurial-scm.org/D2872
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 19 Mar 2018 16:55:07 -0700] rev 37062
wireproto: service multiple command requests per HTTP request
Now that our new frame-based protocol server can understand how
to ingest multiple, possibly interleaved, command requests, let's
hook it up to the HTTP server.
The code on the HTTP side of things is still a bit hacky. We need
a bit of work around error handling, content types, etc. But it's
a start.
Among the added tests, we demonstrate that a client can send frames
for multiple commands iterleaved with each other and that a later
issued command can respond before the first one has finished
sending. This makes our multi-request model technically superior
to the previous "batch" command.
Differential Revision: https://phab.mercurial-scm.org/D2871
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 16:53:30 -0700] rev 37061
wireproto: support for receiving multiple requests
Now that we have request IDs on each frame and a specification
that allows multiple requests to be issued simultaneously,
possibly interleaved, let's teach the server to deal with that.
Instead of tracking the state for *the* active command request,
we instead track the state of each receiving command by its
request ID. The multiple states in our state machine for processing
each command's state has been collapsed into a single state for
"receiving commands."
Tests have been added so our branch coverage covers all meaningful
branches.
However, we did lose some logical coverage. The implementation
of this new feature opens up the door to a server having partial
command requests when end of input is reached. We will probably
want a mechanism to deal with partial requests. For now, I've
tracked that as a known issue in the class docstring. I've
also noted an abuse vector that becomes a little bit easier to
exploit with this feature.
Differential Revision: https://phab.mercurial-scm.org/D2870
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 14 Mar 2018 16:51:34 -0700] rev 37060
wireproto: add request IDs to frames
One of my primary goals with the new wire protocol is to make
operations faster and enable both client and server-side
operations to scale to multiple CPU cores.
One of the ways we make server interactions faster is by reducing
the number of round trips to that server.
With the existing wire protocol, the "batch" command facilitates
executing multiple commands from a single request payload. The way
it works is the requests for multiple commands are serialized. The
server executes those commands sequentially then serializes all
their results. As an optimization for reducing round trips, this
is very effective. The technical implementation, however, is pretty
bad and suffers from a number of deficiencies. For example, it
creates a new place where authorization to run a command must be
checked. (The lack of this checking in older Mercurial releases
was CVE-2018-1000132.)
The principles behind the "batch" command are sound. However, the
execution is not. Therefore, I want to ditch "batch" in the
new wire protocol and have protocol level support for issuing
multiple requests in a single round trip.
This commit introduces support in the frame-based wire protocol to
facilitate this. We do this by adding a "request ID" to each frame.
If a server sees frames associated with different "request IDs," it
handles them as separate requests. All of this happening possibly
as part of the same message from client to server (the same request
body in the case of HTTP).
We /could/ model the exchange the way pipelined HTTP requests do,
where the server processes requests in order they are issued and
received. But this artifically constrains scalability. A better
model is to allow multi-requests to be executed concurrently and
for responses to be sent and handled concurrently. So the
specification explicitly allows this. There is some work to be done
around specifying dependencies between multi-requests. We take
the easy road for now and punt on this problem, declaring that
if order is important, clients must not issue the request until
responses to dependent requests have been received.
This commit focuses on the boilerplate of implementing the request
ID. The server reactor still can't manage multiple, in-flight
request IDs. This will be addressed in a subsequent commit.
Because the wire semantics have changed, we bump the version of the
media type.
Differential Revision: https://phab.mercurial-scm.org/D2869