Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:16:29 +0300] rev 39445
py3: don't return the revid as unicode in hgext/convert/subversion.py
I tried digging why u'' was added in first place, and I was unable to found
something relevant. This might be because some API's takes unicodes, I am not
sure.
Differential Revision: https://phab.mercurial-scm.org/D4454
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:15:17 +0300] rev 39444
py3: make sure we pass str in os.sysconf in hgext/convert/common.py
# skip-blame because just r'' prefix
Differential Revision: https://phab.mercurial-scm.org/D4453
Augie Fackler <augie@google.com> [Tue, 04 Sep 2018 12:16:28 -0400] rev 39443
merge with stable
Yuya Nishihara <yuya@tcha.org> [Tue, 04 Sep 2018 13:29:21 +0900] rev 39442
revlog: fix size of Python nodetree object
Follows up 9f097214fbf3.
Yuya Nishihara <yuya@tcha.org> [Mon, 03 Sep 2018 23:03:19 +0900] rev 39441
revert: stabilize status message of chunks selected interactively
Unfortunately, patch.filterpatch() doesn't preserve the order of the input
files. We have to sort them manually.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 29 Aug 2018 14:29:01 -0700] rev 39440
cborutil: add a buffering decoder
The sansiodecoder leaves it up to the callers to feed in data that
wasn't fully consumed last time.
This commit implements a decoder that performs buffering of
leftover chunks from the previous invocation. It otherwise
behaves identically to sansiodecoder.
Differential Revision: https://phab.mercurial-scm.org/D4434
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 31 Aug 2018 15:54:17 -0700] rev 39439
cborutil: remove readindefinitebytestringtoiter()
This was implemented as part of implementing streaming encoding.
It was never used outside of tests.
Now that we have a full CBOR decoder, it can be used for incremental
decoding of indefinite-length byte strings.
This also removes the last use of the vendored cbor2 package from this
module.
Differential Revision: https://phab.mercurial-scm.org/D4433
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:02:48 -0700] rev 39438
cborutil: implement sans I/O decoder
The vendored CBOR package decodes by calling read(n) on an object.
There are a number of disadvantages to this:
* Uses blocking I/O. If sufficient data is not available, the decoder
will hang until it is.
* No support for partial reads. If the read(n) returns less data than
requested, the decoder raises an error.
* Requires the use of a file like object. If the original data is in
say a buffer, we need to "cast" it to e.g. a BytesIO to appease the
decoder.
In addition, the vendored CBOR decoder doesn't provide flexibility
that we desire. Specifically:
* It buffers indefinite length bytestrings instead of streaming them.
* It doesn't allow limiting the set of types that can be decoded. This
property is useful when implementing a "hardened" decoder that is
less susceptible to abusive input.
* It doesn't provide sufficient "hook points" and introspection to
institute checks around behavior. These are useful for implementing
a "hardened" decoder.
This all adds up to a reasonable set of justifications for writing our
own decoder.
So, this commit implements our own CBOR decoder.
At the heart of the decoder is a function that decodes a single "item"
from a buffer. This item can be a complete simple value or a special
value, such as "start of array." Using this function, we can build a
decoder that effectively iterates over the stream of decoded items and
builds up higher-level values, such as arrays, maps, sets, and indefinite
length bytestrings. And we can do this without performing I/O in the
decoder itself.
The core of the sans I/O decoder will probably not be used directly.
Instead, it is expected that we'll build utility functions for invoking
the decoder given specific input types. This will allow extreme
flexibility in how data is delivered to the decoder.
I'm pretty happy with the state of the decoder modulo the TODO items
to track wanted features to help with a "hardened" decoder. The one
thing I could be convinced to change is the handling of semantic tags.
Since we only support a single semantic tag (sets), I thought it would
be easier to handle them inline in decodeitem(). This is simpler now.
But if we add support for other semantic tags, it will likely be easier
to move semantic tag handling outside of decodeitem(). But, properly
supporting semantic tags opens up a whole can of worms, as many
semantic tags imply new types. I'm optimistic we won't need these in
Mercurial. But who knows.
I'm also pretty happy with the test coverage. Writing comprehensive
tests for partial decoding did flush out a handful of bugs. One
general improvement to testing would be fuzz testing for partial
decoding. I may implement that later. I also anticipate switching the
wire protocol code to this new decoder will flush out any lingering
bugs.
Differential Revision: https://phab.mercurial-scm.org/D4414
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:22:06 -0700] rev 39437
tests: remove use of string in CBOR test
We don't use the CBOR string major type in the wire protocol. Let's
not test for it.
Differential Revision: https://phab.mercurial-scm.org/D4413
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 20:27:36 -0700] rev 39436
internals: document CBOR utilization
I spoke with some people at Mozilla about CBOR and they advised me
that we should be careful about the subset of CBOR we use in order
to mitigate security, performance, and compatibility concerns.
This commit establishes a document that attempts to formalize our
use of CBOR.
Its main limitations are on what types are allowed. It explicitly
enumerates which types are supported. Notable missing features
include:
* Indefinite-length arrays and maps
* Text strings (bytes all the way)
* Floats
* Date/time types
* Big integers
* Use of indefinite-length byte strings for map keys, values in
containers.
If we have a need for any of these, we can have a discussion about
them when the time comes.
Differential Revision: https://phab.mercurial-scm.org/D4412