Augie Fackler <augie@google.com> [Fri, 07 Sep 2018 11:35:43 -0400] rev 39484
merge with stable
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 15:16:22 +0300] rev 39483
tests: improve the widening testing in test-narrow-widen*
Before this patch, we are testing `hg tracked --addinclude` by adding a command
which is not introduced in the changesets till now.
If you closely look at the tests, wider/f was introduced on the server after the
narrow clone was done and extending the existing clone to include wider/f does
not make sense. We should test extending a file which exists.
Differential Revision: https://phab.mercurial-scm.org/D4452
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 19:26:50 +0300] rev 39482
narrow: use util.readfile() and improve error message using --narrowspec
This patch improves the error message and uses util.readfile() for reading
narrowspecs file while cloning.
Differential Revision: https://phab.mercurial-scm.org/D4462
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 04 Sep 2018 15:55:23 -0700] rev 39481
merge: use vfs methods for I/O
All I/O is supposed to be performed via vfs instances so filesystems
can be abstracted. The previous commit ported the old code in purge,
which didn't go through the vfs layer. This commit ports the purge
code to use the vfs layer.
The vfs layer didn't have a method to remove a single directory, so
it was added as part of implementing this.
Differential Revision: https://phab.mercurial-scm.org/D4478
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 06 Sep 2018 18:30:12 -0700] rev 39480
merge: move purge logic from extension
Working directory purging feels like functionality that should be
in core rather than in an extension.
This commit effectively moves the core purging logic from the
purge extension to merge.py.
Code was refactored slightly. Rather than deal with printing in
this function, the function is a generator of paths and the caller
can worry about printing.
Differential Revision: https://phab.mercurial-scm.org/D4477
Matt Harbison <matt_harbison@yahoo.com> [Thu, 06 Sep 2018 23:37:24 -0400] rev 39479
tests: stabilize test-removeemptydirs.t on Windows
Yuya Nishihara <yuya@tcha.org> [Thu, 06 Sep 2018 21:55:30 +0900] rev 39478
help: add internals.wireprotocolv2 to the table, and remove redundant header
Kyle Lippincott <spectral@google.com> [Fri, 17 Aug 2018 19:18:53 -0700] rev 39477
match: improve includematcher.visitchildrenset to be much faster and cached
This improves the speed of visitchildrenset considerably, especially when there
are complicated matchers involved that may have many entries in _dirs or
_parents.
Unfortunately the benchmark isn't easily upstreamed due to its reliance on
https://github.com/vstinner/perf (primarily due to the conflict when importing
it if I were to contribute the benchmark as contrib/matcherbenchmarks.py)
instead of asv or some other perf measurement system.
To describe the benchmark briefly: I generated an includematcher of either 5 or
3500 "rootfilesin:prefix1/prefix2/prefix3/<randomsubdirs, 1-8 levels deep>"
items in the 'setup' function, and then called
`im.visitchildrenset('prefix1/prefix2')` in the 'stmt' function in perf.timeit.
For the set of 5:
- before: 15.3 us +- 2.9 us
- after: 1.59 us +- 0.02 us
For the set of 3500:
- before: 3.90 ms +- 0.10 ms
- after: 3.15 us +- 0.09 us (note the m->u change)
Differential Revision: https://phab.mercurial-scm.org/D4351
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 06 Sep 2018 03:21:05 +0530] rev 39476
py3: add new passing tests spotted by the buildbot
Differential Revision: https://phab.mercurial-scm.org/D4495
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 06 Sep 2018 03:24:27 +0530] rev 39475
tests: order the imports in test-fastannotate-hg.t
The wrong ordering breaks test-check-module-imports.t on Python 3. I am not sure
why that test is so much active on py3.
Differential Revision: https://phab.mercurial-scm.org/D4496
Matt Harbison <matt_harbison@yahoo.com> [Thu, 06 Sep 2018 00:51:21 -0400] rev 39474
lfs: ensure the blob is linked to the remote store on skipped uploads
I noticed a "missing" blob when pushing two repositories with common blobs to a
fresh server, and then running `hg verify` as a user different from the one
running the web server. When pushing the second repo, several of the blobs
already existed in the user cache, so the server indicated to the client that it
doesn't need to upload the blobs. That's good enough for the web server process
to serve up in the future. But a different user has a different cache by
default, so verify complains that `lfs.url` needs to be set, because it wants to
fetch the missing blobs.
Aside from that corner case, it's better to keep all of the blobs in the repo
whenever possible. Especially since the largefiles wiki says the user cache can
be deleted at any time to reclaim disk space- users switching over may have the
same expectations.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 04 Sep 2018 22:29:38 -0400] rev 39473
cbor: teach the encoder to handle python `long` type for Windows
The tests for 2**32 and -7000000000 were blowing up, complaining about not
knowing how to encode type 'long'. sys.maxint tops out at 2**31-1 on Windows,
but I guess is 2^63-1 on Linux? I *think* we're OK on the decode side, as there
is an assertion that the decoded value is equal to the original primitive value.
I opted for the pycompat alias instead of swallowing the NameError because the
vendored cbor package uses an alias, and I see at least pywatchman and
templatefilters open codes their own aliases.
Matt Harbison <matt_harbison@yahoo.com> [Tue, 04 Sep 2018 20:48:22 -0400] rev 39472
absorb: clarify the reason for not finding changesets to modify
I'm used to pushing to non-publishing repos, so this was the last thing I
thought to check.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 05 Sep 2018 09:36:31 -0700] rev 39471
merge with stable
Includes an extra bump of the version number for parsers because the
merge produces a new, distinct version of the code.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 04 Sep 2018 10:22:42 -0700] rev 39470
state: use our CBOR module
This was the last consumer of the vendored CBOR package in
core.
Differential Revision: https://phab.mercurial-scm.org/D4471
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:41:09 -0700] rev 39469
wireprotov2peer: use our CBOR decoder
Behavior should be the same in order to preserve backwards
compatibility. We obviously want to stream values in this code.
We'll do that in subsequent commits.
Differential Revision: https://phab.mercurial-scm.org/D4470
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:37:55 -0700] rev 39468
debugcommands: use our CBOR decoder
It implements the set of CBOR needed for the wire protocol.
Differential Revision: https://phab.mercurial-scm.org/D4469
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 18:12:04 -0700] rev 39467
wireprotov2server: use our CBOR encoder
Again, test output changed slightly because of map key ordering
differences. This shouldn't matter.
I could have called oncommandresponsereadygen() with the raw output
from the stream encoder. However, this changed test output further.
I left a TODO to follow up on that later.
Differential Revision: https://phab.mercurial-scm.org/D4468
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 18:05:48 -0700] rev 39466
wireprotoserver: use our CBOR encoder
Moving away from the vendored package.
Again, our encoder normalizes keys differently from the vendored
package (for now), hence the test changes.
Differential Revision: https://phab.mercurial-scm.org/D4467
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 18:05:08 -0700] rev 39465
wireprotoframing: use our CBOR module
Tests changed because our CBOR encoder appears to sort map keys
differently from the vendored CBOR package. The CBOR specification
does define canonical sorting rules for keys based on the
byte values. I'm guessing our implementation doesn't follow them.
But our encoder doesn't guarantee that it conforms with the canonical
specification. Right now, we just care that output is deterministic.
And our encoder does guarantee that.
Differential Revision: https://phab.mercurial-scm.org/D4466
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:10:56 -0700] rev 39464
httppeer: use our CBOR decoder
We just implemented our own CBOR decoder. Let's use it in
httppeer.
Differential Revision: https://phab.mercurial-scm.org/D4465
Anton Shestakov <av6@dwimlabs.net> [Wed, 22 Aug 2018 09:02:07 +0800] rev 39463
zsh_completion: complete shelve and unshelve
Differential Revision: https://phab.mercurial-scm.org/D4428
Anton Shestakov <av6@dwimlabs.net> [Tue, 21 Aug 2018 20:30:53 +0800] rev 39462
zsh_completion: use revsets to exclude this rev from suggestions to hg merge
One of the most important aspects of a completion system is its speed, so 1
call to hg is definitely better than 4.
Sorting by rev (descending) is to preserve the same order as in `hg heads`
output.
While at it, declare branches as an array too.
Differential Revision: https://phab.mercurial-scm.org/D4426
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 Sep 2018 10:36:34 -0700] rev 39461
drawdag: correctly pass repo to super constructor
This was caught by someone who was looking for the
super(cls, self).__init__(self, ...)
pattern in all code in our Google-internal repo.
Differential Revision: https://phab.mercurial-scm.org/D4464
Matt Harbison <matt_harbison@yahoo.com> [Tue, 04 Sep 2018 20:54:41 -0400] rev 39460
help: correct a typo in tweakdefaults
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 06 Aug 2018 14:06:19 +0300] rev 39459
narrow: add '--import-rules' flag to tracked command
This patch adds a `--import-rules` flag to tracked command provided by narrow
extension. Using the --import-rules flag, you can pass a filename from which
narrowspecs should be read and added to main narrowspec.
A lot of times, in automation or manually also, when you are working with big
repo, specifying each path name on commandline using '--addinclude' and
'--addexclude' is tedious and something which can scale. So we needed something
where we can pass a file to extend the narrowspecs.
Nice thing about this is that the automations which reads some file to change
the sparse profile, can now read the same file for changing narrowspecs too.
Tests are added for the new feature.
Differential Revision: https://phab.mercurial-scm.org/D4125
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 23 Aug 2018 13:11:13 -0700] rev 39458
internals: extract wire protocol version 2 commands to standalone doc
wireprotocol.txt is a bit long and unwieldy. I think splitting it up
will help aid comprehension.
Let's start by extracting wire protocol version 2 commands to a
standalone document.
As part of the port, I munged with the section titles a bit and
expanded the TODO around node namespaces.
Differential Revision: https://phab.mercurial-scm.org/D4442
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 23 Aug 2018 13:46:39 -0700] rev 39457
wireprotov2peer: split responsedata handling into separate function
So we don't have so much logic inside an if/elif block.
Differential Revision: https://phab.mercurial-scm.org/D4439
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 22 Aug 2018 10:25:47 -0700] rev 39456
httppeer: add TODO about delayed handling of ^C
When testing wire protocol v2 with large responses, I noticed that
^C doesn't result in immediate termination. I'm not sure what the
root cause is. Let's add a TODO so we don't forget about it.
Differential Revision: https://phab.mercurial-scm.org/D4438
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 31 Aug 2018 08:57:14 -0700] rev 39455
httppeer: log commands for version 2 peer
This will aid testing and debugging.
Differential Revision: https://phab.mercurial-scm.org/D4437
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 22 Aug 2018 10:29:30 -0700] rev 39454
wireprotov2peer: add TODO about streaming responses
This is a pretty big deficiency in the current implementation.
We don't want to forget about this.
Differential Revision: https://phab.mercurial-scm.org/D4435
Augie Fackler <augie@google.com> [Tue, 04 Sep 2018 12:36:33 -0400] rev 39453
tests: update relnotes test for actual output
I'm unclear why I didn't see this before but I consistently see it
now. In any event, it doesn't appear to be a regression, so let's just
bake it into the test.
Differential Revision: https://phab.mercurial-scm.org/D4463
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:57:49 +0300] rev 39452
py3: use print as a function in tests/test-hgweb.t
Differential Revision: https://phab.mercurial-scm.org/D4461
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:58:39 +0300] rev 39451
py3: bytestr a bytes val to make sure we get bytechr while iterating
Differential Revision: https://phab.mercurial-scm.org/D4460
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:53:54 +0300] rev 39450
py3: make regexes in hgext/keyword.py bytes
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4459
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:33:59 +0300] rev 39449
py3: add more missing b'' prefixes in test files
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4458
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:33:22 +0300] rev 39448
py3: write the file in str mode in tests/test-rename-merge2.t
This is better then converting the value the value of sys.argv to bytes.
This makes the test pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4457
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:32:09 +0300] rev 39447
py3: add missing b'' prefixes in tests/badserverext.py
# skip-blame because just b'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D4456
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:19:30 +0300] rev 39446
py3: handle keyword arguments correctly in hgext/acl.py
The keys of keyword arguments on python 3 should be str, so when we try to get
some key from them, we must make sure we are using str.
# skip-blame because just b'' prefix
Differential Revision: https://phab.mercurial-scm.org/D4455
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:16:29 +0300] rev 39445
py3: don't return the revid as unicode in hgext/convert/subversion.py
I tried digging why u'' was added in first place, and I was unable to found
something relevant. This might be because some API's takes unicodes, I am not
sure.
Differential Revision: https://phab.mercurial-scm.org/D4454
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 04 Sep 2018 17:15:17 +0300] rev 39444
py3: make sure we pass str in os.sysconf in hgext/convert/common.py
# skip-blame because just r'' prefix
Differential Revision: https://phab.mercurial-scm.org/D4453
Augie Fackler <augie@google.com> [Tue, 04 Sep 2018 12:16:28 -0400] rev 39443
merge with stable
Yuya Nishihara <yuya@tcha.org> [Tue, 04 Sep 2018 13:29:21 +0900] rev 39442
revlog: fix size of Python nodetree object
Follows up 9f097214fbf3.
Yuya Nishihara <yuya@tcha.org> [Mon, 03 Sep 2018 23:03:19 +0900] rev 39441
revert: stabilize status message of chunks selected interactively
Unfortunately, patch.filterpatch() doesn't preserve the order of the input
files. We have to sort them manually.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 29 Aug 2018 14:29:01 -0700] rev 39440
cborutil: add a buffering decoder
The sansiodecoder leaves it up to the callers to feed in data that
wasn't fully consumed last time.
This commit implements a decoder that performs buffering of
leftover chunks from the previous invocation. It otherwise
behaves identically to sansiodecoder.
Differential Revision: https://phab.mercurial-scm.org/D4434
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 31 Aug 2018 15:54:17 -0700] rev 39439
cborutil: remove readindefinitebytestringtoiter()
This was implemented as part of implementing streaming encoding.
It was never used outside of tests.
Now that we have a full CBOR decoder, it can be used for incremental
decoding of indefinite-length byte strings.
This also removes the last use of the vendored cbor2 package from this
module.
Differential Revision: https://phab.mercurial-scm.org/D4433
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:02:48 -0700] rev 39438
cborutil: implement sans I/O decoder
The vendored CBOR package decodes by calling read(n) on an object.
There are a number of disadvantages to this:
* Uses blocking I/O. If sufficient data is not available, the decoder
will hang until it is.
* No support for partial reads. If the read(n) returns less data than
requested, the decoder raises an error.
* Requires the use of a file like object. If the original data is in
say a buffer, we need to "cast" it to e.g. a BytesIO to appease the
decoder.
In addition, the vendored CBOR decoder doesn't provide flexibility
that we desire. Specifically:
* It buffers indefinite length bytestrings instead of streaming them.
* It doesn't allow limiting the set of types that can be decoded. This
property is useful when implementing a "hardened" decoder that is
less susceptible to abusive input.
* It doesn't provide sufficient "hook points" and introspection to
institute checks around behavior. These are useful for implementing
a "hardened" decoder.
This all adds up to a reasonable set of justifications for writing our
own decoder.
So, this commit implements our own CBOR decoder.
At the heart of the decoder is a function that decodes a single "item"
from a buffer. This item can be a complete simple value or a special
value, such as "start of array." Using this function, we can build a
decoder that effectively iterates over the stream of decoded items and
builds up higher-level values, such as arrays, maps, sets, and indefinite
length bytestrings. And we can do this without performing I/O in the
decoder itself.
The core of the sans I/O decoder will probably not be used directly.
Instead, it is expected that we'll build utility functions for invoking
the decoder given specific input types. This will allow extreme
flexibility in how data is delivered to the decoder.
I'm pretty happy with the state of the decoder modulo the TODO items
to track wanted features to help with a "hardened" decoder. The one
thing I could be convinced to change is the handling of semantic tags.
Since we only support a single semantic tag (sets), I thought it would
be easier to handle them inline in decodeitem(). This is simpler now.
But if we add support for other semantic tags, it will likely be easier
to move semantic tag handling outside of decodeitem(). But, properly
supporting semantic tags opens up a whole can of worms, as many
semantic tags imply new types. I'm optimistic we won't need these in
Mercurial. But who knows.
I'm also pretty happy with the test coverage. Writing comprehensive
tests for partial decoding did flush out a handful of bugs. One
general improvement to testing would be fuzz testing for partial
decoding. I may implement that later. I also anticipate switching the
wire protocol code to this new decoder will flush out any lingering
bugs.
Differential Revision: https://phab.mercurial-scm.org/D4414
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 15:22:06 -0700] rev 39437
tests: remove use of string in CBOR test
We don't use the CBOR string major type in the wire protocol. Let's
not test for it.
Differential Revision: https://phab.mercurial-scm.org/D4413
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 28 Aug 2018 20:27:36 -0700] rev 39436
internals: document CBOR utilization
I spoke with some people at Mozilla about CBOR and they advised me
that we should be careful about the subset of CBOR we use in order
to mitigate security, performance, and compatibility concerns.
This commit establishes a document that attempts to formalize our
use of CBOR.
Its main limitations are on what types are allowed. It explicitly
enumerates which types are supported. Notable missing features
include:
* Indefinite-length arrays and maps
* Text strings (bytes all the way)
* Floats
* Date/time types
* Big integers
* Use of indefinite-length byte strings for map keys, values in
containers.
If we have a need for any of these, we can have a discussion about
them when the time comes.
Differential Revision: https://phab.mercurial-scm.org/D4412
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 03 Sep 2018 13:56:53 +0300] rev 39435
py3: add new passing test to whitelist found by buildbot
Differential Revision: https://phab.mercurial-scm.org/D4451
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 03 Sep 2018 13:53:30 +0300] rev 39434
py3: alias xrange to range in tests/test-fastannotate-revmap.py
xrange is not available on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D4450
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 03 Sep 2018 13:52:49 +0300] rev 39433
py3: add r'' prefix to prevent b'' being prepended
The keys of keyword arguments should be str.
# skip-blame because we are adding just r'' prefixes.
Differential Revision: https://phab.mercurial-scm.org/D4449
Sushil khanchi <sushilkhanchi97@gmail.com> [Fri, 31 Aug 2018 23:28:09 +0530] rev 39432
revert: fix the inconsistency of status msgs in --interactive mode
Before this patch we were priting every action msg before actually
performing that action and that was resulting in inconsistencies;
like in --interactive session if user decided to not revert any
changes in a file foo, still there will be a msg on console saying
"reverting foo".
To fix this, I have made some changes to print status msg just
before the action it is going to perform, no matter if --interactive
or not.
Changes made in test-revert-interactive.t reflect the changed behavior.
There are also some changes in test-revert.t because of change in the
order of messages.
Differential Revision: https://phab.mercurial-scm.org/D4380
Sushil khanchi <sushilkhanchi97@gmail.com> [Mon, 03 Sep 2018 10:44:52 +0530] rev 39431
revert: remove unnecessary check as `msg` should never be a callable now
Differential Revision: https://phab.mercurial-scm.org/D4448
Yuya Nishihara <yuya@tcha.org> [Sun, 02 Sep 2018 16:32:11 +0900] rev 39430
run-tests: drop support for --with-python3 and $PYTHON3
This backs out cdbc25306696. We can run many tests on python3 without
this hack.
Yuya Nishihara <yuya@tcha.org> [Sun, 02 Sep 2018 16:26:33 +0900] rev 39429
test-check-py3-compat: don't run py3 part if not build against python3
This test depends on C extensions built for python3, which aren't available
if run-tests.py runs on python2 and no -l/--local is specified.