exchangev2: use filesdata filesdata is a more efficient mechanism for bulk fetching files data for a range of changesets. Let's use it in exchangev2. With this change, a client performing a full clone of mozilla-unified transmits substantially fewer bytes across the wire: before: 139,124,863 bytes sent after: 20,522,499 bytes sent The bulk of the remaining bytes is likely the transfer of ~1M nodes for changesets and manifests. We can eliminate this by making requests in terms of node ranges instead of explicit node lists... Differential Revision: https://phab.mercurial-scm.org/D4982

wireprotov2: define and implement "filesdata" command Previously, the only way to access file revision data was the "filedata" command. This command is useful to have. But, it only allowed resolving revision data for a single file. This meant that clients needed to send 1 command for each tracked path they were seeking data on. Furthermore, those commands would need to enumerate the exact file nodes they wanted data for. This approach meant that clients were sending a lot of data to remotes in order to request file data. e.g. if there were 1M file revisions, we'd need at least 20,000,000 bytes just to encode file nodes! Many clients on the internet don't have that kind of upload capacity. In order to limit the amount of data that clients must send, we'll need more efficient ways to request repository data. This commit defines and implements a new "filesdata" command. This command allows the retrieval of data for multiple files by specifying changeset revisions and optional file patterns. The command figures out what file revisions are "relevant" and sends them in bulk. The logic around choosing which file revisions to send in the case of haveparents not being set is overly simple and will over-send files. We will need more smarts here eventually. (Specifically, the client will need to tell the server which revisions it knows about.) This work is deferred until a later time. Differential Revision: https://phab.mercurial-scm.org/D4981

wireprotov2: extract file object emission to own function An upcoming commit will introduce another caller. Differential Revision: https://phab.mercurial-scm.org/D4980

wireprotov2: change how revisions are specified to changesetdata Right now, we have a handful of arguments for specifying the revisions whose data should be returned. Defining how all these arguments interact when various combinations are present is difficult. This commit establishes a new, generic mechanism for specifying revisions. Instead of a hodgepodge of arguments defining things, we have a list of dicts that specify revision selectors. The final set of revisions is a union of all these selectors. We implement support for specifying revisions based on: * An explicit list of changeset revisions * An explicit list of changeset revisions plus ancestry depth * A DAG range between changeset roots and heads If you squint hard enough, this problem has already been solved by revsets. But I'm reluctant to expose revsets to the wire protocol because that would require servers to implement a revset parser. Plus there are security and performance implications: the set of revision selectors needs to be narrowly and specifically tailored for what is appropriate to be executing on a server. Perhaps there would be a way for us to express the "parse tree" of a revset query, for example. I'm not sure. We can explore this space another time. For now, the new mechanism should bring sufficient flexibility while remaining relatively simple. The selector "types" are prefixed with "changeset" because I plan to add manifest and file-flavored selectors as well. This will enable us to e.g. select file revisions based on a range of changeset revisions. Differential Revision: https://phab.mercurial-scm.org/D4979

wireprotov2: stop sending phase updates for base revisions This feature is broken and doesn't work properly in all scenarios. e.g. if we have the following DAGs: client server D draft C draft C draft B draft B public A public A public The current code would only send the phase data for C. The client wouldn't see that B moved from draft to public. This feature will be restored in a future commit. For now, it is making refactoring of how revisions are specified in the wire protocol a bit difficult... Differential Revision: https://phab.mercurial-scm.org/D4978

debugcommands: support wrapping long lines If a line within a block is indented more than the line that came before, we automatically concatenate it with the previous line. This allows us to pretty format data. This will make tests easier to read. At some point we may just want to evaluate entire blocks as Python code or something, as even with this change, things aren't perfect, as we can't e.g. have formatting like: foo eval:[ True ] But this is strictly better than before, where we couldn't wrap long lines. Differential Revision: https://phab.mercurial-scm.org/D4977

exchangev2: honor server advertised manifestdata recommended batch size Let's plug the client up to the server-advertised recommended batch size for manifestdata requests. Differential Revision: https://phab.mercurial-scm.org/D4976

wireprotov2: advertise recommended batch size for requests Currently, exchangev2 hardcodes the batch size for how many revisions to fetch per command request. A single value is not appropriate for every repository because some repositories may have a drastically different "shape" from other repositories. e.g. a repo with lots of small files may benefit from larger batch sizes than a repo with lots of large files. And depending on caching used by the server, the server may wish to control the number of commands (to e.g. mitigate overhead of following content redirects). This commit teaches wireprotov2 commands to declare extra metadata which is advertised as part of the command descriptor. The manifestdata command has been taught to advertise a recommended batch size for requests. Differential Revision: https://phab.mercurial-scm.org/D4975

httppeer: expose API descriptor on httpv2peer The API descriptor in wireprotov2 is much more expressive than space-delimited tokens and it will be difficult to define methods to query it in all of the ways we'll want to query it. So let's just declare defeat and expose the API descriptor on the peer instance. As part of this, we define a new interface for version 2 peers, fulfilling a TODO in the process. Differential Revision: https://phab.mercurial-scm.org/D4974

tests: use baseurl instead of advertisedbaseurl The distinction matters for e.g. hosts behind load balancers. But for the test environment, it doesn't matter. For whatever reason, advertisedbaseurl is resolving to http://1.0.0.127.in-addr.arpa:$HGPORT on my MBP. This hostname fails to resolve, causing the test to fail. No clue what's up with that behavior. Differential Revision: https://phab.mercurial-scm.org/D4973

py3: another one started passing Differential Revision: https://phab.mercurial-scm.org/D4990

py3: one more passing test Differential Revision: https://phab.mercurial-scm.org/D4989

scmutil: wrap locker information in bytestr before repr()ing it Differential Revision: https://phab.mercurial-scm.org/D4988

py3: more passing tests Differential Revision: https://phab.mercurial-scm.org/D4987

py3: whitelist another passing test Differential Revision: https://phab.mercurial-scm.org/D4986

obsolete: convert error string to a sysstr This should be okay because we're just %-formatting a hash into a localized string. Differential Revision: https://phab.mercurial-scm.org/D4985

tests: ensure the python 3 passing test list stays sorted Differential Revision: https://phab.mercurial-scm.org/D4984

py3: sort passing test list Differential Revision: https://phab.mercurial-scm.org/D4983

py3: tweak stdout writing in test-hgweb-no-path-info.t We want to write bytes for convenience. This requires sys.stdout.buffer. But using sys.stdout.buffer introducing buffered output. So we sprinkle code with sys.stdout.flush() to force immediate writes. After all that, Python 3 was emitting b'' prefixed output for errors. So we only print errors if there were some. There aren't, so b'' don't come into play and output is identical in Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D4972

py3: use .+ instead of .* in regexp pattern Python 3.7 changed the behavior of re.sub(). See https://bugs.python.org/issue33585. The new code should work on old and new Pythons. Differential Revision: https://phab.mercurial-scm.org/D4971

py3: coerce bytestr to bytes to appease urllib.parse.quote_from_bytes() Differential Revision: https://phab.mercurial-scm.org/D4969

py3: pass str and return bytes from mimetypes.guess_type() This function wants a str (which represents a path) and returns a str. We normalize input to str and output to bytes. Differential Revision: https://phab.mercurial-scm.org/D4967

py3: use bytes literal in test-hgweb-json.t # skip-blame just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4970

py3: convert diff opcode name to bytes Differential Revision: https://phab.mercurial-scm.org/D4968

py3: byteify hgweberror.py # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4966

py3: encode JSON str to bytes json.dumps() will emit UTF-8 str on Python 2 and 3. Use sysbytes to force the .encode('utf-8') on Python 3 and no-op on Python 2. Differential Revision: https://phab.mercurial-scm.org/D4965

py3: use b'' in test-check-interfaces.py # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4964

revlog: optimize ancestors() to not check filtered revisions for each While reviewing the Rust implementation, I noticed iter(ancestors) doesn't need to check filtering state for each parent revision. And doing that appears to have some measurable perf win. $ hg perfancestors -R mercurial (orig) wall 0.038093 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) (this) wall 0.024795 comb 0.020000 user 0.020000 sys 0.000000 (best of 117)

phabricator: drop support for the legacy phabricator.auth.token config (BC) The test for this broke in dc82ad1b7f77 when statistics started being tracked. It wasn't noticed because none of the bots have the vcr module installed. It looks like the custom_patches argument should patch in the custom httpconnection, and I can't figure out what is going on.

absorb: print '{rev}:' as a prefix to the hash I don't see a sane way to print the summary to identify the changed commit, but this at least makes it a bit easier to identify a commit when a group of them changes.

packaging: fix "make centos{5,6,7}" Without this change, make centos{5,6,7} fails with error: cp: cannot stat '<basedir>/rpmbuild/RPMS/*/*': No such file or directory This change is the exact equivalent of bc4bbc42899a (which applied to Fedora).

narrow: don't compress the bundle2 when sending 'error:abort' This is similar to waht getbundle() does and also explicitly specifies that we should get a compressed bundle2 in normal cases when not sending 'error:abort'. Differential Revision: https://phab.mercurial-scm.org/D4934

push: add "remote" to 'repository changed while pushing' messages (issue5971) Differential Revision: https://phab.mercurial-scm.org/D4933

fuzz: try setting PYTHONNOUSERSITE=1 to avoid loading site-packages Right now the fuzzer is crashing trying to look up the home dir for uid 0, which is breaking in the fuzz environment. Differential Revision: https://phab.mercurial-scm.org/D4936

perf: extract result formatting in its own function This will make it easier to reuse in another changesets.

perf: extract the timing of a section in a context manager This makes it easier to reuse it in other (future) part of the code that requires their own time management.

pref: support negative indexing in perfrevlogrevisions This is useful to check for a consistent final slice in multiple different repositories.

perf: accept formatter option for perfmanifest

perf: fix -T json The previous code was mixing formatting and data, breaking `-T json` with unexpected data. We fix the issue and add a test to prevent future regression.

formatter: more details on assertion failure This is useful when the assertion fails.

wireprotov2: raise ProgrammingError on unknown action Suggested by @durin42 in review of D4923. Differential Revision: https://phab.mercurial-scm.org/D4935

wireprotov2: send content encoded frames from server Now that we have support for negotiating encodings and configuring an encoder, we can start sending content encoded frames from the server. This commit teaches the wireprotov2 server code to send content encoded frames. On the mozilla-unified repository with zstd enabled peers, this change reduces the total amount of data transferred from server to client drastically: befor: 7,190,995,812 bytes after: 1,605,508,691 bytes Differential Revision: https://phab.mercurial-scm.org/D4927

wireprotov2: raise exception in objects() if future has been resolved Differential Revision: https://phab.mercurial-scm.org/D4926

wireprotov2: don't emit empty frames Staring at logs revealed the presence of empty frames that should have contained payload. Let's stop that from happening. Differential Revision: https://phab.mercurial-scm.org/D4925

wireprotov2: remove functions for creating response frames from bytes All code in the actual server uses oncommandresponsereadyobjects(). Test code was ported to that method. This resulted in a handful of subtle test changes. Differential Revision: https://phab.mercurial-scm.org/D4924

wireprotov2: handle noop action This action can be returned from the client reactor. We should handle it. Differential Revision: https://phab.mercurial-scm.org/D4923

wireprotov2: send protocol settings frame from client Now that we have client and server reactor support for protocol settings and encoding frames, we can start to send them out over the wire! This commit teaches the client reactor to send out a protocol settings frame when needed. The httpv2 peer has been taught to gather a list of supported content encoders and to advertise them through the client reactor. Because the client is now sending new frame types by default, this constitutes a compatibility break in the framing protocol. The media type version has been bumped accordingly. This will ensure existing clients won't attempt to send the new frames to old servers not supporting this explicit media type. I'm not bothering with the BC annotation because everything wireprotov2 is highly experimental and nobody should be running a server yet. Differential Revision: https://phab.mercurial-scm.org/D4922

wireprotov2: define and use stream encoders Now that we have basic support for defining stream encoding, it is time to start doing something with it. We define various classes implementing stream encoders/decoders for the defined encoding profiles. This is relatively straightforward. We teach the inputstream and outputstream classes how to encode, decode, and flush data. We then teach the clientreactor how to filter received data through the inputstream decoder. One of the features of the framing format is that streams can span requests. This is a differentiating feature from say HTTP/2, which associates streams with requests. By allowing streams to span requests, we can reuse compression context data across requests/responses. But in order to do this, we need a mechanism to "flush" the encoder at logical boundaries so that receivers receive all data where it is expected. And a "flush" event is distinct from a "finish" event from the perspective of certain compressors because a "flush" will retain compression context state whereas a "finish" operation will not. This is why encoders have both a flush() and a finish() and each uses specific flushing semantics on the underlying compressor. The added tests verify various behavior of decoders via clientreactor. These tests do test some compression behavior via use of outputstream. But for all intents and purposes, server reactor support for encoding is not yet implemented. Differential Revision: https://phab.mercurial-scm.org/D4921

lfs: avoid a potential variable reference before assignment error in cmdserver A coworker hit this once yesterday when pulling in thg (a retry worked), and then I hit it with strip after a pull. I had a difficult time recreating a test for this (at least one of the tricks was to not use '-R', which seems to cause reposetup() to be called for each command), so I'm not sure how large of a window there actually is for this. Calling reposetup() *after* the requirement is added will skip the hook entirely. The other issue I had was adding a couple `ui.status()` lines around the check that installs the hook. On Windows, the cmdserver process ballooned to 1.6GB and hung. Changing that to `ui.warn()` avoided the hang. It also hung on macOS, but without the large memory usage.

wireprotov2: establish dedicated classes for input and output streams Streams are unidirectional. As part of implementing encoding/decoding support, it became clear that it didn't make sense for a generic "stream" class to hold functionality related to both encoding and decoding. So we create new classes to represent the flavor of stream. Differential Revision: https://phab.mercurial-scm.org/D4920

wireprotov2: pass ui into clientreactor and serverreactor This will allow us to use config options to influence compression settings. Differential Revision: https://phab.mercurial-scm.org/D4919

wireprotov2: handle stream encoding settings frames Like what we just did for the server reactor, we teach the client reactor to handle stream encoding settings frames. The code is very similar. We define a method on the stream class to handle processing the data within the decoded frames. However, it doesn't yet do anything useful. Differential Revision: https://phab.mercurial-scm.org/D4918

wireprotov2: document client reactor actions We should document these so consumers have an easier life. Differential Revision: https://phab.mercurial-scm.org/D4917

wireprotov2: handle sender protocol settings frames We teach the server reactor to handle the optional sender protocol settings frames, which can only be sent at the beginning of frame exchange. Right now, we simply decode the data and record the sender protocol settings on the server reactor instance: we don't yet do anything meaningful with the data. Differential Revision: https://phab.mercurial-scm.org/D4916

wireprotov2: update stream encoding specification The encoding of data within streams in the frame-based protocol is not yet defined or implemented. This means that all data in wire protocol version 2 is currently being sent out raw, without compression. That's obviously not ideal. This commit formalizes the beginnings of stream encoding support in the protocol. I suspect we'll change behavior substantially in the future. My goal is to get something landed so we can use compression. We can build out more robust support later. Because the frame type ID changed, this is strictly BC. But existing code wasn't using the frame. I'll bump the framing protocol version later once code is introduced to use the new frame. Differential Revision: https://phab.mercurial-scm.org/D4915

cborutil: cast bytearray to bytes This code didn't like passing in bytearray instances. Let's cast bytearray to bytes so it works. Differential Revision: https://phab.mercurial-scm.org/D4914

tests: disable zstd in test This makes the test pass in pure installs. Differential Revision: https://phab.mercurial-scm.org/D4913

wireprotov2: remove "compression" from capabilities response This is not used. And future commits will change how this mechanism works. Let's remove it. As a bonus, this fixes some test failures on pure installs (due to zstd references). Differential Revision: https://phab.mercurial-scm.org/D4912

zstandard: vendor python-zstandard 0.10.1 This was just released. The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. setup.py was updated to pass a new argument to python-zstandard's function for returning an Extension instance. Upstream had to change to use relative paths because Python 3.7's packaging doesn't seem to like absolute paths when defining sources, includes, etc. The default relative path calculation is relative to setup_zstd.py which is different from the directory of Mercurial's setup.py. The project contains a vendored copy of zstandard 1.3.6. The old version was 1.3.4. The API should be backwards compatible and nothing in core should need adjusted. However, there is a new "chunker" API that we may find useful in places where we want to emit compressed chunks of a fixed size. There are a pair of bug fixes in 0.10.0 with regards to compressobj() and decompressobj() when block flushing is used. I actually found these bugs when introducing these APIs in Mercurial! But existing Mercurial code is not affected because we don't perform block flushing. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D4911

rust-chg: install signal handlers to forward signals to server I use sync::Once as a synchronization primitive because it's quite easy to use, and is good enough to prevent data race in these C functions.