Wed, 11 Jan 2017 11:40:40 -0800 tests: use `hg help dates` instead of `hg help revs` in test
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:40:40 -0800] rev 30768
tests: use `hg help dates` instead of `hg help revs` in test The revisions help is already long and will get longer, so switch to another short and stable topic.
Wed, 11 Jan 2017 11:28:54 -0800 help: use a single paragraph to describe full and abbreviated nodeids
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:28:54 -0800] rev 30767
help: use a single paragraph to describe full and abbreviated nodeids The texts describing 40-digit strings and the abbreviated form are closely related, so make it a single paragraph.
Tue, 10 Jan 2017 23:37:08 -0800 hgweb: support Content Security Policy
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 23:37:08 -0800] rev 30766
hgweb: support Content Security Policy Content-Security-Policy (CSP) is a web security feature that allows servers to declare what loaded content is allowed to do. For example, a policy can prevent loading of images, JavaScript, CSS, etc unless the source of that content is whitelisted (by hostname, URI scheme, hashes of content, etc). It's a nifty security feature that provides extra mitigation against some attacks, notably XSS. Mitigation against these attacks is important for Mercurial because hgweb renders repository data, which is commonly untrusted. While we make attempts to escape things, etc, there's the possibility that malicious data could be injected into the site content. If this happens today, the full power of the web browser is available to that malicious content. A restrictive CSP policy (defined by the server operator and sent in an HTTP header which is outside the control of malicious content), could restrict browser capabilities and mitigate security problems posed by malicious data. CSP works by emitting an HTTP header declaring the policy that browsers should apply. Ideally, this header would be emitted by a layer above Mercurial (likely the HTTP server doing the WSGI "proxying"). This works for some CSP policies, but not all. For example, policies to allow inline JavaScript may require setting a "nonce" attribute on <script>. This attribute value must be unique and non-guessable. And, the value must be present in the HTTP header and the HTML body. This means that coordinating the value between Mercurial and another HTTP server could be difficult: it is much easier to generate and emit the nonce in a central location. This commit introduces support for emitting a Content-Security-Policy header from hgweb. A config option defines the header value. If present, the header is emitted. A special "%nonce%" syntax in the value triggers generation of a nonce and inclusion in <script> elements in templates. The inclusion of a nonce does not occur unless "%nonce%" is present. This makes this commit completely backwards compatible and the feature opt-in. The nonce is a type 4 UUID, which is the flavor that is randomly generated. It has 122 random bits, which should be plenty to satisfy the guarantees of a nonce.
Tue, 10 Jan 2017 20:47:48 -0800 hgweb: call process_dates() via DOM event listener
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 20:47:48 -0800] rev 30765
hgweb: call process_dates() via DOM event listener All the hgweb templates include mercurial.js in their header. All the hgweb templates have the same <script> boilerplate to run process_dates(). This patch factors that function call into mercurial.js as part of a DOMContentLoaded event listener.
Sat, 24 Dec 2016 15:29:32 -0700 protocol: send application/mercurial-0.2 responses to capable clients
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:29:32 -0700] rev 30764
protocol: send application/mercurial-0.2 responses to capable clients With this commit, the HTTP transport now parses the X-HgProto-<N> header to determine what media type and compression engine to use for responses. So far, we only compress responses that are already being compressed with zlib today (stream response types to specific commands). We can expand things to cover additional response types later. The practical side-effect of this commit is that non-zlib compression engines will be used if both ends support them. This means if both ends have zstd support, zstd - not zlib - will be used to compress data! When cloning the mozilla-unified repository between a local HTTP server and client, the benefits of non-zlib compression are quite noticeable: engine server CPU (s) client CPU (s) bundle size zlib (l=6) 174.1 283.2 1,148,547,026 zstd (l=1) 99.2 267.3 1,127,513,841 zstd (l=3) 103.1 266.9 1,018,861,363 zstd (l=7) 128.3 269.7 919,190,278 zstd (l=10) 162.0 - 894,547,179 none 95.3 277.2 4,097,566,064 The default zstd compression level is 3. So if you deploy zstd capable Mercurial to your clients and servers and CPU time on your server is dominated by "getbundle" requests (clients cloning and pulling) - and my experience at Mozilla tells me this is often the case - this commit could drastically reduce your server-side CPU usage *and* save on bandwidth costs! Another benefit of this change is that server operators can install *any* compression engine. While it isn't enabled by default, the "none" compression engine can now be used to disable wire protocol compression completely. Previously, commands like "getbundle" always zlib compressed output, adding considerable overhead to generating responses. If you are on a high speed network and your server is under high load, it might be advantageous to trade bandwidth for CPU. Although, zstd at level 1 doesn't use that much CPU, so I'm not convinced that disabling compression wholesale is worthwhile. And, my data seems to indicate a slow down on the client without compression. I suspect this is due to a lack of buffering resulting in an increase in socket read() calls and/or the fact we're transferring an extra 3 GB of data (parsing HTTP chunked transfer and processing extra TCP packets can add up). This is definitely worth investigating and optimizing. But since the "none" compressor isn't enabled by default, I'm inclined to punt on this issue. This commit introduces tons of tests. Some of these should arguably have been implemented on previous commits. But it was difficult to test without the server functionality in place.
Sat, 24 Dec 2016 15:22:18 -0700 httppeer: advertise and support application/mercurial-0.2
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:22:18 -0700] rev 30763
httppeer: advertise and support application/mercurial-0.2 Now that servers expose a capability indicating they support application/mercurial-0.2 and compression, clients can key off this to say they support responses that are compressed with various compression formats. After this commit, the HTTP wire protocol client now sends an "X-HgProto-<N>" request header indicating its support for "application/mercurial-0.2" media type and various compression formats. This commit also implements support for handling "application/mercurial-0.2" responses. It simply reads the header compression engine identifier then routes the remainder of the response to the appropriate decompressor. There were some test changes, but only to logging. That points to an obvious gap in our test coverage. This will be addressed in a subsequent commit once server support is in place (it is hard to test without server support).
Sat, 24 Dec 2016 15:21:46 -0700 wireproto: advertise supported media types and compression formats
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:21:46 -0700] rev 30762
wireproto: advertise supported media types and compression formats This commit introduces support for advertising a server's support for media types and compression formats in accordance with the spec defined in internals.wireproto. The bulk of the new code is a helper function in wireproto.py to obtain a prioritized list of compression engines available to the wire protocol. While not utilized yet, we implement support for obtaining the list of compression engines advertised by the client. The upcoming HTTP protocol enhancements are a bit lower-level than existing tests (most existing tests are command centric). So, this commit establishes a new test file that will be appropriate for holding tests around the functionality of the HTTP protocol itself. Rounding out this change, `hg debuginstall` now prints compression engines available to the server.
Sat, 24 Dec 2016 13:51:12 -0700 util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 13:51:12 -0700] rev 30761
util: declare wire protocol support of compression engines This patch implements a new compression engine API allowing compression engines to declare support for the wire protocol. Support is declared by returning a compression format string identifier that will be added to payloads to signal the compression type of data that follows and default integer priorities of the engine. Accessor methods have been added to the compression engine manager class to facilitate use. Note that the "none" and "bz2" engines declare wire protocol support but aren't enabled by default due to their priorities being 0. It is essentially free from a coding perspective to support these compression formats, so we do it in case anyone may derive use from it.
(0) -30000 -10000 -3000 -1000 -300 -100 -30 -10 -8 +8 +10 +30 +100 +300 +1000 +3000 +10000 tip