Fri, 03 Feb 2017 15:10:27 -0800 util: always force line buffered stdout when stdout is a tty (BC)
Simon Farnsworth <simonfar@fb.com> [Fri, 03 Feb 2017 15:10:27 -0800] rev 30876
util: always force line buffered stdout when stdout is a tty (BC) pager replaced stdout with a line buffered version to work around glibc deciding on a buffering strategy on the first write to stdout. This is going to make my next patch hard, as replacing stdout will make tracking time spent blocked on it more challenging. Move the line buffering requirement to util.py, and remove it from pager. This means that the abuse of ui.formatted=True and pager set to cat or equivalent no longer results in a line-buffered output to a pipe, hence (BC), although I don't expect anyone to be affected
Thu, 02 Feb 2017 02:56:38 -0800 localrepo: avoid unnecessary conversion from node to rev
Stanislau Hlebik <stash@fb.com> [Thu, 02 Feb 2017 02:56:38 -0800] rev 30875
localrepo: avoid unnecessary conversion from node to rev changelog.heads() first calls headrevs then converts them to nodes. localrepo.heads() then sorts them using self.changelog.rev function and makes useless conversion back to revs. Instead let's call changelog.headrevs() from localrepo.heads(), sort the output and then convert to nodes. Because headrevs does not support start parameter this optimization only works if start is None.
Sat, 04 Feb 2017 20:29:34 +0800 debian: update copyright years stable
Anton Shestakov <av6@dwimlabs.net> [Sat, 04 Feb 2017 20:29:34 +0800] rev 30874
debian: update copyright years
Sat, 04 Feb 2017 20:29:13 +0800 debian: update mailing list address stable
Anton Shestakov <av6@dwimlabs.net> [Sat, 04 Feb 2017 20:29:13 +0800] rev 30873
debian: update mailing list address
Thu, 02 Feb 2017 14:19:48 +0100 bundle2: implement a basic __repr__ for bundle2 part
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 14:19:48 +0100] rev 30872
bundle2: implement a basic __repr__ for bundle2 part We display basic data as the part id and part type. This make debugging bundle2 related code friendlier.
Thu, 02 Feb 2017 11:03:41 +0100 bundle2: drop an outdated comment
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 11:03:41 +0100] rev 30871
bundle2: drop an outdated comment The function is no longer in "early" stage and have been used in production for years. We can probably drop that part of the docstring...
Thu, 02 Feb 2017 10:53:55 +0100 unbundle: swap conditional branches for clarity
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 10:53:55 +0100] rev 30870
unbundle: swap conditional branches for clarity This is a small style update for clarity. The previous situation was: if foo: 50 lines else: 2 lines In such case I tend to invert these to get the simpler branch out of the way earlier: if not foo: 2 lines else: 50 lines This makes the conditional and various alternatives fit on the same screen, simpler to read overall.
Thu, 02 Feb 2017 10:55:38 +0100 unbundle: add a small comment to tag the bundle1 case as such
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 10:55:38 +0100] rev 30869
unbundle: add a small comment to tag the bundle1 case as such This makes the code clearer to understand for someone new to it (or rusted)
Thu, 02 Feb 2017 10:51:04 +0100 unbundle: add a small comment to clarify the 'check_heads' call
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 02 Feb 2017 10:51:04 +0100] rev 30868
unbundle: add a small comment to clarify the 'check_heads' call Bundle2 has its own mechanisms to check for heads (and other) changes, so push using bundle2 is relying on the "check:heads" bundle part of unbundle and the 'check_heads' call is not checking anything. We add a small comment to make this clearer.
Thu, 02 Feb 2017 11:17:36 -0800 pager: don't terminate with extreme prejudice on SIGPIPE (BC)
Simon Farnsworth <simonfar@fb.com> [Thu, 02 Feb 2017 11:17:36 -0800] rev 30867
pager: don't terminate with extreme prejudice on SIGPIPE (BC) The default SIGPIPE handler causes Mercurial to exit immediately, without running any Python cleanup code (except and finally blocks, atexit handlers etc). This creates problems if you want to do something at exit. If we need a different exit code for broken pipe from pager, then we should code that ourselves in Python; this appears to have been cargo-culted from the fork implementation of pager that's no longer used, where it was needed to stop Broken Pipe errors appearing on the user's terminal.
Mon, 23 Jan 2017 10:48:55 -0800 verify: replace _validpath() by matcher
Martin von Zweigbergk <martinvonz@google.com> [Mon, 23 Jan 2017 10:48:55 -0800] rev 30866
verify: replace _validpath() by matcher The verifier calls out to _validpath() to check if it should verify that path and the narrowhg extension overrides _validpath() to tell the verifier to skip that path. In treemanifest repos, the verifier calls the same method to check if it should visit a directory. However, the decision to visit a directory is different from the condition that it's a matching path, and narrowhg was working around it by returning True from its _validpath() override if *either* was true. Similar to how one can do "hg files -I foo/bar/ -X foo/" (making the include pointless), narrowhg can be configured to track the same paths. In that case match("foo/bar/baz") would be false, but match.visitdir("foo/bar/baz") turns out to be true, causing verify to fail. This may seem like a bug in visitdir(), but it's explicitly documented to be undefined for subdirectories of excluded directories. When using treemanifests, the walk would not descend into foo/, so verification would pass. However, when using flat manifests, there is no recursive directory walk and the file path "foo/bar/baz" would be passed to _validpath() without "foo/" (actually without the slash) being passed first. As explained above, _validpath() would return true for the file path and "hg verify" would fail. Replacing the _validpath() method by a matcher seems like the obvious fix. Narrowhg can then pass in its own matcher and not have to conflate the two matching functions (for dirs and files). I think it also makes the code clearer.
Wed, 01 Feb 2017 08:47:27 -0800 rebase: fix code comment to refer to right issue (4504, not 4505)
Martin von Zweigbergk <martinvonz@google.com> [Wed, 01 Feb 2017 08:47:27 -0800] rev 30865
rebase: fix code comment to refer to right issue (4504, not 4505) The comment was introduced in 8a544fb645bb (rebase: ensure rebase revision remains visible (issue4504), 2015-01-27), which mentions the right issue in the description.
Wed, 01 Feb 2017 11:30:26 -0600 merge with stable
Kevin Bullock <kbullock+mercurial@ringworld.org> [Wed, 01 Feb 2017 11:30:26 -0600] rev 30864
merge with stable
Wed, 01 Feb 2017 10:19:49 -0600 Added signature for changeset e1526da1e6d8 stable
Kevin Bullock <kbullock@ringworld.org> [Wed, 01 Feb 2017 10:19:49 -0600] rev 30863
Added signature for changeset e1526da1e6d8
Wed, 01 Feb 2017 10:18:59 -0600 Added tag 4.1 for changeset e1526da1e6d8 stable
Kevin Bullock <kbullock@ringworld.org> [Wed, 01 Feb 2017 10:18:59 -0600] rev 30862
Added tag 4.1 for changeset e1526da1e6d8
Wed, 01 Feb 2017 10:15:10 -0600 merge with i18n stable 4.1
Kevin Bullock <kbullock+mercurial@ringworld.org> [Wed, 01 Feb 2017 10:15:10 -0600] rev 30861
merge with i18n
Wed, 01 Feb 2017 08:47:11 -0200 i18n-pt_BR: synchronized with dfc6663f97ca stable
Wagner Bruna <wbruna@softwareexpress.com.br> [Wed, 01 Feb 2017 08:47:11 -0200] rev 30860
i18n-pt_BR: synchronized with dfc6663f97ca
Wed, 01 Feb 2017 02:10:30 +0100 merge: more safe detection of criss cross merge conflict between dm and r stable
Mads Kiilerich <mads@kiilerich.com> [Wed, 01 Feb 2017 02:10:30 +0100] rev 30859
merge: more safe detection of criss cross merge conflict between dm and r 41f6af50c0d8 introduced handling of a crash in this case. A review comment suggested that it was not entirely obvious that a 'dm' always would have a 'r' for the source file. To mitigate that risk, make the code more conservative and make less assumptions.
Mon, 30 Jan 2017 18:03:17 -0500 tests: correct (I think) command in test-largefiles-update stable
Augie Fackler <augie@google.com> [Mon, 30 Jan 2017 18:03:17 -0500] rev 30858
tests: correct (I think) command in test-largefiles-update When this test was introduced, it used the short-form of all the flags on this update invocation. I suspect, based on the "start with clean dirstates" comment and the fact that the no-exec branch of the #if guard leaves dirstate clean, that this should have been 'update -qCr' instead of 'update -qcr', but that a bug in largefiles --check handling left this problem unnoticed. I'll leave a breadcrumb further up about the current failure mode in the hopes that we can fix this some day. This was previously discussed in [0] but the trail in that thread goes cold after a few replies. Given that this is still a flaky test, that appears to only be passing by bad fortune, I think it's worth correcting the code of the test to make a correct assertion, and to keep track of the suspected bug with some other mechanism than an invalid test (if we had support for "expected failure" blocks this might be a worthwhile use of them?). 0: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089501.html
Mon, 30 Jan 2017 17:57:21 -0500 tests: expand flags to long form in test-largefiles-update.t stable
Augie Fackler <augie@google.com> [Mon, 30 Jan 2017 17:57:21 -0500] rev 30857
tests: expand flags to long form in test-largefiles-update.t I spent some time confused by this test. I'm pretty sure that this line intends to be cleaning the dirstate, not checking that it's clean before updating: the preceding #if block leaves the dirstate clean in the noexec case, and dirty in the exec case, so we can't expect consistent behavior across that platform variation. A subsequent patch will modify this command to use --clean instead of --check. I'll elaborate in that patch about the hypothetical bug here.
Tue, 31 Jan 2017 03:25:59 +0100 merge: fix crash on criss cross merge with dir move and delete (issue5020) stable
Mads Kiilerich <mads@kiilerich.com> [Tue, 31 Jan 2017 03:25:59 +0100] rev 30856
merge: fix crash on criss cross merge with dir move and delete (issue5020) Work around that 'dm' in the data model only can have one operation for the target file, but still can have multiple and conflicting operations on the source file where the other operation is a 'rm'. The move would thus fail with 'abort: No such file or directory'. In this case it is "obvious" that the file should be removed, either before or after moving it. We thus keep the 'rm' of the source file but drop the 'dm'. This is not a pretty fix but quite "obviously" safe (famous last words...) as it only touches a rare code path that used to crash. It is possible that it would be better to swap the files for 'dm' as suggested on https://bz.mercurial-scm.org/show_bug.cgi?id=5020#c13 but it is not entirely obvious that it not just would create conflicts on the other file. That can be revisited later.
Tue, 31 Jan 2017 03:20:07 +0100 tests: use 'f' in test-merge-criss-cross.t to prepare for recursive dumping stable
Mads Kiilerich <mads@kiilerich.com> [Tue, 31 Jan 2017 03:20:07 +0100] rev 30855
tests: use 'f' in test-merge-criss-cross.t to prepare for recursive dumping Prepare for adding a test case with files in a directory.
Mon, 30 Jan 2017 22:58:56 -0800 util: make sortdict.keys() return a copy stable
Martin von Zweigbergk <martinvonz@google.com> [Mon, 30 Jan 2017 22:58:56 -0800] rev 30854
util: make sortdict.keys() return a copy dict.keys() is documented to return a copy, so it's surprising that sortdict.keys() did not. I noticed this because we have an extension that calls readlocaltags(). That method tries to remove any tags that point to non-existent revisions (most likely stripped). However, since it's unintentionally working on the instance it's modifying, it sometimes fails to remove tags when there are multiple bad tags in a row. This was not caught because localrepo.tags() does an additional layer of filtering. sortdict is also used in other places, but I have not checked whether its keys() and/or __delitem__() methods are used there.
Mon, 30 Jan 2017 22:50:20 +0900 test-highlight: add normalization rule for Pygments 2.2 stable
Yuya Nishihara <yuya@tcha.org> [Mon, 30 Jan 2017 22:50:20 +0900] rev 30853
test-highlight: add normalization rule for Pygments 2.2 The test failed on Debian sid because of new class="vm".
Sun, 29 Jan 2017 12:40:56 -0800 tests: account for different newline behavior between Solaris and GNU grep stable
Danek Duvall <danek.duvall@oracle.com> [Sun, 29 Jan 2017 12:40:56 -0800] rev 30852
tests: account for different newline behavior between Solaris and GNU grep GNU grep, when emitting a matching line that doesn't have a terminating newline, will add an extra newline. Solaris grep passes the original line through without the newline. This causes differences in test output when looking at the last line of the output of get-with-headers.py, which doesn't usually emit (and certainly doesn't guarantee) a terminating newline. Both grep implementations succeed in matching the requested pattern, though, so rely on specifying the full pattern on grep's commandline instead of expecting it in the output, and send the output to /dev/null.
Fri, 20 Jan 2017 10:17:34 -0500 tests: also allow "Protocol not supported" in test-http-proxy error stable
Augie Fackler <augie@google.com> [Fri, 20 Jan 2017 10:17:34 -0500] rev 30851
tests: also allow "Protocol not supported" in test-http-proxy error I've seen this in a (misconfigured) FreeBSD jail which has ::1 as an entry for localhost, but IPv6 support is disabled in the jail. It took me months to figure out what was going on (and I only figured it out when tinyproxy.py got confused by similar IPv4-level misconfiguration of the localhost domain in /etc/hosts.) I don't feel strongly about this patch: on the one hand, it's papering over a host-level misconfiguration, but on the other it avoids some weird and hard to diagnose problems that can occur in weirdly restricted environments.
Fri, 20 Jan 2017 21:33:18 +0900 revset: prevent using outgoing() and remote() in hgweb session (BC) stable
Yuya Nishihara <yuya@tcha.org> [Fri, 20 Jan 2017 21:33:18 +0900] rev 30850
revset: prevent using outgoing() and remote() in hgweb session (BC) outgoing() and remote() may stall for long due to network I/O, which seems unsafe per definition, "whether a predicate is safe for DoS attack." But I'm not 100% sure about this. If our concern isn't elapsed time but CPU resource, these predicates are considered safe. Perhaps that would be up to the web/application server configuration? Anyway, outgoing() and remote() wouldn't be useful in hgweb, so I think it's okay to ban them.
Thu, 19 Jan 2017 16:23:49 -0500 tests: use an absolute path to get around '..' being invalid on a dead CWD stable
Augie Fackler <augie@google.com> [Thu, 19 Jan 2017 16:23:49 -0500] rev 30849
tests: use an absolute path to get around '..' being invalid on a dead CWD Only FreeBSD seems to be this picky. Note that this explicit absolute-path `cd` exposes a defect in the test, in that we end up still inside the cwd-vanish repository, but that's not a regression in this change. Since we're in a code freeze, I'm doing the smallest thing possible to try and fix bugs on FreeBSD, rather than cleaning up the entire problem. I'll follow up with a more complete fix after the freeze.
Wed, 18 Jan 2017 18:25:51 -0800 ui: rename tmpdir parameter to more specific repopath stable
Sean Farley <sean@farley.io> [Wed, 18 Jan 2017 18:25:51 -0800] rev 30848
ui: rename tmpdir parameter to more specific repopath This was requested by Augie and I agree that repopath is more descriptive.
Thu, 19 Jan 2017 23:01:32 +0900 pager: wrap _runcommand() no matter if stdout is redirected stable
Yuya Nishihara <yuya@tcha.org> [Thu, 19 Jan 2017 23:01:32 +0900] rev 30847
pager: wrap _runcommand() no matter if stdout is redirected We've made chg utilize the common code path implemented in pager.py (by 815e1cefd082 and 493935e0327a), but the chg server does not always start with a tty. Because of this, uisetup() of the pager extension could be skipped on the chg server. Kudos given to Sean Farley for dogfooding new chg and spotting this problem.
Thu, 19 Jan 2017 09:48:40 -0800 shelve: make unshelve not crash when there are missing files (issue4176) stable
Kostia Balytskyi <ikostia@fb.com> [Thu, 19 Jan 2017 09:48:40 -0800] rev 30846
shelve: make unshelve not crash when there are missing files (issue4176) This patch makes it possible to unshelve while having missing files in your repo as long as shelved changes don't touch those missing files. It also makes error message better otherwise.
Wed, 18 Jan 2017 22:45:07 -0800 statprof: require input file stable
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 18 Jan 2017 22:45:07 -0800] rev 30845
statprof: require input file statprof has a __main__ handler that allows viewing of previously written data files. As Yuya pointed out during review, f42cd5434cc2 broke this. This patch fixes that.
Wed, 18 Jan 2017 23:43:41 -0500 tests: work around FreeBSD's unzip having slightly different output stable
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 23:43:41 -0500] rev 30844
tests: work around FreeBSD's unzip having slightly different output According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's what comes as /usr/bin/unzip, so we may as well cater to it since it's easy.
Wed, 18 Jan 2017 23:34:35 -0500 contrib: fix check-commit to not reject commits from `hg sign` and `hg tag` stable
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 23:34:35 -0500] rev 30843
contrib: fix check-commit to not reject commits from `hg sign` and `hg tag` I'm tired of having a spurious red build every time we do a release. Fix it once and for all.
Wed, 18 Jan 2017 20:03:00 -0500 Added signature for changeset a1dd2c0c479e stable
Augie Fackler <raf@durin42.com> [Wed, 18 Jan 2017 20:03:00 -0500] rev 30842
Added signature for changeset a1dd2c0c479e
Wed, 18 Jan 2017 20:02:58 -0500 Added tag 4.1-rc for changeset a1dd2c0c479e stable
Augie Fackler <raf@durin42.com> [Wed, 18 Jan 2017 20:02:58 -0500] rev 30841
Added tag 4.1-rc for changeset a1dd2c0c479e
Wed, 18 Jan 2017 11:54:51 -0500 tests: fix up some http tests for no-zstd case stable 4.1-rc
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 11:54:51 -0500] rev 30840
tests: fix up some http tests for no-zstd case
Wed, 18 Jan 2017 11:43:36 -0500 freeze: merge default into stable for 4.1 code freeze stable
Augie Fackler <augie@google.com> [Wed, 18 Jan 2017 11:43:36 -0500] rev 30839
freeze: merge default into stable for 4.1 code freeze
Mon, 16 Jan 2017 21:17:39 -0800 patchbomb: add tmpdir parameter to ui.edit call
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:17:39 -0800] rev 30838
patchbomb: add tmpdir parameter to ui.edit call
Mon, 16 Jan 2017 21:15:57 -0800 histedit: add tmpdir parameter to ui.edit call
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:15:57 -0800] rev 30837
histedit: add tmpdir parameter to ui.edit call
Mon, 16 Jan 2017 21:15:21 -0800 cmdutil: add tmpdir parament to ui.edit calls
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:15:21 -0800] rev 30836
cmdutil: add tmpdir parament to ui.edit calls
Mon, 16 Jan 2017 21:05:22 -0800 ui: add a parameter to set the temporary directory for edit
Sean Farley <sean@farley.io> [Mon, 16 Jan 2017 21:05:22 -0800] rev 30835
ui: add a parameter to set the temporary directory for edit Until callsites are updated, this will have no effect. Once callsites are updated, specifying experimental.editortmpinhg will create editor temporary files in a subdirectory of .hg, which will make it easier for tool integrations to determine what repository is in play when they're asked to edit an hg-related file.
Wed, 18 Jan 2017 03:44:19 +0530 help: update help for `hg update` which was misleading (issue5427)
Pulkit Goyal <7895pulkit@gmail.com> [Wed, 18 Jan 2017 03:44:19 +0530] rev 30834
help: update help for `hg update` which was misleading (issue5427)
Tue, 17 Jan 2017 23:12:54 -0500 templater: add '{envvars}' to access environment variables
Matt Harbison <matt_harbison@yahoo.com> [Tue, 17 Jan 2017 23:12:54 -0500] rev 30833
templater: add '{envvars}' to access environment variables Since the option for ui.exportableenviron is experimental, so is this template until the underlying API is sorted out.
Tue, 17 Jan 2017 23:05:12 -0500 ui: introduce an experimental dict of exportable environment variables
Matt Harbison <matt_harbison@yahoo.com> [Tue, 17 Jan 2017 23:05:12 -0500] rev 30832
ui: introduce an experimental dict of exportable environment variables Care needs to be taken to prevent leaking potentially sensitive environment variables through hgweb, if template support for environment variables is to be introduced. There are a few ideas about the API for preventing accidental leaking [1]. Option 3 seems best from the POV of not needing to configure anything in the normal case. I couldn't figure out how to do that, so guard it with an experimental option for now. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-January/092383.html
Tue, 17 Jan 2017 13:44:53 +0800 tests: test experimental.spacemovesdown config for commit -i
Anton Shestakov <av6@dwimlabs.net> [Tue, 17 Jan 2017 13:44:53 +0800] rev 30831
tests: test experimental.spacemovesdown config for commit -i The feature is still very experimental, but at least its behavior is captured in the test.
Tue, 17 Jan 2017 10:17:13 -0800 zstd: prevent potential free() of uninitialized memory
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 17 Jan 2017 10:17:13 -0800] rev 30830
zstd: prevent potential free() of uninitialized memory This is a cherry pick of an upstream fix. The free() of uninitialed memory could likely only occur if a malloc() inside zstd fails. The patched functions aren't currently used by Mercurial. But I don't like leaving footguns sitting around.
Tue, 17 Jan 2017 11:25:02 -0800 revlog: give EXTSTORED flag value to narrowhg
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:25:02 -0800] rev 30829
revlog: give EXTSTORED flag value to narrowhg Narrowhg has been using "1 << 14" as its revlog flag value for a long time. We (Google) have many repos with that value in production already. When the same value was reserved for EXTSTORED, it made those repos invalid. Upgrading them will be a little painful. We should clearly have reserved the value for narrowhg a long time ago. Since the EXTSTORED flag is not yet in any release and Facebook also says they have not started using it in production, so it should be okay to change it. This patch gives the current value (1 << 14) back to narrowhg and gives a new value (1 << 13) to EXTSTORED.
Tue, 17 Jan 2017 11:45:10 -0800 help: don't let tools reflow revlog flags list
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:45:10 -0800] rev 30828
help: don't let tools reflow revlog flags list Before this change, the text about revlog flags was reflowed into a single paragraph, which made it a bit hard to read. I don't even know the rules around this, but adding a blank line before each flag seems to prevent the reflowing.
Tue, 17 Jan 2017 11:29:06 -0800 help: format revlog.txt more closely to result
Martin von Zweigbergk <martinvonz@google.com> [Tue, 17 Jan 2017 11:29:06 -0800] rev 30827
help: format revlog.txt more closely to result The rendered text has spaces before each item in the list
Tue, 17 Jan 2017 09:19:24 +0100 hgweb: simplify calculation of first revision in filelog command
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 17 Jan 2017 09:19:24 +0100] rev 30826
hgweb: simplify calculation of first revision in filelog command
Tue, 17 Jan 2017 09:17:29 +0100 hgweb: restore ascending iteration on revs in filelog web command
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 17 Jan 2017 09:17:29 +0100] rev 30825
hgweb: restore ascending iteration on revs in filelog web command Follow-up on 96f811bceb85. Adjust back the "parity" generator's offset to keep rendering the same.
Mon, 16 Jan 2017 09:22:32 +0100 context: extract _changesinrange() out of blockancestors()
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 16 Jan 2017 09:22:32 +0100] rev 30824
context: extract _changesinrange() out of blockancestors() We'll need it to write a blockdescendants function in next changeset.
Sat, 14 Jan 2017 01:23:07 +0530 shelve: allow multiple shelves with --patch and --stat
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Jan 2017 01:23:07 +0530] rev 30823
shelve: allow multiple shelves with --patch and --stat Before this patch, there was a single way to see multiple shelves using `--patch --list` which show all the shelves. Doing `--patch s1 s2` returns an error. This patch allows to show multiple shelves using `--patch` and `--stat`.
Sat, 14 Jan 2017 19:41:43 -0800 zstd: vendor python-zstandard 0.6.0
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 14 Jan 2017 19:41:43 -0800] rev 30822
zstd: vendor python-zstandard 0.6.0 Commit 63c68d6f5fc8de4afd9bde81b13b537beb4e47e8 from https://github.com/indygreg/python-zstandard is imported without modifications (other than removing unwanted files). This includes minor performance and feature improvements. It also changes the vendored zstd library from 1.1.1 to 1.1.2. # no-check-commit
Sat, 14 Jan 2017 20:05:15 +0530 util: add length argument to util.buffer()
Pulkit Goyal <7895pulkit@gmail.com> [Sat, 14 Jan 2017 20:05:15 +0530] rev 30821
util: add length argument to util.buffer() util.buffer() either returns inbuilt buffer function or defines a new one which slices. The inbuilt buffer() also has a length argument which is missing from the ones we defined. This patch adds that length argument.
Sun, 15 Jan 2017 13:17:05 +0530 py3: replace pycompat.getenv with encoding.environ.get
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 15 Jan 2017 13:17:05 +0530] rev 30820
py3: replace pycompat.getenv with encoding.environ.get pycompat.getenv returns os.getenvb on py3 which is not available on Windows. This patch replaces them with encoding.environ.get and checks to ensure no new instances of os.getenv or os.setenv are introduced.
Sun, 15 Jan 2017 16:33:15 +0900 patch: check length of git index header only if integer is specified
Yuya Nishihara <yuya@tcha.org> [Sun, 15 Jan 2017 16:33:15 +0900] rev 30819
patch: check length of git index header only if integer is specified Otherwise TypeError would be raised. Follows up d1901c4c8ec0.
Fri, 13 Jan 2017 20:16:56 -0800 localrepo: experimental support for non-zlib revlog compression
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 20:16:56 -0800] rev 30818
localrepo: experimental support for non-zlib revlog compression The final part of integrating the compression manager APIs into revlog storage is the plumbing for repositories to advertise they are using non-zlib storage and for revlogs to instantiate a non-zlib compression engine. The main intent of the compression manager work was to zstd all of the things. Adding zstd to revlogs has proved to be more involved than other places because revlogs are... special. Very small inputs and the use of delta chains (which are themselves a form of compression) are a completely different use case from streaming compression, which bundles and the wire protocol employ. I've conducted numerous experiments with zstd in revlogs and have yet to formalize compression settings and a storage architecture that I'm confident I won't regret later. In other words, I'm not yet ready to commit to a new mechanism for using zstd - or any other compression format - in revlogs. That being said, having some support for zstd (and other compression formats) in revlogs in core is beneficial. It can allow others to conduct experiments. This patch introduces *highly experimental* support for non-zlib compression formats in revlogs. Introduced is a config option to control which compression engine to use. Also introduced is a namespace of "exp-compression-*" requirements to denote support for non-zlib compression in revlogs. I've prefixed the namespace with "exp-" (short for "experimental") because I'm not confident of the requirements "schema" and in no way want to give the illusion of supporting these requirements in the future. I fully intend to drop support for these requirements once we figure out what we're doing with zstd in revlogs. A good portion of the patch is teaching the requirements system about registered compression engines and passing the requested compression engine as an opener option so revlogs can instantiate the proper compression engine for new operations. That's a verbose way of saying "we can now use zstd in revlogs!" On an `hg pull` conversion of the mozilla-unified repo with no extra redelta settings (like aggressivemergedeltas), we can see the impact of zstd vs zlib in revlogs: $ hg perfrevlogchunks -c ! chunk ! wall 2.032052 comb 2.040000 user 1.990000 sys 0.050000 (best of 5) ! wall 1.866360 comb 1.860000 user 1.820000 sys 0.040000 (best of 6) ! chunk batch ! wall 1.877261 comb 1.870000 user 1.860000 sys 0.010000 (best of 6) ! wall 1.705410 comb 1.710000 user 1.690000 sys 0.020000 (best of 6) $ hg perfrevlogchunks -m ! chunk ! wall 2.721427 comb 2.720000 user 2.640000 sys 0.080000 (best of 4) ! wall 2.035076 comb 2.030000 user 1.950000 sys 0.080000 (best of 5) ! chunk batch ! wall 2.614561 comb 2.620000 user 2.580000 sys 0.040000 (best of 4) ! wall 1.910252 comb 1.910000 user 1.880000 sys 0.030000 (best of 6) $ hg perfrevlog -c -d 1 ! wall 4.812885 comb 4.820000 user 4.800000 sys 0.020000 (best of 3) ! wall 4.699621 comb 4.710000 user 4.700000 sys 0.010000 (best of 3) $ hg perfrevlog -m -d 1000 ! wall 34.252800 comb 34.250000 user 33.730000 sys 0.520000 (best of 3) ! wall 24.094999 comb 24.090000 user 23.320000 sys 0.770000 (best of 3) Only modest wins for the changelog. But manifest reading is significantly faster. What's going on? One reason might be data volume. zstd decompresses faster. So given more bytes, it will put more distance between it and zlib. Another reason is size. In the current design, zstd revlogs are *larger*: debugcreatestreamclonebundle (size in bytes) zlib: 1,638,852,492 zstd: 1,680,601,332 I haven't investigated this fully, but I reckon a significant cause of larger revlogs is that the zstd frame/header has more bytes than zlib's. For very small inputs or data that doesn't compress well, we'll tend to store more uncompressed chunks than with zlib (because the compressed size isn't smaller than original). This will make revlog reading faster because it is doing less decompression. Moving on to bundle performance: $ hg bundle -a -t none-v2 (total CPU time) zlib: 102.79s zstd: 97.75s So, marginal CPU decrease for reading all chunks in all revlogs (this is somewhat disappointing). $ hg bundle -a -t <engine>-v2 (total CPU time) zlib: 191.59s zstd: 115.36s This last test effectively measures the difference between zlib->zlib and zstd->zstd for revlogs to bundle. This is a rough approximation of what a server does during `hg clone`. There are some promising results for zstd. But not enough for me to feel comfortable advertising it to users. We'll get there...
Fri, 13 Jan 2017 19:58:00 -0800 revlog: use compression engine APIs for decompression
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 19:58:00 -0800] rev 30817
revlog: use compression engine APIs for decompression Now that compression engines declare their header in revlog chunks and can decompress revlog chunks, we refactor revlog.decompress() to use them. Making full use of the property that revlog compressor objects are reusable, revlog instances now maintain a dict mapping an engine's revlog header to a compressor object. This is not only a performance optimization for engines where compressor object reuse can result in better performance, but it also serves as a cache of header values so we don't need to perform redundant lookups against the compression engine manager. (Yes, I measured and the overhead of a function call versus a dict lookup was observed.) Replacing the previous inline lookup table with a dict lookup was measured to make chunk reading ~2.5% slower on changelogs and ~4.5% slower on manifests. So, the inline lookup table has been mostly preserved so we don't lose performance. This is unfortunate. But many decompression operations complete in microseconds, so Python attribute lookup, dict lookup, and function calls do matter. The impact of this change on mozilla-unified is as follows: $ hg perfrevlogchunks -c ! chunk ! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6) ! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6) ! chunk batch ! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6) ! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6) $ hg perfrevlogchunks -m ! chunk ! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4) ! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4) ! chunk batch ! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5) ! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4) Changelog chunk reading is slightly faster but manifest reading is slower. What gives? On this repo, 99.85% of changelog entries are zlib compressed (the 'x' header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch swapped the test order of 'x' and '\0' so now 'x' is tested first. This makes changelogs faster since they almost always hit the first branch. This makes a significant percentage of manifest '\0' chunks slower because that code path now performs an extra test. Yes, I too can't believe we're able to measure the impact of an if..elif with simple string compares. I reckon this code would benefit from being written in C...
Fri, 13 Jan 2017 10:22:25 +0100 hgweb: build the "entries" list directly in filelog command
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 13 Jan 2017 10:22:25 +0100] rev 30816
hgweb: build the "entries" list directly in filelog command There's no apparent reason to have this "entries" generator function that builds a list and then yields its elements in reverse order and which is only called to build the "entries" list. So just build the list directly, in reverse order. Adjust "parity" generator's offset to keep rendering the same.
Sat, 14 Jan 2017 10:11:19 -0800 convert: remove "replacecommitter" action
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 14 Jan 2017 10:11:19 -0800] rev 30815
convert: remove "replacecommitter" action As pointed out by Yuya, this action doesn't add much (any?) value.
Sat, 14 Jan 2017 20:31:35 +0900 ui: check EOF of getpass() response read from command-server channel
Yuya Nishihara <yuya@tcha.org> [Sat, 14 Jan 2017 20:31:35 +0900] rev 30814
ui: check EOF of getpass() response read from command-server channel readline() returns '' only when EOF is encountered, in which case, Python's getpass() raises EOFError. We should do the same to abort the session as "response expected." This bug was reported to https://bitbucket.org/tortoisehg/thg/issues/4659/
Fri, 13 Jan 2017 23:21:10 -0800 convert: config option to control Git committer actions
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 23:21:10 -0800] rev 30813
convert: config option to control Git committer actions When converting a Git repository to Mercurial at Mozilla, I encountered a scenario where I didn't want `hg convert` to automatically add the "committer: <committer>" line to commit messages. While I can hack around this by rewriting the Git commit before it is fed into `hg convert`, I figured it would be a useful knob to control. This patch introduces a config option that allows lots of control over the committer value. I initially implemented this as a single boolean flag to control whether to save the committer message. But then there was feedback that it would be useful to save the committer in extra data. While this patch doesn't implement support for saving in extra data, it does add a mechanism for extending which actions to take on the committer field. We should be able to easily add actions to save in extra data. Some of the implemented features weren't asked for. But I figured they could be useful. If nothing else they demonstrate the extensibility of this mechanism.
Fri, 13 Jan 2017 21:21:02 -0800 help: make "mergetool" an alias for "merge-tools"
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 13 Jan 2017 21:21:02 -0800] rev 30812
help: make "mergetool" an alias for "merge-tools" I've probably typed `hg help mergetool` dozens of times. I'm tired of it not working.
Thu, 12 Jan 2017 21:06:55 +0900 templatekw: force noprefix=False to insure diffstat consistency (issue4755)
Matthieu Laneuville <mlaneuville@protonmail.com> [Thu, 12 Jan 2017 21:06:55 +0900] rev 30811
templatekw: force noprefix=False to insure diffstat consistency (issue4755) The result of diffstatdata should not depend on having noprefix set or not, as was reported in issue 4755. Forcing noprefix to false on call makes sure the parser receives the diff in the correct format and returns the proper result. Another way to fix this would have been to change the regular expressions in path.diffstatdata(), but that would have introduced many unecessary special cases.
Fri, 13 Jan 2017 10:11:37 -0800 check-code: reject module-level @cachefunc
Martin von Zweigbergk <martinvonz@google.com> [Fri, 13 Jan 2017 10:11:37 -0800] rev 30810
check-code: reject module-level @cachefunc Module-level @cachefunc usage is risky because it can easily create a memory "leak". Let's reject it completely for now. If a valid usage comes up in the future, we can always improve the check or reconsider.
Fri, 13 Jan 2017 11:42:36 -0800 similar: remove caching from the module level
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Fri, 13 Jan 2017 11:42:36 -0800] rev 30809
similar: remove caching from the module level To prevent Bad Thingsâ„¢ from happening, let's rework the logic to not use util.cachefunc.
Mon, 09 Jan 2017 11:01:45 -0800 patch: add label for coloring the similarity extended header
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 11:01:45 -0800] rev 30808
patch: add label for coloring the similarity extended header Just like the summary says, this will colorize the: similarity index 88% line in the diff output.
Mon, 09 Jan 2017 11:24:18 -0800 patch: use opt.showsimilarity to calculate and show the similarity
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 11:24:18 -0800] rev 30807
patch: use opt.showsimilarity to calculate and show the similarity Tests have been added.
Mon, 09 Jan 2017 10:51:44 -0800 patch: add similarity config knob in experimental section
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 10:51:44 -0800] rev 30806
patch: add similarity config knob in experimental section This config knob will control whether or not to show the similarity calculation in the diff output: diff --git a/README.md b/foo.md similarity index 88% rename from README.md rename to foo.md --- a/README.md +++ b/foo.md
Sat, 07 Jan 2017 20:47:57 -0800 similar: move score function to module level
Sean Farley <sean@farley.io> [Sat, 07 Jan 2017 20:47:57 -0800] rev 30805
similar: move score function to module level Future patches will use this to report the similarity of a rename / copy in the patch output.
Mon, 09 Jan 2017 17:58:19 +0900 revset: abuse x:y syntax to specify line range of followlines()
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 17:58:19 +0900] rev 30804
revset: abuse x:y syntax to specify line range of followlines() This slightly complicates the parsing (see the previous patch), but the overall result seems not bad. I keep x:, :y and : for future extension.
Mon, 09 Jan 2017 16:55:56 +0900 revset: do not transform range* operators in parsed tree
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 16:55:56 +0900] rev 30803
revset: do not transform range* operators in parsed tree This allows us to handle x:y range as a general range object. A primary user of it is followlines().
Mon, 09 Jan 2017 17:45:11 +0900 revset: add default value to getinteger() helper
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 17:45:11 +0900] rev 30802
revset: add default value to getinteger() helper This seems handy.
Mon, 09 Jan 2017 17:39:44 +0900 revset: factor out getinteger() helper
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 17:39:44 +0900] rev 30801
revset: factor out getinteger() helper We have 4 revset functions that take integer arguments, and they handle their arguments in slightly different ways. This patch unifies them: - getstring() in place of getsymbol(), which is more consistent with the handling of integer revisions (both 1 and '1' are valid) - say "expects" instead of "requires" for type errors We don't need to catch TypeError since getstring() must return a string.
Mon, 09 Jan 2017 16:16:26 +0900 revset: rename rev argument of followlines() to startrev
Yuya Nishihara <yuya@tcha.org> [Mon, 09 Jan 2017 16:16:26 +0900] rev 30800
revset: rename rev argument of followlines() to startrev The rev argument has the same meaning as startrev of follow(), and I think startrev is more informative. followlines() is new function, we can make BC now.
Fri, 13 Jan 2017 23:48:21 +0900 help: use :hg: role and canonical name to point to revset string patterns
Yuya Nishihara <yuya@tcha.org> [Fri, 13 Jan 2017 23:48:21 +0900] rev 30799
help: use :hg: role and canonical name to point to revset string patterns Follows up 5dd67f0993ce. Now revisions.txt and revsets.txt has been merged, so use revisions.* as a pointer.
Mon, 02 Jan 2017 13:27:20 -0800 util: compression APIs to support revlog decompression
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 13:27:20 -0800] rev 30798
util: compression APIs to support revlog decompression Previously, compression engines had APIs for performing revlog compression but no mechanism to perform revlog decompression. This patch changes that. Revlog decompression is slightly more complicated than compression because in the compression case there is (currently) only a single engine that can be used at a time. However for decompression, a revlog could contain chunks from multiple compression engines. This means decompression needs to map to multiple engines and decompressors. This functionality is outside the scope of this patch. But it drives the decision for engines to declare a byte header sequence that identifies revlog data as belonging to an engine and an API for obtaining an engine from a revlog header.
Sun, 08 Jan 2017 10:08:29 +0800 crecord: add an experimental option for space key to move cursor down
Anton Shestakov <av6@dwimlabs.net> [Sun, 08 Jan 2017 10:08:29 +0800] rev 30797
crecord: add an experimental option for space key to move cursor down I really want to have an option of toggling a selection on a line and also moving cursor down as a single keystroke. It also kinda makes sense for space key to do this, because some other curses UIs in the wild do this (e.g. various file managers, htop). So I got an idea to make a config option that defaults to False for compatibility, but allows making crecord UI a lot more useful for people with big hunks. We add this an experimental option to experiment with this behavior.
Mon, 02 Jan 2017 12:02:08 -0800 perf: support multiple compression engines in perfrevlogchunks
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 12:02:08 -0800] rev 30796
perf: support multiple compression engines in perfrevlogchunks Now that the revlog has a reference to a compressor, it is possible to swap in other compression engines. So, teach `hg perfrevlogchunks` to do that. The default behavior of `hg perfrevlogchunks` is now to measure the compression performance of all compression engines implementing the revlog compressor API. This effectively adds the no-op "none" compressor and zstd (when available) into the default set. While we can't yet plug alternate compressors into revlogs, this command gives us a preview of the performance. On the mozilla-unified repository: $ hg perfrevlogchunks -c ! compress w/ none ! wall 0.115159 comb 0.110000 user 0.110000 sys 0.000000 (best of 86) ! compress w/ zlib ! wall 5.681406 comb 5.680000 user 5.680000 sys 0.000000 (best of 3) ! compress w/ zstd ! wall 2.624781 comb 2.620000 user 2.620000 sys 0.000000 (best of 4) $ hg perfrevlogchunks -m ! compress w/ none ! wall 0.124486 comb 0.120000 user 0.120000 sys 0.000000 (best of 79) ! compress w/ zlib ! wall 10.144701 comb 10.150000 user 10.150000 sys 0.000000 (best of 3) ! compress w/ zstd ! wall 4.383118 comb 4.390000 user 4.390000 sys 0.000000 (best of 3) Those numbers for zstd look promising. But they aren't the full story. For that, we'll need to look at decompression times and storage sizes. Stay tuned...
Mon, 02 Jan 2017 11:22:52 -0800 revlog: use compression engine API for compression
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 11:22:52 -0800] rev 30795
revlog: use compression engine API for compression This commit swaps in the just-added revlog compressor API into the revlog class. Instead of implementing zlib compression inline in compress(), we now store a cached-on-first-use revlog compressor on each revlog instance and invoke its "compress()" method. As part of this, revlog.compress() has been refactored a bit to use a cleaner code flow and modern formatting (e.g. avoiding parenthesis around returned tuples). On a mozilla-unified repo, here are the "compress" times for a few commands: $ hg perfrevlogchunks -c ! wall 5.772450 comb 5.780000 user 5.780000 sys 0.000000 (best of 3) ! wall 5.795158 comb 5.790000 user 5.790000 sys 0.000000 (best of 3) $ hg perfrevlogchunks -m ! wall 9.975789 comb 9.970000 user 9.970000 sys 0.000000 (best of 3) ! wall 10.019505 comb 10.010000 user 10.010000 sys 0.000000 (best of 3) Compression times did seem to slow down just a little. There are 360,210 changelog revisions and 359,342 manifest revisions. For the changelog, mean time to compress a revision increased from ~16.025us to ~16.088us. That's basically a function call or an attribute lookup. I suppose this is the price you pay for abstraction. It's so low that I'm not concerned.
Mon, 02 Jan 2017 12:39:03 -0800 util: compression APIs to support revlog compression
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 12:39:03 -0800] rev 30794
util: compression APIs to support revlog compression As part of "zstd all of the things," we need to teach revlogs to use non-zlib compression formats. Because we're routing all compression via the "compression manager" and "compression engine" APIs, we need to introduction functionality there for performing revlog operations. Ideally, revlog compression and decompression operations would be implemented in terms of simple "compress" and "decompress" primitives. However, there are a few considerations that make us want to have a specialized primitive for handling revlogs: 1) Performance. Revlogs tend to do compression and especially decompression operations in batches. Any overhead for e.g. instantiating a "context" for performing an operation can be noticed. For this reason, our "revlog compressor" primitive is reusable. For zstd, we reuse the same compression "context" for multiple operations. I've measured this to have a performance impact versus constructing new contexts for each operation. 2) Specialization. By having a primitive dedicated to revlog use, we can make revlog-specific choices and leave the door open for more functionality in the future. For example, the zstd revlog compressor may one day make use of dictionary compression. A future patch will introduce a decompress() on the compressor object. The code for the zlib compressor is basically copied from revlog.compress(). Although it doesn't handle the empty input case, the null first byte case, and the 'u' prefix case. These cases will continue to be handled in revlog.py once that code is ported to use this API.
Mon, 02 Jan 2017 13:00:16 -0800 revlog: move decompress() from module to revlog class (API)
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 13:00:16 -0800] rev 30793
revlog: move decompress() from module to revlog class (API) Upcoming patches will convert revlogs to use the compression engine APIs to perform all things compression. The yet-to-be-introduced APIs support a persistent "compressor" object so the same object can be reused for multiple compression operations, leading to better performance. In addition, compression engines like zstd may wish to tweak compression engine state based on the revlog (e.g. per-revlog compression dictionaries). A global and shared decompress() function will shortly no longer make much sense. So, we move decompress() to be a method of the revlog class. It joins compress() there. On the mozilla-unified repo, we can measure the impact of this change on reading performance: $ hg perfrevlogchunks -c ! chunk ! wall 1.932573 comb 1.930000 user 1.900000 sys 0.030000 (best of 6) ! wall 1.955183 comb 1.960000 user 1.930000 sys 0.030000 (best of 6) ! chunk batch ! wall 1.787879 comb 1.780000 user 1.770000 sys 0.010000 (best of 6 ! wall 1.774444 comb 1.770000 user 1.750000 sys 0.020000 (best of 6) "chunk" appeared to become slower but "chunk batch" got faster. Upon further examination by running both sets multiple times, the numbers appear to converge across all runs. This tells me that there is no perceived performance impact to this refactor.
Mon, 02 Jan 2017 11:50:17 -0800 revlog: make compressed size comparisons consistent
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 02 Jan 2017 11:50:17 -0800] rev 30792
revlog: make compressed size comparisons consistent revlog.compress() compares the compressed size to the input size and throws away the compressed data if it is larger than the input. This is the correct thing to do, as storing compressed data that is larger than the input takes up more storage space and makes reading slower. However, the comparison was implemented inconsistently. For the streaming compression mode, we threw away the result if it was greater than or equal to the input size. But for the one-shot compression, we threw away the compression only if it was greater than the input size! This patch changes the comparison for the simple case so it is consistent with the streaming case. As a few tests demonstrate, this adds 1 byte to some revlog entries. This is because of an added 'u' header on the chunk. It seems somewhat wrong to increase the revlog size here. However, IMO the cost of 1 byte in storage is insignificant compared to the performance gains of avoiding decompression. This patch should invite questions around the heuristic for throwing away compressed data. For example, I'd argue we should be more liberal about rejecting compressed data, additionally doing so where the number of bytes saved fails to reach a threshold. But we can have this discussion another time.
Sat, 07 Jan 2017 20:43:49 -0800 similar: rename local variable to not collide with previous
Sean Farley <sean@farley.io> [Sat, 07 Jan 2017 20:43:49 -0800] rev 30791
similar: rename local variable to not collide with previous Future patches will move the score function to the module level, so let's not shadow that.
Mon, 09 Jan 2017 10:59:45 -0800 patch: add label for coloring the index extended header
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 10:59:45 -0800] rev 30790
patch: add label for coloring the index extended header Just like the summary says, this will colorize the: index 3d3ba4b65e11..57274a0f46b2 100644 line in the diff output.
Sat, 31 Dec 2016 15:41:57 -0600 patch: add index line for diff output
Sean Farley <sean@farley.io> [Sat, 31 Dec 2016 15:41:57 -0600] rev 30789
patch: add index line for diff output This helps highlighting in third-party diff coloring (which assumes git output) and maintains pedantic correctness with diff --git. Tests will be added at the end of the series.
Mon, 09 Jan 2017 11:13:47 -0800 patch: add config knob for displaying the index header
Sean Farley <sean@farley.io> [Mon, 09 Jan 2017 11:13:47 -0800] rev 30788
patch: add config knob for displaying the index header This config knob can take an integer between 0 and 40 or a keyword ('none', 'short', 'full') to control the length of hash to output. It will display diffs with the git index header as such, diff --git a/mercurial/mdiff.py b/mercurial/mdiff.py index 112edf7..d6b52c5 100644 We'll put this in the experimental section for now.
Thu, 12 Jan 2017 12:05:23 -0800 bisect: refer directly to bisect() revset predicate in help
Martin von Zweigbergk <martinvonz@google.com> [Thu, 12 Jan 2017 12:05:23 -0800] rev 30787
bisect: refer directly to bisect() revset predicate in help We have specific syntax for displaying the help text for a particular revset predicate, so let's refer directly to the bisect() revset in the verbose bisect help. It seems likely that the user doesn't care about other revsets at that point, so they will probably not miss the text about the other revset predicates.
Thu, 12 Jan 2017 11:43:26 -0800 tests: use "hg help revisions.<predicate>" instead of grepping
Martin von Zweigbergk <martinvonz@google.com> [Thu, 12 Jan 2017 11:43:26 -0800] rev 30786
tests: use "hg help revisions.<predicate>" instead of grepping We have specific syntax for displaying the help text for a particular revset predicate, so use that instead of grepping through the full output.
Thu, 12 Jan 2017 11:52:05 -0800 help: remove now-redundant pointer to revsets help
Martin von Zweigbergk <martinvonz@google.com> [Thu, 12 Jan 2017 11:52:05 -0800] rev 30785
help: remove now-redundant pointer to revsets help "hg help revisions" and "hg help revsets" now point to the same text, so drop the revsets reference.
Sat, 07 Jan 2017 23:35:35 -0500 help: eliminate duplicate text for revset string patterns
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Jan 2017 23:35:35 -0500] rev 30784
help: eliminate duplicate text for revset string patterns There's no reason to duplicate this so many times, and it's likely an instance will be missed if support for a new pattern is added and documented. The stringmatcher is mostly used by revsets, though it is also used for the 'tag' related templates, and namespace filtering in the journal extension. So maybe there's a better place to document it. `hg help patterns` seems inappropriate, because that is all file pattern matching. While here, indicate how to perform case insensitive regex searches.
Sat, 07 Jan 2017 21:26:32 -0500 revset: add regular expression support to 'desc'
Matt Harbison <matt_harbison@yahoo.com> [Sat, 07 Jan 2017 21:26:32 -0500] rev 30783
revset: add regular expression support to 'desc' This is a case insensitive predicate like 'author', so it conforms to the existing behavior of performing a case insensitive regex.
Wed, 11 Jan 2017 22:42:10 -0500 revset: stop lowercasing the regex pattern for 'author'
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Jan 2017 22:42:10 -0500] rev 30782
revset: stop lowercasing the regex pattern for 'author' It was probably unintentional for regex, as the meaning of some sequences like \S and \s is actually inverted by changing the case. For backward compatibility however, the matching is forced to case insensitive.
Thu, 24 Nov 2016 18:45:29 -0800 repair: clean up stale lock file from store backup
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 24 Nov 2016 18:45:29 -0800] rev 30781
repair: clean up stale lock file from store backup Since we did a directory rename on the stores, the source repository's lock path now references the dest repository's lock path and the dest repository's lock path now references a non-existent filename. So releasing the lock on the source will unlock the dest and releasing the lock on the dest will no-op because it fails due to file not found. So we clean up the dest's lock manually.
Thu, 24 Nov 2016 18:34:50 -0800 repair: copy non-revlog store files during upgrade
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 24 Nov 2016 18:34:50 -0800] rev 30780
repair: copy non-revlog store files during upgrade The store contains more than just revlogs. This patch teaches the upgrade code to copy regular files as well. As the test changes demonstrate, the phaseroots file is now copied.
Sun, 18 Dec 2016 17:00:15 -0800 repair: migrate revlogs during upgrade
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 17:00:15 -0800] rev 30779
repair: migrate revlogs during upgrade Our next step for in-place upgrade is to migrate store data. Revlogs are the biggest source of data within the store and a store is useless without them, so we implement their migration first. Our strategy for migrating revlogs is to walk the store and call `revlog.clone()` on each revlog. There are some minor complications. Because revlogs have different storage options (e.g. changelog has generaldelta and delta chains disabled), we need to obtain the correct class of revlog so inserted data is encoded properly for its type. Various attempts at implementing progress indicators that didn't lead to frustration from false "it's almost done" indicators were made. I initially used a single progress bar based on number of revlogs. However, this quickly churned through all filelogs, got to 99% then effectively froze at 99.99% when it got to the manifest. So I converted the progress bar to total revision count. This was a little bit better. But the manifest was still significantly slower than filelogs and it took forever to process the last few percent. I then tried both revision/chunk bytes and raw bytes as the denominator. This had the opposite effect: because so much data is in manifests, it would churn through filelogs without showing much progress. When it got to manifests, it would fill in 90+% of the progress bar. I finally gave up having a unified progress bar and instead implemented 3 progress bars: 1 for filelog revisions, 1 for manifest revisions, and 1 for changelog revisions. I added extra messages indicating the total number of revisions of each so users know there are more progress bars coming. I also added extra messages before and after each stage to give extra details about what is happening. Strictly speaking, this isn't necessary. But the numbers are impressive. For example, when converting a non-generaldelta mozilla-central repository, the messages you see are: migrating 2475593 total revisions (1833043 in filelogs, 321156 in manifests, 321394 in changelog) migrating 1.67 GB in store; 2508 GB tracked data migrating 267868 filelogs containing 1833043 revisions (1.09 GB in store; 57.3 GB tracked data) finished migrating 1833043 filelog revisions across 267868 filelogs; change in size: -415776 bytes migrating 1 manifests containing 321156 revisions (518 MB in store; 2451 GB tracked data) That "2508 GB" figure really blew me away. I had no clue that the raw tracked data in mozilla-central was that large. Granted, 2451 GB is in the manifest and "only" 57.3 GB is in filelogs. But still. It's worth noting that gratuitous loading of source revlogs in order to display numbers and progress bars does serve a purpose: it ensures we can open all source revlogs. We don't want to spend several minutes copying revlogs only to encounter a permissions error or similar later. As part of this commit, we also add swapping of the store directory to the upgrade function. After revlogs are converted, we move the old store into the backup directory then move the temporary repo's store into the old store's location. On well-behaved systems, this should be 2 atomic operations and the window of inconsistency show be very narrow. There are still a few improvements to be made to store copying and upgrading. But this commit gets the bulk of the work out of the way.
Sun, 18 Dec 2016 17:02:57 -0800 revlog: add clone method
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 17:02:57 -0800] rev 30778
revlog: add clone method Upcoming patches will introduce functionality for in-place repository/store "upgrades." Copying the contents of a revlog feels sufficiently low-level to warrant being in the revlog class. So this commit implements that functionality. Because full delta recomputation can be *very* expensive (we're talking several hours on the Firefox repository), we support multiple modes of execution with regards to delta (re)use. This will allow repository upgrades to choose the "level" of processing/optimization they wish to perform when converting revlogs. It's not obvious from this commit, but "addrevisioncb" will be used for progress reporting.
Sun, 18 Dec 2016 16:59:04 -0800 repair: begin implementation of in-place upgrading
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 16:59:04 -0800] rev 30777
repair: begin implementation of in-place upgrading Now that all the upgrade planning work is in place, we can start doing the real work: actually upgrading a repository. The main goal of this commit is to get the "framework" for running in-place upgrade actions in place. Rather than get too clever and low-level with regards to in-place upgrades, our strategy is to create a new, temporary repository, copy data to it, then replace the old data with the new. This allows us to reuse a lot of code in localrepo.py around store interaction, which will eventually consume the bulk of the upgrade code. But we have to start small. This patch implements adding new repository requirements. But it still sets up a temporary repository and locks it and the source repo before performing the requirements file swap. This means all the plumbing is in place to implement store copying in subsequent commits.
Sun, 18 Dec 2016 16:51:09 -0800 repair: determine what upgrade will do
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 16:51:09 -0800] rev 30776
repair: determine what upgrade will do This commit introduces code for determining what actions/improvements an upgrade should perform. The "upgradefindimprovements" function introduces a mechanism to return a list of improvements that can be made to a repository. Each improvement is effectively an action that an upgrade will perform. Associated with each of these improvements is metadata that will be used to inform users what's wrong and what an upgrade will do. Each "improvement" is categorized as a "deficiency" or an "optimization." TBH, I'm not thrilled about the terminology and am receptive to constructive bikeshedding. The main difference between a "deficiency" and an "optimization" is a deficiency is always corrected (if it deviates from the current config) and an "optimization" is an optional action that goes above and beyond to improve the state of the repository (usually by requiring more CPU during upgrade). Our initial set of improvements identifies missing repository requirements, a single, easily correctable problem with changelog storage, and a set of "optimizations" related to delta recalculation. The main "upgraderepo" function has been expanded to handle improvements. It queries for the list of improvements and determines which of them will run based on the current repository state and user I went through numerous iterations of the output format before settling on a ReST-inspired definition list format. (I used bulleted lists in the first submission of this commit and could not get it to format just right.) Even with the various iterations, I'm still not super thrilled with the format. But, this is a debug* command, so that should mean we can refine the output without BC concerns.
Sun, 18 Dec 2016 16:16:54 -0800 repair: implement requirements checking for upgrades
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Dec 2016 16:16:54 -0800] rev 30775
repair: implement requirements checking for upgrades This commit introduces functionality for upgrading a repository in place. The first part that's implemented is testing for upgrade "compatibility." This is done by examining repository requirements. There are 5 functions returning sets of requirements that control upgrading. Why so many functions? Mainly to support extensions. Functions are easier to monkeypatch than module variables. Astute readers will see that we don't support "manifestv2" and "treemanifest" requirements in the upgrade mechanism. I don't have a great answer for why other than this is a complex set of patches and I don't want to deal with the complexity of these experimental features just yet. We can teach the upgrade mechanism about them later, once the basic upgrade mechanism is in place. This commit also introduces the "upgraderepo" function. This will be our main routine for performing an in-place upgrade. Currently, it just implements requirements checking. The structure of some code in this function may look a bit weird (e.g. the inline function that is only called once). But this will make sense after future commits.
Thu, 24 Nov 2016 16:24:09 -0800 debugcommands: stub for debugupgraderepo command
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 24 Nov 2016 16:24:09 -0800] rev 30774
debugcommands: stub for debugupgraderepo command Currently, if Mercurial introduces a new repository/store feature or changes behavior of an existing feature, users must perform an `hg clone` to create a new repository with hopefully the correct/optimal settings. Unfortunately, even `hg clone` may not give the correct results. For example, if you do a local `hg clone`, you may get hardlinks to revlog files that inherit the old state. If you `hg clone` from a remote or `hg clone --pull`, changegroup application may bypass some optimization, such as converting to generaldelta. Optimizing a repository is harder than it seems and requires more than a simple `hg` command invocation. This commit starts the process of changing that. We introduce `hg debugupgraderepo`, a command that performs an in-place upgrade of a repository to use new, optimal features. The command is just a stub right now. Features will be added in subsequent commits. This commit does foreshadow some of the behavior of the new command, notably that it doesn't do anything by default and that it takes arguments that influence what actions it performs. These will be explained more in subsequent commits.
Wed, 11 Jan 2017 21:47:19 -0500 util: teach stringmatcher to handle forced case insensitive matches
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Jan 2017 21:47:19 -0500] rev 30773
util: teach stringmatcher to handle forced case insensitive matches The 'author' and 'desc' revsets are documented to be case insensitive. Unfortunately, this was implemented in 'author' by forcing the input to lowercase, including for regex like '\B'. (This actually inverts the meaning of the sequence.) For backward compatibility, we will keep that a case insensitive regex, but by using matcher options instead of brute force. This doesn't preclude future hypothetical 'icase-literal:' style prefixes that can be provided by the user. Such user specified cases can probably be handled up front by stripping 'icase-', setting the variable, and letting it drop through the existing code.
Wed, 11 Jan 2017 23:13:51 -0500 revset: point to 'grep' in the 'keyword' help for regex searches
Matt Harbison <matt_harbison@yahoo.com> [Wed, 11 Jan 2017 23:13:51 -0500] rev 30772
revset: point to 'grep' in the 'keyword' help for regex searches The help for 'grep' already points to 'keyword'.
Wed, 11 Jan 2017 23:13:00 -0800 help: explain that revsets can be used where 1 or 2 revs are wanted
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 23:13:00 -0800] rev 30771
help: explain that revsets can be used where 1 or 2 revs are wanted We did not seem to document that one can do things like "hg up :@" where the last revision of the revset ":@".
Wed, 11 Jan 2017 22:46:07 -0800 help: explain what the term "revset" means
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 22:46:07 -0800] rev 30770
help: explain what the term "revset" means We refer to revsets in a few places (e.g. in "hg help config"), but we never explained what they are. Until now.
Wed, 11 Jan 2017 11:37:38 -0800 help: merge revsets.txt into revisions.txt
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:37:38 -0800] rev 30769
help: merge revsets.txt into revisions.txt Selecting single and multiple revisions is closely related, so let's put it in one place, so users can easily find it. We actually did not even point to "hg help revsets" from "hg help revisions", but now that they're on a single page, that won't be necessary.
Wed, 11 Jan 2017 11:40:40 -0800 tests: use `hg help dates` instead of `hg help revs` in test
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:40:40 -0800] rev 30768
tests: use `hg help dates` instead of `hg help revs` in test The revisions help is already long and will get longer, so switch to another short and stable topic.
Wed, 11 Jan 2017 11:28:54 -0800 help: use a single paragraph to describe full and abbreviated nodeids
Martin von Zweigbergk <martinvonz@google.com> [Wed, 11 Jan 2017 11:28:54 -0800] rev 30767
help: use a single paragraph to describe full and abbreviated nodeids The texts describing 40-digit strings and the abbreviated form are closely related, so make it a single paragraph.
Tue, 10 Jan 2017 23:37:08 -0800 hgweb: support Content Security Policy
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 23:37:08 -0800] rev 30766
hgweb: support Content Security Policy Content-Security-Policy (CSP) is a web security feature that allows servers to declare what loaded content is allowed to do. For example, a policy can prevent loading of images, JavaScript, CSS, etc unless the source of that content is whitelisted (by hostname, URI scheme, hashes of content, etc). It's a nifty security feature that provides extra mitigation against some attacks, notably XSS. Mitigation against these attacks is important for Mercurial because hgweb renders repository data, which is commonly untrusted. While we make attempts to escape things, etc, there's the possibility that malicious data could be injected into the site content. If this happens today, the full power of the web browser is available to that malicious content. A restrictive CSP policy (defined by the server operator and sent in an HTTP header which is outside the control of malicious content), could restrict browser capabilities and mitigate security problems posed by malicious data. CSP works by emitting an HTTP header declaring the policy that browsers should apply. Ideally, this header would be emitted by a layer above Mercurial (likely the HTTP server doing the WSGI "proxying"). This works for some CSP policies, but not all. For example, policies to allow inline JavaScript may require setting a "nonce" attribute on <script>. This attribute value must be unique and non-guessable. And, the value must be present in the HTTP header and the HTML body. This means that coordinating the value between Mercurial and another HTTP server could be difficult: it is much easier to generate and emit the nonce in a central location. This commit introduces support for emitting a Content-Security-Policy header from hgweb. A config option defines the header value. If present, the header is emitted. A special "%nonce%" syntax in the value triggers generation of a nonce and inclusion in <script> elements in templates. The inclusion of a nonce does not occur unless "%nonce%" is present. This makes this commit completely backwards compatible and the feature opt-in. The nonce is a type 4 UUID, which is the flavor that is randomly generated. It has 122 random bits, which should be plenty to satisfy the guarantees of a nonce.
Tue, 10 Jan 2017 20:47:48 -0800 hgweb: call process_dates() via DOM event listener
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 20:47:48 -0800] rev 30765
hgweb: call process_dates() via DOM event listener All the hgweb templates include mercurial.js in their header. All the hgweb templates have the same <script> boilerplate to run process_dates(). This patch factors that function call into mercurial.js as part of a DOMContentLoaded event listener.
Sat, 24 Dec 2016 15:29:32 -0700 protocol: send application/mercurial-0.2 responses to capable clients
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:29:32 -0700] rev 30764
protocol: send application/mercurial-0.2 responses to capable clients With this commit, the HTTP transport now parses the X-HgProto-<N> header to determine what media type and compression engine to use for responses. So far, we only compress responses that are already being compressed with zlib today (stream response types to specific commands). We can expand things to cover additional response types later. The practical side-effect of this commit is that non-zlib compression engines will be used if both ends support them. This means if both ends have zstd support, zstd - not zlib - will be used to compress data! When cloning the mozilla-unified repository between a local HTTP server and client, the benefits of non-zlib compression are quite noticeable: engine server CPU (s) client CPU (s) bundle size zlib (l=6) 174.1 283.2 1,148,547,026 zstd (l=1) 99.2 267.3 1,127,513,841 zstd (l=3) 103.1 266.9 1,018,861,363 zstd (l=7) 128.3 269.7 919,190,278 zstd (l=10) 162.0 - 894,547,179 none 95.3 277.2 4,097,566,064 The default zstd compression level is 3. So if you deploy zstd capable Mercurial to your clients and servers and CPU time on your server is dominated by "getbundle" requests (clients cloning and pulling) - and my experience at Mozilla tells me this is often the case - this commit could drastically reduce your server-side CPU usage *and* save on bandwidth costs! Another benefit of this change is that server operators can install *any* compression engine. While it isn't enabled by default, the "none" compression engine can now be used to disable wire protocol compression completely. Previously, commands like "getbundle" always zlib compressed output, adding considerable overhead to generating responses. If you are on a high speed network and your server is under high load, it might be advantageous to trade bandwidth for CPU. Although, zstd at level 1 doesn't use that much CPU, so I'm not convinced that disabling compression wholesale is worthwhile. And, my data seems to indicate a slow down on the client without compression. I suspect this is due to a lack of buffering resulting in an increase in socket read() calls and/or the fact we're transferring an extra 3 GB of data (parsing HTTP chunked transfer and processing extra TCP packets can add up). This is definitely worth investigating and optimizing. But since the "none" compressor isn't enabled by default, I'm inclined to punt on this issue. This commit introduces tons of tests. Some of these should arguably have been implemented on previous commits. But it was difficult to test without the server functionality in place.
Sat, 24 Dec 2016 15:22:18 -0700 httppeer: advertise and support application/mercurial-0.2
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:22:18 -0700] rev 30763
httppeer: advertise and support application/mercurial-0.2 Now that servers expose a capability indicating they support application/mercurial-0.2 and compression, clients can key off this to say they support responses that are compressed with various compression formats. After this commit, the HTTP wire protocol client now sends an "X-HgProto-<N>" request header indicating its support for "application/mercurial-0.2" media type and various compression formats. This commit also implements support for handling "application/mercurial-0.2" responses. It simply reads the header compression engine identifier then routes the remainder of the response to the appropriate decompressor. There were some test changes, but only to logging. That points to an obvious gap in our test coverage. This will be addressed in a subsequent commit once server support is in place (it is hard to test without server support).
Sat, 24 Dec 2016 15:21:46 -0700 wireproto: advertise supported media types and compression formats
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 15:21:46 -0700] rev 30762
wireproto: advertise supported media types and compression formats This commit introduces support for advertising a server's support for media types and compression formats in accordance with the spec defined in internals.wireproto. The bulk of the new code is a helper function in wireproto.py to obtain a prioritized list of compression engines available to the wire protocol. While not utilized yet, we implement support for obtaining the list of compression engines advertised by the client. The upcoming HTTP protocol enhancements are a bit lower-level than existing tests (most existing tests are command centric). So, this commit establishes a new test file that will be appropriate for holding tests around the functionality of the HTTP protocol itself. Rounding out this change, `hg debuginstall` now prints compression engines available to the server.
Sat, 24 Dec 2016 13:51:12 -0700 util: declare wire protocol support of compression engines
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 13:51:12 -0700] rev 30761
util: declare wire protocol support of compression engines This patch implements a new compression engine API allowing compression engines to declare support for the wire protocol. Support is declared by returning a compression format string identifier that will be added to payloads to signal the compression type of data that follows and default integer priorities of the engine. Accessor methods have been added to the compression engine manager class to facilitate use. Note that the "none" and "bz2" engines declare wire protocol support but aren't enabled by default due to their priorities being 0. It is essentially free from a coding perspective to support these compression formats, so we do it in case anyone may derive use from it.
Sat, 24 Dec 2016 13:56:36 -0700 internals: document compression negotiation
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 13:56:36 -0700] rev 30760
internals: document compression negotiation As part of adding zstd support to all of the things, we'll need to teach the wire protocol to support non-zlib compression formats. This commit documents how we'll implement that. To understand how we arrived at this proposal, let's look at how things are done today. The wire protocol today doesn't have a unified format. Instead, there is a limited facility for differentiating replies as successful or not. And, each command essentially defines its own response format. A significant deficiency in the current protocol is the lack of payload framing over the SSH transport. In the HTTP transport, chunked transfer is used and the end of an HTTP response body (and the end of a Mercurial command response) can be identified by a 0 length chunk. This is how HTTP chunked transfer works. But in the SSH transport, there is no such framing, at least for certain responses (notably the response to "getbundle" requests). Clients can't simply read until end of stream because the socket is persistent and reused for multiple requests. Clients need to know when they've encountered the end of a request but there is nothing simple for them to key off of to detect this. So what happens is the client must decode the payload (as opposed to being dumb and forwarding frames/packets). This means the payload itself needs to support identifying end of stream. In some cases (bundle2), it also means the payload can encode "error" or "interrupt" events telling the client to e.g. abort processing. The lack of framing on the SSH transport and the transfer of its responsibilities to e.g. bundle2 is a massive layering violation and a wart on the protocol architecture. It needs to be fixed someday by inventing a proper framing protocol. So about compression. The client transport abstractions have a "_callcompressable()" API. This API is called to invoke a remote command that will send a compressible response. The response is essentially a "streaming" response (no framing data at the Mercurial layer) that is fed into a decompressor. On the HTTP transport, the decompressor is zlib and only zlib. There is currently no mechanism for the client to specify an alternate compression format. And, clients don't advertise what compression formats they support or ask the server to send a specific compression format. Instead, it is assumed that non-error responses to "compressible" commands are zlib compressed. On the SSH transport, there is no compression at the Mercurial protocol layer. Instead, compression must be handled by SSH itself (e.g. `ssh -C`) or within the payload data (e.g. bundle compression). For the HTTP transport, adding new compression formats is pretty straightforward. Once you know what decompressor to use, you can stream data into the decompressor until you reach a 0 size HTTP chunk, at which point you are at end of stream. So our wire protocol changes for the HTTP transport are pretty straightforward: the client and server advertise what compression formats they support and an appropriate compression format is chosen. We introduce a new HTTP media type to hold compressed payloads. The header of the payload defines the compression format being used. Whoever is on the receiving end can sniff the first few bytes route to an appropriate decompressor. Support for multiple compression formats is advertised on both server and client. The server advertises a "compression" capability saying which compression formats it supports and in what order they are preferred. Clients advertise their support for multiple compression formats and media types via the introduced "X-HgProto" request header. Strictly speaking, servers don't need to advertise which compression formats they support. But doing so allows clients to fail fast if they don't support any of the formats the server does. This is useful in situations like sending bundles, where the client may have to perform expensive computation before sending data to the server. Rather than simply advertise a list of supported compression formats, we introduce an additional "httpmediatype" server capability advertising which media types the server supports. This means servers are explicit about what formats they exchange. IMO, this is superior to inferring support from other capabilities (like "compression"). By advertising compression support on each request in the "X-HgProto" header and media type and direction at the server level, we are able to gradually transition existing commands/responses to the new media type and possibly compression. Contrast with the old world, where we only supported a single media type and the use of compression was built-in to the semantics of the command on both client and server. In the new world, if "application/mercurial-0.2" is supported, compression is supported. It's that simple. It's worth noting that we explicitly don't use "Accept," "Accept-Encoding," "Content-Encoding," or "Transfer-Encoding" for content negotiation and compression. People knowledgeable of the HTTP specifications will say that we should use these because that's what they are designed to be used for. They have a point and I sympathize with the argument. Earlier versions of this commit even defined supported media types in the "Accept" header. However, my years of experience rolling out services leveraging HTTP has taught me to not trust the HTTP layer, especially if you are going outside the normal spec (such as using a custom "Content-Encoding" value to represent zstd streams). I've seen load balancers, proxies, and other network devices do very bad and unexpected things to HTTP messages (like insisting zlib compressed content is decoded and then re-encoded at a different compression level or even stripping compression completely). I've found that the best way to avoid surprises when writing protocols on top of HTTP is to use HTTP as a dumb transport as much as possible to minimize the chances that an "intelligent" agent between endpoints will muck with your data. While the widespread use of TLS is mitigating many intermediate network agents interfering with HTTP, there are still problems at the edges, with e.g. the origin HTTP server needing to convert HTTP to and from WSGI and buggy or feature-lacking HTTP client implementations. I've found the best way to avoid these problems is to avoid using headers like "Content-Encoding" and to bake as much logic as possible into media types and HTTP message bodies. The protocol changes in this commit do rely on a custom HTTP request header and the "Content-Type" headers. But we used them before, so we shouldn't be increasing our exposure to "bad" HTTP agents. For the SSH transport, we can't easily implement content negotiation to determine compression formats because the SSH transport has no content negotiation capabilities today. And without a framing protocol, we don't know how much data to feed into a decompressor. So in order to implement compression support on the SSH transport, we'd need to invent a mechanism to represent content types and an outer framing protocol to stream data robustly. While I'm fully capable of doing that, it is a lot of work and not something that should be undertaken lightly. My opinion is that if we're going to change the SSH transport protocol, we should take a long hard look at implementing a grand unified protocol that attempts to address all the deficiencies with the existing protocol. While I want this to happen, that would be massive scope bloat standing in the way of zstd support. So, I've decided to take the easy solution: the SSH transport will not gain support for multiple compression formats. Keep in mind it doesn't support *any* compression today. So essentially nothing is changing on the SSH front.
Sat, 24 Dec 2016 14:46:02 -0700 httppeer: extract code for HTTP header spanning
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Dec 2016 14:46:02 -0700] rev 30759
httppeer: extract code for HTTP header spanning A second consumer of HTTP header spanning will soon be introduced. Factor out the code to do this so it can be reused.
Tue, 10 Jan 2017 11:20:32 -0800 commands: config option to control bundle compression level
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 11:20:32 -0800] rev 30758
commands: config option to control bundle compression level Currently, bundle compression uses the default compression level for the active compression engine. The default compression level is tuned as a compromise between speed and size. Some scenarios may call for a different compression level. For example, with clone bundles, bundles are generated once and used several times. Since the cost to generate is paid infrequently, server operators may wish to trade extra CPU time for better compression ratios. This patch introduces an experimental and undocumented config option to control the bundle compression level. As the inline comment says, this approach is a bit hacky. I'd prefer for the compression level to be encoded in the bundle spec. e.g. "zstd-v2;complevel=15." However, given that the 4.1 freeze is imminent, I'm not comfortable implementing this user-facing change without much time to test and consider the implications. So, we're going with the quick and dirty solution for now. Having this option in the 4.1 release will enable Mozilla to easily produce and test zlib and zstd bundles with non-default compression levels in production. This will help drive future development of the feature and zstd integration with Mercurial.
Tue, 10 Jan 2017 11:19:37 -0800 bundle2: allow compression options to be passed to compressor
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 10 Jan 2017 11:19:37 -0800] rev 30757
bundle2: allow compression options to be passed to compressor Compression engines allow options to be passed to them to control behavior. This patch exposes an argument to bundle2.writebundle() that passes options to the compression engine when writing compressed bundles. The argument is honored for both bundle1 and bundle2, the latter requiring a bit of plumbing to pass the value around.
(0) -30000 -10000 -3000 -1000 -120 +120 +1000 +3000 +10000 tip