Fri, 09 Mar 2018 15:57:16 +0100 graft: check for missing revision first before scanning working copy
Joerg Sonnenberger <joerg@bec.de> [Fri, 09 Mar 2018 15:57:16 +0100] rev 36842
graft: check for missing revision first before scanning working copy Differential Revision: https://phab.mercurial-scm.org/D2753
Sat, 10 Mar 2018 22:02:58 -0500 hook: ensure stderr is flushed when an exception is raised, for test stability
Matt Harbison <matt_harbison@yahoo.com> [Sat, 10 Mar 2018 22:02:58 -0500] rev 36841
hook: ensure stderr is flushed when an exception is raised, for test stability Windows has had issues with output order in test-ssh-proto-unbundle.t[1] since it was created a few weeks ago. Each of the problems occurred when an exception was thrown out of the hook. Now the only thing blocking D2720 is the fact that the "abort: ..." lines on stderr are totally AWOL. I have no idea where there are. [1] https://buildbot.mercurial-scm.org/builders/Win7%20x86_64%20hg%20tests/builds/541/steps/run-tests.py%20%28python%202.7.13%29/logs/stdio
Sat, 10 Mar 2018 10:27:56 -0800 wireproto: raise ProgrammingError instead of Abort
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 10 Mar 2018 10:27:56 -0800] rev 36840
wireproto: raise ProgrammingError instead of Abort This isn't a user-facing error and can only be caused by bad Python code. Thanks to Yuya for spotting this. Differential Revision: https://phab.mercurial-scm.org/D2777
Sat, 10 Mar 2018 19:56:47 +0900 py3: make test-commit-interactive.t byte-safe
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Mar 2018 19:56:47 +0900] rev 36839
py3: make test-commit-interactive.t byte-safe
Sat, 10 Mar 2018 19:49:09 +0900 py3: open patch file in binary mode and convert eol manually
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Mar 2018 19:49:09 +0900] rev 36838
py3: open patch file in binary mode and convert eol manually Here we don't introduce a reader wrapper since it wouldn't be easy to make read(n) handle partial data and length correctly.
Tue, 06 Mar 2018 07:45:57 -0600 py3: wrap file object to write patch in native eol preserving byte-ness
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 07:45:57 -0600] rev 36837
py3: wrap file object to write patch in native eol preserving byte-ness
Tue, 06 Mar 2018 07:24:12 -0600 py3: drop b'' from debug message "moving bookmarks"
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 07:24:12 -0600] rev 36836
py3: drop b'' from debug message "moving bookmarks"
Sat, 10 Mar 2018 15:57:16 +0900 py3: use r'' instead of sysstr('') to get around code transformer
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Mar 2018 15:57:16 +0900] rev 36835
py3: use r'' instead of sysstr('') to get around code transformer Fewer function calls should be better.
Sat, 10 Mar 2018 15:50:09 +0900 ui: remove any combinations of CR|LF from prompt response
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Mar 2018 15:50:09 +0900] rev 36834
ui: remove any combinations of CR|LF from prompt response On Windows, we have to accept both CR+LF and LF. This patch simply makes any trailing CRs and LFs removed from a user input instead of doing stricter parsing, as an input must be a readable text.
Sat, 10 Mar 2018 12:45:10 -0500 sshpeer: check pipe validity before forwarding output from it
Matt Harbison <matt_harbison@yahoo.com> [Sat, 10 Mar 2018 12:45:10 -0500] rev 36833
sshpeer: check pipe validity before forwarding output from it After the previous fix, fileobjectproxy._observedcall() (called when win32.peekpipe() accesses .fileno) started exploding. With this fix, similar checks are needed inside debugwireproto(). Since that is hardcoded to not use os.devnull, IDK if those are worth fixing.
Sat, 10 Mar 2018 12:22:08 -0500 util: forward __bool__()/__nonzero__() on fileobjectproxy
Matt Harbison <matt_harbison@yahoo.com> [Sat, 10 Mar 2018 12:22:08 -0500] rev 36832
util: forward __bool__()/__nonzero__() on fileobjectproxy In trying to debug the Windows process hang in D2720, I changed the stderr pipe to the peer to be os.devnull instead. That caused sshpeer._cleanuppipes()[1] to explode, complaining NoneType has no __iter__ attribute, even though the previous line checked for None. [1] https://www.mercurial-scm.org/repo/hg/file/b434965f984e/mercurial/sshpeer.py#l133
Tue, 06 Mar 2018 07:16:41 -0600 py3: fix slicing of bisect label in templatefilters.shortbisect()
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 07:16:41 -0600] rev 36831
py3: fix slicing of bisect label in templatefilters.shortbisect()
Tue, 06 Mar 2018 07:15:01 -0600 templatefilters: inline hbisect.shortlabel()
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 07:15:01 -0600] rev 36830
templatefilters: inline hbisect.shortlabel() It's pretty simple. I don't think the business logic has to be placed in hbisect.py.
Tue, 06 Mar 2018 07:11:24 -0600 py3: make test-bisect.t bytes-safe
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 07:11:24 -0600] rev 36829
py3: make test-bisect.t bytes-safe
Tue, 06 Mar 2018 07:10:50 -0600 py3: fix integer formatting in bisect error
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 07:10:50 -0600] rev 36828
py3: fix integer formatting in bisect error
Sat, 10 Mar 2018 16:55:54 +0900 py3: silence f.write() in test-annotate.t
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Mar 2018 16:55:54 +0900] rev 36827
py3: silence f.write() in test-annotate.t
Fri, 09 Mar 2018 14:52:36 -0800 xdiff: resolve signed unsigned comparison warning
Jun Wu <quark@fb.com> [Fri, 09 Mar 2018 14:52:36 -0800] rev 36826
xdiff: resolve signed unsigned comparison warning Since the value won't be changed inside the code (because context lines feature was removed by D2705), let's just remove the variable and inline the 0 value. The code might be potentially further simplified. But I'd like to make sure correctness is easily verifiable in this patch. Differential Revision: https://phab.mercurial-scm.org/D2766
Fri, 09 Mar 2018 14:47:29 -0800 xdiff: use int64 for hash table size
Jun Wu <quark@fb.com> [Fri, 09 Mar 2018 14:47:29 -0800] rev 36825
xdiff: use int64 for hash table size Follow-up of the previous "long" -> "int64" change. Now xdiff only uses int for return values and small integers (ex. booleans, shifting score, bits in hash table size, etc) so it should be able to handle large input. Differential Revision: https://phab.mercurial-scm.org/D2765
Fri, 09 Mar 2018 14:39:35 -0800 xdiff: remove unused xpp and xecfg parameters
Jun Wu <quark@fb.com> [Fri, 09 Mar 2018 14:39:35 -0800] rev 36824
xdiff: remove unused xpp and xecfg parameters They are unused. Thus removed. Differential Revision: https://phab.mercurial-scm.org/D2764
Fri, 09 Mar 2018 14:37:55 -0800 xdiff: remove unused flags parameter
Jun Wu <quark@fb.com> [Fri, 09 Mar 2018 14:37:55 -0800] rev 36823
xdiff: remove unused flags parameter After D2683, the flags parameter in some functions is no longer needed. Thus removed. Differential Revision: https://phab.mercurial-scm.org/D2763
Fri, 09 Mar 2018 14:24:27 -0800 xdiff: replace {unsigned ,}long with {u,}int64_t
Jun Wu <quark@fb.com> [Fri, 09 Mar 2018 14:24:27 -0800] rev 36822
xdiff: replace {unsigned ,}long with {u,}int64_t MSVC treats "long" as 4-byte. That could cause overflows since the xdiff code uses "long" in places where "size_t" or "ssize_t" should be used. Let's use explicit 8 byte integers to avoid FWIW git avoids that overflow by limiting diff size to 1GB [1]. After examining the code, I think the remaining risk (the use of "int") is low since "int" is only used for return values and hash table size. Although a wrong hash table size would not affect the correctness of the code, but that could make the code extremely slow. The next patch will change hash table size to 8-byte integer so the 1GB limit is unlikely needed. This patch was done by using `sed`. [1]: https://github.com/git/git/commit/dcd1742e56ebb944c4ff62346da4548e1e3be67 Differential Revision: https://phab.mercurial-scm.org/D2762
Sun, 04 Mar 2018 11:30:16 -0800 xdiff: add comments for fields in xdfile_t
Jun Wu <quark@fb.com> [Sun, 04 Mar 2018 11:30:16 -0800] rev 36821
xdiff: add comments for fields in xdfile_t This makes the related code easier to understand. Differential Revision: https://phab.mercurial-scm.org/D2685
Wed, 07 Mar 2018 14:45:31 -0800 xdiff: add a preprocessing step that trims files
Jun Wu <quark@fb.com> [Wed, 07 Mar 2018 14:45:31 -0800] rev 36820
xdiff: add a preprocessing step that trims files xdiff has a `xdl_trim_ends` step that removes common lines, unmatchable lines. That is in theory good, but happens too late - after splitting, hashing, and adjusting the hash values so they are unique. Those splitting, hashing and adjusting hash values steps could have noticeable overhead. Diffing two large files with minor (one-line-ish) changes are not uncommon. In that case, the raw performance of those preparation steps seriously matter. Even allocating an O(N) array and storing line offsets to it is expensive. Therefore my previous attempts [1] [2] cannot be good enough since they do not remove the O(N) array assignment. This patch adds a preprocessing step - `xdl_trim_files` that runs before other preprocessing steps. It counts common prefix and suffix and lines in them (needed for displaying line number), without doing anything else. Testing with a crafted large (169MB) file, with minor change: ``` open('a','w').write(''.join('%s\n' % (i % 100000) for i in xrange(30000000) if i != 6000000)) open('b','w').write(''.join('%s\n' % (i % 100000) for i in xrange(30000000) if i != 6003000)) ``` Running xdiff by a simple binary [3], this patch improves the xdiff perf by more than 10x for the above case: ``` # xdiff before this patch 2.41s user 1.13s system 98% cpu 3.592 total # xdiff after this patch 0.14s user 0.16s system 98% cpu 0.309 total # gnu diffutils 0.12s user 0.15s system 98% cpu 0.272 total # (best of 20 runs) ``` It's still slightly slower than GNU diffutils. But it's pretty close now. Testing with real repo data: For the whole repo, this patch makes xdiff 25% faster: ``` # hg perfbdiff --count 100 --alldata -c d334afc585e2 --blocks [--xdiff] # xdiff, after ! wall 0.058861 comb 0.050000 user 0.050000 sys 0.000000 (best of 100) # xdiff, before ! wall 0.077816 comb 0.080000 user 0.080000 sys 0.000000 (best of 91) # bdiff ! wall 0.117473 comb 0.120000 user 0.120000 sys 0.000000 (best of 67) ``` For files that are long (ex. commands.py), the speedup is more than 3x, very significant: ``` # hg perfbdiff --count 3000 --blocks commands.py.i 1 [--xdiff] # xdiff, after ! wall 0.690583 comb 0.690000 user 0.690000 sys 0.000000 (best of 12) # xdiff, before ! wall 2.240361 comb 2.210000 user 2.210000 sys 0.000000 (best of 4) # bdiff ! wall 2.469852 comb 2.440000 user 2.440000 sys 0.000000 (best of 4) ``` [1]: https://phab.mercurial-scm.org/D2631 [2]: https://phab.mercurial-scm.org/D2634 [3]: ``` // Code to run xdiff from command line. No proper error handling. #include <stdlib.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include "mercurial/thirdparty/xdiff/xdiff.h" #define ensure(x) if (!(x)) exit(255); mmfile_t readfile(const char *path) { struct stat st; int fd = open(path, O_RDONLY); fstat(fd, &st); mmfile_t file = { malloc(st.st_size), st.st_size }; ensure(read(fd, file.ptr, st.st_size) == st.st_size); close(fd); return file; } int main(int argc, char const *argv[]) { mmfile_t a = readfile(argv[1]), b = readfile(argv[2]); xpparam_t xpp = {0}; xdemitconf_t xecfg = {0}; xdemitcb_t ecb = {0}; xdl_diff(&a, &b, &xpp, &xecfg, &ecb); return 0; } ``` Differential Revision: https://phab.mercurial-scm.org/D2686
Fri, 09 Mar 2018 14:30:15 -0800 transaction: add a name and a __repr__ implementation (API)
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Mar 2018 14:30:15 -0800] rev 36819
transaction: add a name and a __repr__ implementation (API) This has been useful for me for debugging. Differential Revision: https://phab.mercurial-scm.org/D2758
Fri, 09 Mar 2018 16:10:55 +0100 phabricator: update doc string for deprecated token argument
Joerg Sonnenberger <joerg@bec.de> [Fri, 09 Mar 2018 16:10:55 +0100] rev 36818
phabricator: update doc string for deprecated token argument Differential Revision: https://phab.mercurial-scm.org/D2755
Fri, 09 Mar 2018 16:09:27 +0100 phabricator: print deprecation warning only once
Joerg Sonnenberger <joerg@bec.de> [Fri, 09 Mar 2018 16:09:27 +0100] rev 36817
phabricator: print deprecation warning only once Differential Revision: https://phab.mercurial-scm.org/D2754
Thu, 08 Mar 2018 21:17:26 -0800 tests: add a few tests involving --collapse and rebase.singletransaction=1
Martin von Zweigbergk <martinvonz@google.com> [Thu, 08 Mar 2018 21:17:26 -0800] rev 36816
tests: add a few tests involving --collapse and rebase.singletransaction=1 I'm about to change the rebase code quite a bit and this was poorly tested. Differential Revision: https://phab.mercurial-scm.org/D2757
Thu, 08 Mar 2018 20:55:51 -0800 tests: simplify test-rebase-transaction.t
Martin von Zweigbergk <martinvonz@google.com> [Thu, 08 Mar 2018 20:55:51 -0800] rev 36815
tests: simplify test-rebase-transaction.t The file was extracted from test-rebase-base.t in 8cef8f7d51d0 (test-rebase-base: clarify it is about the "--base" flag, 2017-10-05). This patch follows up that and clarifies the new file's purpose and simplifies it a bit. Differential Revision: https://phab.mercurial-scm.org/D2756
Thu, 08 Mar 2018 16:22:25 -0800 hgweb: parse and store HTTP request headers
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 16:22:25 -0800] rev 36814
hgweb: parse and store HTTP request headers WSGI transmits HTTP request headers as HTTP_* environment variables. We teach our parser about these and hook up a dict-like data structure that supports case insensitive header manipulation. Differential Revision: https://phab.mercurial-scm.org/D2742
Thu, 08 Mar 2018 16:43:32 -0800 wireprotoserver: remove broken optimization for non-httplib client
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 16:43:32 -0800] rev 36813
wireprotoserver: remove broken optimization for non-httplib client There was an experimental non-httplib client in core for several years. It was removed a week or so ago. We kept the optimization for this client in the server code. I'm not sure if that was intended or not. But it doesn't matter: the code was wrong. Because the code was accessing a WSGI environment dict, it needed to access the HTTP_X_HGHTTP2 key to actually read the HTTP header. So the code deleted by this commit wasn't actually doing anything meaningful. Doh. Differential Revision: https://phab.mercurial-scm.org/D2741
Thu, 08 Mar 2018 15:58:52 -0800 wireprotoserver: move all wire protocol handling logic out of hgweb
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 15:58:52 -0800] rev 36812
wireprotoserver: move all wire protocol handling logic out of hgweb Previous patches from several days ago worked to isolate processing of HTTP wire protocol requests to wireprotoserver. We still had a little logic in hgweb. If feels like the right time to finish the job. This commit moves WSGI request servicing from hgweb to wireprotoserver. The ugly dict holding the parsed request is no more. I think the new code is cleaner. As part of this, we now process wire protocol requests before the block to obtain the "query" variable. This makes it clear that this wonky "query" variable is not used by the wire protocol. The wonkiest part about this code is the HTTP 404. I'm actually not sure what all is going on here. It looks like the code is trying to prevent URL with path components that specify a command from not working. That part I grok. What I don't grok is why we need to send a 404. I would think it would be OK to no-op and let another handler try to service the request. But if we do this, we get some subrepo test failures. So it looks like something is expecting the HTTP 404 and reacting to it in a specific way. It /might/ be possible to change the behavior here. But it isn't something I'm comfortable doing because I don't understand the problem space. Differential Revision: https://phab.mercurial-scm.org/D2740
Thu, 08 Mar 2018 15:37:05 -0800 hgweb: use parsed request to construct query parameters
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 15:37:05 -0800] rev 36811
hgweb: use parsed request to construct query parameters The way hgweb routes requests is kind of bonkers. If PATH_INFO is set, we take the URL path after the repository. Otherwise, we take the first part of the query string before "&" and the part before ";" in that. We then kinda/sorta treat this as a path and route based on that. This commit ports that code to use the parsed request object. This required a new attribute on the parsed request to indicate whether there is any PATH_INFO. The new code still feels a bit convoluted for my liking. But we'll need to rewrite more of the code before a better solution becomes apparant. This code feels strictly better since we're no longer doing low-level WSGI manipulation during routing. Differential Revision: https://phab.mercurial-scm.org/D2739
Thu, 08 Mar 2018 11:33:33 -0800 hgweb: only recognize wire protocol commands from query string (BC)
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 11:33:33 -0800] rev 36810
hgweb: only recognize wire protocol commands from query string (BC) Previously, we attempted to parse the wire protocol command from `req.form`. Data could have come from the query string or POST form data. The wire protocol states that the command must be declared in the query string. And AFAICT all Mercurial releases from at least 1.0 send the command in the query string. So let's actual require this behavior. This is technically BC. But I'm not sure how anyone in the wild would encounter this. POST has historically been used for sending bundle data. So there's no opportunity to encode arguments there. And the experimental HTTP POST args also takes over the body. So the only way someone would be impacted by this is if they wrote a custom client that both used POST for everything and sent arguments via the HTTP body. I don't believe such a client exists. .. bc:: The HTTP wire protocol server no longer accepts the ``cmd`` argument to control which command to run via HTTP POST bodies. The ``cmd`` argument must be specified on the URL query string. Differential Revision: https://phab.mercurial-scm.org/D2738
Thu, 08 Mar 2018 11:21:46 -0800 hgweb: teach WSGI parser about query strings
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 11:21:46 -0800] rev 36809
hgweb: teach WSGI parser about query strings Currently, req.form uses cgi.parse() to populate form data. Depending on the request, form data can come from POST multipart/form-data, application/x-www-form-urlencoded, or the URL query string. Putting all these things into one data structure makes it difficult to reason about how exactly parameters got to the request. It can lead to wonkiness such as pulling parameters from both the URL and POST data. This commit teaches our WSGI request parser about argument data in query strings. We populate fields containing the query string data and only the query string data so it can't be confused with POST data. Differential Revision: https://phab.mercurial-scm.org/D2737
Thu, 08 Mar 2018 15:08:20 -0800 hgweb: use the parsed application path directly
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 15:08:20 -0800] rev 36808
hgweb: use the parsed application path directly Previously, we assigned a custom system string with a trailing slash to wsgirequest.url. The addition of the trailing slash felt arbitrary and seems to go against how things typically work in WSGI. We also want our URLs to be bytes, not system strings. And, assigning a custom attribute to wsgirequest felt wrong. This commit fixes all those things by removing the trailing slash from the app path, changing consumers to use that variable and to use it without a trailing slash, and removing the custom attribute from wsgirequest. We preserve the trailing slash on {url}. Also, makebreadcrumb strips the trailing slash. So no change to it was needed. Differential Revision: https://phab.mercurial-scm.org/D2736
Thu, 08 Mar 2018 12:59:25 -0800 hgweb: use computed base URL from parsed request
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 12:59:25 -0800] rev 36807
hgweb: use computed base URL from parsed request Let's not reinvent URL construction in a function that runs the templater. Differential Revision: https://phab.mercurial-scm.org/D2735
Sat, 10 Mar 2018 10:20:51 -0800 hgweb: parse WSGI request into a data structure
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 10 Mar 2018 10:20:51 -0800] rev 36806
hgweb: parse WSGI request into a data structure Currently, our WSGI applications (hgweb_mod and hgwebdir_mod) process the raw WSGI request instance themselves. This means they have to talk in terms of system strings. And they need to know details about what's in the WSGI request. And in the case of hgweb_mod, it is doing some very funky things with URL parsing to impact dispatching. The code is difficult to read and maintain. This commit introduces parsing of the WSGI request into a higher-level and easier-to-reason-about data structure. To prove it works, we hook it up to hgweb_mod and use it for populating the relative URL on the request instance. We hold off on using it in more places because the logic in hgweb_mod is crazy and I don't want to involve those changes with review of the parsing code. The URL construction code has variations that use the HTTP: Host header (the canonical WSGI way of reconstructing the URL) and with the use of SERVER_NAME. We need to differentiate because hgweb is currently using SERVER_NAME for URL construction. Differential Revision: https://phab.mercurial-scm.org/D2734
Thu, 08 Mar 2018 15:14:32 -0800 hgweb: always use "?" when writing session vars
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 15:14:32 -0800] rev 36805
hgweb: always use "?" when writing session vars This code resolves a string to insert in URLs as part of a query string. Essentially, it resolves the {sessionvars} template keyword, which is used by hgweb templates to build a URL as a string. The whole approach here feels wrong because there's no way of knowing when this code runs how the final URL will look. There could be additional URL fragments added before this template keyword that add a query string component. Furthermore, I don't think there's *any* for req.url to have a query string. That's because the code that populates this variable only takes SCRIPT_NAME and REPO_NAME into account. The "?" character it is searching for would only be added if some code attempted to add QUERY_STRING to the URL. Hacking the code up to raise if "?" is present in the URL yields a clean test suite run. I'm not sure if we broke this code or if it has always been broken. Anyway, this commit removes support for emitting "&" as the first character in {sessionvars} and makes it always emit "?", which is what it was always doing before AFAICT. Differential Revision: https://phab.mercurial-scm.org/D2733
Thu, 08 Mar 2018 15:15:59 -0800 hgweb: rename req to wsgireq
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 15:15:59 -0800] rev 36804
hgweb: rename req to wsgireq We will soon introduce a parsed WSGI request object so we don't have to concern ourselves with low-level WSGI matters. Prepare for multiple request objects by renaming the existing one so it is clear it deals with WSGI. We also remove a symbol import to avoid even more naming confusion. # no-check-commit because of some new foo_bar naming that's required Differential Revision: https://phab.mercurial-scm.org/D2732
Thu, 08 Mar 2018 09:44:27 -0800 hgweb: validate WSGI environment dict
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 09:44:27 -0800] rev 36803
hgweb: validate WSGI environment dict The wsgiref.validate module contains useful functions for validating that various WSGI data structures are proper. This commit adds validation of the environment dict to our built-in HTTP server, which turns an HTTP request into an environment dict. The check discovered that we weren't always setting QUERY_STRING, which would cause the cgi module to fall back to sys.argv. So we change things to always set QUERY_STRING. The check passes on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D2731
Thu, 08 Mar 2018 09:26:51 -0800 hgweb: ensure all wsgi environment values are str
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 08 Mar 2018 09:26:51 -0800] rev 36802
hgweb: ensure all wsgi environment values are str Previously, we had a few entries that were bytes on Python 3. PEP-0333 states that all entries must be the native str type (bytes on Python 2, str on Python 3). This required a number of changes to hgweb_mod to unbreak things on Python 3. I suspect there still may be some regressions. I'm going to introduce a data structure that represents a parsed WSGI request in upcoming commits. This will hold bytes and will allow us to stop using raw literals throughout the WSGI code. Differential Revision: https://phab.mercurial-scm.org/D2730
Wed, 07 Mar 2018 16:18:52 -0800 wireproto: formalize permissions checking as part of protocol interface
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 07 Mar 2018 16:18:52 -0800] rev 36801
wireproto: formalize permissions checking as part of protocol interface Per the inline comment desiring to formalize permissions checking in the protocol interface, we do that. I'm not convinced this is the best way to go about things. I would love for there to e.g. be a better exception for denoting permissions problems. But it does feel strictly better than snipping attributes on the proto instance. Differential Revision: https://phab.mercurial-scm.org/D2719
Wed, 07 Mar 2018 16:02:24 -0800 wireproto: declare permissions requirements in @wireprotocommand (API)
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 07 Mar 2018 16:02:24 -0800] rev 36800
wireproto: declare permissions requirements in @wireprotocommand (API) With the security patches from 4.5.2 merged into default, we now have a per-command attribute defining what permissions are needed to run that command. We now have a richer @wireprotocommand that can be extended to record additional command metadata. So we port the permissions mechanism to be based on @wireprotocommand. .. api:: hgweb_mod.perms and wireproto.permissions have been removed. Wire protocol commands should declare their required permissions in the @wireprotocommand decorator. Differential Revision: https://phab.mercurial-scm.org/D2718
Tue, 06 Mar 2018 15:08:33 -0800 wireprotoserver: check permissions in main dispatch function
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 06 Mar 2018 15:08:33 -0800] rev 36799
wireprotoserver: check permissions in main dispatch function The permissions checking code merged from stable is out of place in the refactored hgweb_mod module. This commit moves the main call to wireprotoserver. We still have some lingering code in hgweb_mod. This will get addressed later. Differential Revision: https://phab.mercurial-scm.org/D2717
Tue, 06 Mar 2018 15:02:53 -0800 wireprotoserver: check if command available before calling it
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 06 Mar 2018 15:02:53 -0800] rev 36798
wireprotoserver: check if command available before calling it The previous behavior was just plain wrong. I have no clue how it landed. My guess is a merge conflict resolution gone wrong on my end a few weeks ago. Differential Revision: https://phab.mercurial-scm.org/D2716
Tue, 06 Mar 2018 02:43:17 -0600 py3: drop encoding.strio()
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 02:43:17 -0600] rev 36797
py3: drop encoding.strio() Its buffered nature makes TextIOWrapper unsuitable for temporarily wrapping bytes I/O.
Tue, 06 Mar 2018 02:42:37 -0600 ui: adjust Windows workaround to new _readline() code
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 02:42:37 -0600] rev 36796
ui: adjust Windows workaround to new _readline() code It's only needed when rawinput() is called. Also made it Py3 compatible.
Tue, 06 Mar 2018 02:38:53 -0600 ui: do not use rawinput() when we have to replace sys.stdin/stdout
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 02:38:53 -0600] rev 36795
ui: do not use rawinput() when we have to replace sys.stdin/stdout See the inline comment for why. The current Python3 hack doesn't work if more than one user inputs are expected because TextIOWrapper fills its internal buffer at the first read() request. Maybe we could write an unbuffered TextIOWrapper, but I don't want to make things more complicated. Instead, this patch reinvents raw_input(' ') of no readline support.
Tue, 06 Mar 2018 02:32:26 -0600 ui: do not try readline support if fin/fout aren't standard streams
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 02:32:26 -0600] rev 36794
ui: do not try readline support if fin/fout aren't standard streams It's unlikely for a non-stdio stream to be a tty. Minimizing readline support makes it much simpler to work around the unicode input() function of Python 3. This also works on chg which duplicates client's tty to stdio fds.
Tue, 06 Mar 2018 02:28:59 -0600 util: add public isstdin/isstdout() functions
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 02:28:59 -0600] rev 36793
util: add public isstdin/isstdout() functions
Tue, 06 Mar 2018 03:05:49 -0600 ui: add debug commands to test interactive prompt
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 03:05:49 -0600] rev 36792
ui: add debug commands to test interactive prompt Interactive operations aren't easily covered by tests. So let's add commands to test them manually.
Tue, 06 Mar 2018 02:14:11 -0600 ui: inline util.bytesinput() into ui._readline()
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 02:14:11 -0600] rev 36791
ui: inline util.bytesinput() into ui._readline() Prepares for rework of Python 3 support, which is currently broken due to read-ahead buffer of TextIOWrapper.
Tue, 06 Mar 2018 02:05:25 -0600 hgk: stop using util.bytesinput() to read a single line from stdin
Yuya Nishihara <yuya@tcha.org> [Tue, 06 Mar 2018 02:05:25 -0600] rev 36790
hgk: stop using util.bytesinput() to read a single line from stdin Appears that the stdio here is an IPC channel between hg and hgk (tk) processes, which shouldn't need a fancy readline support.
Mon, 29 Aug 2016 10:42:58 -0400 bookmarks: test for exchanging long bookmark names (issue5165)
Augie Fackler <augie@google.com> [Mon, 29 Aug 2016 10:42:58 -0400] rev 36789
bookmarks: test for exchanging long bookmark names (issue5165) As far as I can tell a test for a long bookmark name never actually got added when this was fixed. Let's document that a 300 byte bookmark is something we're supporting (this patch started out life over a year ago as a way for me to validate the problem, and I recently found it.) I think the nonzero exits from the push operations are only because no new changesets get exchanged. Please correct me if I'm wrong on that. :) Differential Revision: https://phab.mercurial-scm.org/D2727
Sun, 04 Mar 2018 11:46:03 -0500 phabricator: follow-up phab auth improvements with backwards compat mode
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 11:46:03 -0500] rev 36788
phabricator: follow-up phab auth improvements with backwards compat mode We'll rip this out before we ship 4.6, but this gives people a window to migrate. Differential Revision: https://phab.mercurial-scm.org/D2703
Sat, 20 Jan 2018 02:41:10 -0700 phabricator: specify API tokens per host, rather than per repo
Tom Prince <mozilla@hocat.ca> [Sat, 20 Jan 2018 02:41:10 -0700] rev 36787
phabricator: specify API tokens per host, rather than per repo Differential Revision: https://phab.mercurial-scm.org/D1919
Sun, 04 Mar 2018 18:47:07 -0500 py3: drop b'' from generate-working-copy-states.py output
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 18:47:07 -0500] rev 36786
py3: drop b'' from generate-working-copy-states.py output I could make everything bytes, but decoding filename seemed easier and we don't have to worry about non-ascii filenames here.
Sun, 04 Mar 2018 18:41:09 -0500 py3: make test-commit-multiple.t byte-safe
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 18:41:09 -0500] rev 36785
py3: make test-commit-multiple.t byte-safe
Sun, 04 Mar 2018 18:34:46 -0500 py3: fix type of default username
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 18:34:46 -0500] rev 36784
py3: fix type of default username
Sun, 04 Mar 2018 18:21:16 -0500 py3: read/write plain lock file in binary mode
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 18:21:16 -0500] rev 36783
py3: read/write plain lock file in binary mode A lock file shouldn't contain '\n', so this isn't a BC.
Mon, 05 Mar 2018 12:31:08 -0500 util: stop calling os.stat_float_times()
Augie Fackler <augie@google.com> [Mon, 05 Mar 2018 12:31:08 -0500] rev 36782
util: stop calling os.stat_float_times() It had Python-wide side effects, and it disappears in 3.7.0. As of this change, we're mostly working on 3.7.0b2. There are a few worrying failures, mostly around regular expressions, but we'll have to tackle those separately. Differential Revision: https://phab.mercurial-scm.org/D2697
Mon, 05 Mar 2018 12:30:20 -0500 cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime
Augie Fackler <augie@google.com> [Mon, 05 Mar 2018 12:30:20 -0500] rev 36781
cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime The latter is floating point by default, and we've been doing os.stat_float_times(False). Unfortunately, os.stat_float_times was removed between Python 3.7.0a1 and 3.7.0b2, so we have to stop using it. Differential Revision: https://phab.mercurial-scm.org/D2696
Mon, 05 Mar 2018 15:07:32 -0500 osutil: implement minimal __getitem__ compatibility on our custom listdir type
Augie Fackler <augie@google.com> [Mon, 05 Mar 2018 15:07:32 -0500] rev 36780
osutil: implement minimal __getitem__ compatibility on our custom listdir type We previously declined to do this, but the removal of the deprecated os.stat_float_times() method in Python 3.7 forces our hand. Differential Revision: https://phab.mercurial-scm.org/D2695
Sun, 04 Mar 2018 21:14:24 -0500 hgweb: adapt to socket._fileobject changes in Python 3
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 21:14:24 -0500] rev 36779
hgweb: adapt to socket._fileobject changes in Python 3 Differential Revision: https://phab.mercurial-scm.org/D2688
Sun, 04 Mar 2018 16:20:24 -0500 debugcommands: fix some %r output with bytestr() wrappers
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:20:24 -0500] rev 36778
debugcommands: fix some %r output with bytestr() wrappers Almost fixes test-merge-tools.t. I think the remaining failure there is due to some overspecified tempfile names. Differential Revision: https://phab.mercurial-scm.org/D2675
Wed, 07 Mar 2018 11:00:17 -0800 tests: add test for issue 5494 but with --collapse
Martin von Zweigbergk <martinvonz@google.com> [Wed, 07 Mar 2018 11:00:17 -0800] rev 36777
tests: add test for issue 5494 but with --collapse This was not fixed, so the test case currently demonstrates the breakage. Differential Revision: https://phab.mercurial-scm.org/D2714
Wed, 07 Mar 2018 10:55:57 -0800 tests: .hg/merge is a directory, so use `test -d`
Martin von Zweigbergk <martinvonz@google.com> [Wed, 07 Mar 2018 10:55:57 -0800] rev 36776
tests: .hg/merge is a directory, so use `test -d` This part of test-rebase-interrupts.t would have passed before the fix in a580b2d65ded (rebase: make sure merge state is cleaned up for no-op rebases (issue5494), 2017-05-18). Differential Revision: https://phab.mercurial-scm.org/D2713
Tue, 06 Mar 2018 14:29:20 -0800 rebase: only store collapse message once
Martin von Zweigbergk <martinvonz@google.com> [Tue, 06 Mar 2018 14:29:20 -0800] rev 36775
rebase: only store collapse message once The message is determined by the user passing --message or --log when the rebase is started. There's no need to write it to a file for each rebased commit; writing it once at the start of the rebase is enough. Differential Revision: https://phab.mercurial-scm.org/D2712
Tue, 06 Mar 2018 09:39:24 -0800 rebase: collapse two nested if-conditions
Martin von Zweigbergk <martinvonz@google.com> [Tue, 06 Mar 2018 09:39:24 -0800] rev 36774
rebase: collapse two nested if-conditions Also change the order since it feel to me like it's more about --collapse than it is about --keep. Differential Revision: https://phab.mercurial-scm.org/D2711
Thu, 01 Mar 2018 20:12:25 -0800 rebase: reduce scope of "dsguard" variables a bit
Martin von Zweigbergk <martinvonz@google.com> [Thu, 01 Mar 2018 20:12:25 -0800] rev 36773
rebase: reduce scope of "dsguard" variables a bit Differential Revision: https://phab.mercurial-scm.org/D2710
Wed, 07 Mar 2018 09:46:53 -0800 rebase: remove unused argument "state" from rebasenode()
Martin von Zweigbergk <martinvonz@google.com> [Wed, 07 Mar 2018 09:46:53 -0800] rev 36772
rebase: remove unused argument "state" from rebasenode() Differential Revision: https://phab.mercurial-scm.org/D2709
Wed, 07 Mar 2018 10:31:01 -0800 rebase: delete obsolete internal "keepopen" option
Martin von Zweigbergk <martinvonz@google.com> [Wed, 07 Mar 2018 10:31:01 -0800] rev 36771
rebase: delete obsolete internal "keepopen" option The option was apparently introduced for use by the "pbranch" extension, see f2558a8228be (rebase: add option to not commit after a collapsing, 2010-02-07). However, it doesn't seem like it was ever used by that extension (according to `hg grep` in a clone of https://bitbucket.org/parren/hg-pbranch/), so let's delete it. Differential Revision: https://phab.mercurial-scm.org/D2708
Sun, 04 Mar 2018 00:25:58 +0530 releasenotes: allow notes for multiple directives in a single changeset
Rishabh Madan <rishabhmadan96@gmail.com> [Sun, 04 Mar 2018 00:25:58 +0530] rev 36770
releasenotes: allow notes for multiple directives in a single changeset This problem was caught in da91e7309daf8ffc51bf3e6f4b2d8a16ef5af95a. This patch just makes sure there is no warning when we encounter such a case. Differential Revision: https://phab.mercurial-scm.org/D2254
Sun, 04 Mar 2018 00:15:35 +0530 releasenotes: mention changeset with warning and abort
Rishabh Madan <rishabhmadan96@gmail.com> [Sun, 04 Mar 2018 00:15:35 +0530] rev 36769
releasenotes: mention changeset with warning and abort Output the changeset hash with the warning/abort message just to know where things messed up. Differential Revision: https://phab.mercurial-scm.org/D2253
Sat, 03 Mar 2018 23:47:22 +0530 releasenotes: replace abort with warning while parsing (issue5775)
Rishabh Madan <rishabhmadan96@gmail.com> [Sat, 03 Mar 2018 23:47:22 +0530] rev 36768
releasenotes: replace abort with warning while parsing (issue5775) During the 4.5 development cycle, the extension broke on two different changesets. This fixes the issue by ensuring that it just throws a warning when it encounters unexpected behaviour, instead of aborting. Differential Revision: https://phab.mercurial-scm.org/D2255
Wed, 07 Mar 2018 09:07:34 +1100 archival: fileit should not use atomictemp, causes performance regression
Vincent Parrett <vincent@finalbuilder.com> [Wed, 07 Mar 2018 09:07:34 +1100] rev 36767
archival: fileit should not use atomictemp, causes performance regression Differential Revision: https://phab.mercurial-scm.org/D2704
Sat, 03 Mar 2018 18:55:43 -0500 perf: teach perfbdiff to call blocks() and to use xdiff
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 03 Mar 2018 18:55:43 -0500] rev 36766
perf: teach perfbdiff to call blocks() and to use xdiff Differential Revision: https://phab.mercurial-scm.org/D2624
Tue, 06 Mar 2018 19:31:17 -0800 fuzz: fix xdiff build
Jun Wu <quark@fb.com> [Tue, 06 Mar 2018 19:31:17 -0800] rev 36765
fuzz: fix xdiff build Recent xdiff code cleanups removed some files and changed some structures. Update fuzz code. Differential Revision: https://phab.mercurial-scm.org/D2707
Tue, 06 Mar 2018 18:51:11 -0800 xdiff: remove xmerge related logic
Jun Wu <quark@fb.com> [Tue, 06 Mar 2018 18:51:11 -0800] rev 36764
xdiff: remove xmerge related logic hg has its own merge algorithm with flexible config options. Differential Revision: https://phab.mercurial-scm.org/D2706
Tue, 06 Mar 2018 18:41:08 -0800 xdiff: remove xemit related logic
Jun Wu <quark@fb.com> [Tue, 06 Mar 2018 18:41:08 -0800] rev 36763
xdiff: remove xemit related logic xemit handles "diff formatting and output" with options like context lines, whether show function names, etc. That is handled more cleanly at a higher level in hg. Removing context line parameters would also make the trimming logic (D2686) cleaner and more confident. See [1]. [1]: https://github.com/git/git/commit/d2f82950a9226ae1102a7a97f03440a4bf8c6c09 Differential Revision: https://phab.mercurial-scm.org/D2705
Sun, 04 Mar 2018 00:17:49 -0800 xdiff: remove unused structure, functions, and constants
Jun Wu <quark@fb.com> [Sun, 04 Mar 2018 00:17:49 -0800] rev 36762
xdiff: remove unused structure, functions, and constants `bdiffparam_t` is unused. `xdl_fall_back_diff` is no longer used after D2573. `XDL_MMB_READONLY`, `XDL_MMF_ATOMIC` are unused. `XDL_BDOP*` are unused since there is no xdiff binary diff algorithm. `anchors` feature is not used. It's also relatively new in git. Differential Revision: https://phab.mercurial-scm.org/D2684
Sun, 04 Mar 2018 00:07:04 -0800 xdiff: remove whitespace related feature
Jun Wu <quark@fb.com> [Sun, 04 Mar 2018 00:07:04 -0800] rev 36761
xdiff: remove whitespace related feature In Mercurial, whitespace related handling are done at a higher level than the low-level diff algorithm so "ignore spaces". So it's not used by mdiff. Some of the upcoming optimizations would be more difficult with whitespace related features kept. So let's remove them. Differential Revision: https://phab.mercurial-scm.org/D2683
Tue, 06 Mar 2018 14:32:14 -0800 merge with stable
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 06 Mar 2018 14:32:14 -0800] rev 36760
merge with stable There were a handful of merge conflicts in the wire protocol code due to significant refactoring in default. When resolving the conflicts, I tried to produce the minimal number of changes to make the incoming security patches work with the new code. I will send some follow-up commits to get the security patches better integrated into default.
Tue, 06 Mar 2018 13:19:54 -0600 Added signature for changeset 8bba684efde7 stable
Kevin Bullock <kbullock@ringworld.org> [Tue, 06 Mar 2018 13:19:54 -0600] rev 36759
Added signature for changeset 8bba684efde7
Tue, 06 Mar 2018 13:19:52 -0600 Added tag 4.5.2 for changeset 8bba684efde7 stable
Kevin Bullock <kbullock@ringworld.org> [Tue, 06 Mar 2018 13:19:52 -0600] rev 36758
Added tag 4.5.2 for changeset 8bba684efde7
Tue, 06 Mar 2018 13:17:07 -0600 merge with security patches stable 4.5.2
Kevin Bullock <kbullock+mercurial@ringworld.org> [Tue, 06 Mar 2018 13:17:07 -0600] rev 36757
merge with security patches
Sun, 18 Feb 2018 17:20:38 -0800 hgweb: always perform permissions checks on protocol commands (BC) (SEC) stable
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Feb 2018 17:20:38 -0800] rev 36756
hgweb: always perform permissions checks on protocol commands (BC) (SEC) Previously, the HTTP request handling code would only perform permissions checking on a wire protocol command if that wire protocol command defined its permissions / operation type. This meant that commands (possibly provided by extensions) not defining their operation type would bypass permissions check. This could lead to exfiltration of data from servers and mutating repositories that were supposed to be read-only. This security issue has been present since the permissions table was introduced by d3147b4e3e8a in 2008. This commit changes the behavior of the HTTP server to always perform permissions checking for protocol requests. If an explicit permission for a wire protocol command is not defined, the server assumes the command can be used for writing and governs access accordingly. .. bc:: Wire protocol commands not defining their operation type in ``wireproto.PERMISSIONS`` are now assumed to be used for "push" operations and access control to run those commands is now enforced accordingly.
Tue, 20 Feb 2018 18:55:58 -0800 wireproto: check permissions when executing "batch" command (BC) (SEC) stable
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 20 Feb 2018 18:55:58 -0800] rev 36755
wireproto: check permissions when executing "batch" command (BC) (SEC) For as long as the "batch" command has existed (introduced by bd88561afb4b and first released as part of Mercurial 1.9), that command (like most wire commands introduced after 2008) lacked an entry in the hgweb permissions table. And since we don't verify permissions if an entry is missing from the permissions table, this meant that executing a command via "batch" would bypass all permissions checks. The security implications are significant: a Mercurial HTTP server would allow writes via "batch" wire protocol commands as long as the HTTP request were processed by Mercurial and the process running the Mercurial HTTP server had write access to the repository. The Mercurial defaults of servers being read-only and the various web.* config options to define access control were bypassed. In addition, "batch" could be used to exfiltrate data from servers that were configured to not allow read access. Both forms of permissions bypass could be mitigated to some extent by using HTTP authentication. This would prevent HTTP requests from hitting Mercurial's server logic. However, any authenticated request would still be able to bypass permissions checks via "batch" commands. The easiest exploit was to send "pushkey" commands via "batch" and modify the state of bookmarks, phases, and obsolescence markers. However, I suspect a well-crafted HTTP request could trick the server into running the "unbundle" wire protocol command, effectively performing a full `hg push` to create new changesets on the remote. This commit plugs this gaping security hole by having the "batch" command perform permissions checking on each sub-command that is being batched. We do this by threading a permissions checking callable all the way to the protocol handler. The threading is a bit hacky from a code perspective. But it preserves API compatibility, which is the proper thing to do on the stable branch. One of the subtle things we do is assume that a command with an undefined permission is a "push" command. This is the safest thing to do from a security perspective: we don't want to take chances that a command could perform a write even though the server is configured to not allow writes. As the test changes demonstrate, it is no longer possible to bypass permissions via the "batch" wire protocol command. .. bc:: The "batch" wire protocol command now enforces permissions of each invoked sub-command. Wire protocol commands must define their operation type or the "batch" command will assume they can write data and will prevent their execution on HTTP servers unless the HTTP request method is POST, the server is configured to allow pushes, and the (possibly authenticated) HTTP user is authorized to perform a push.
Tue, 20 Feb 2018 18:54:27 -0800 wireproto: declare operation type for most commands (BC) (SEC) stable
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 20 Feb 2018 18:54:27 -0800] rev 36754
wireproto: declare operation type for most commands (BC) (SEC) The permissions model of hgweb relies on a dictionary to declare the operation associated with each command - either "pull" or "push." This dictionary was established by d3147b4e3e8a in 2008. Unfortunately, we neglected to update this dictionary as new wire protocol commands were introduced. This commit defines the operations of most wire protocol commands in the permissions dictionary. The "batch" command is omitted because it is special and requires a more complex solution. Since permissions checking is skipped unless a command has an entry in this dictionary (this security issue will be addressed in a subsequent commit), the practical effect of this change is that various wire protocol commands now HTTP 401 if web.deny_read or web.allow-pull, etc are set to deny access. This is reflected by test changes. Note how various `hg pull` and `hg push` operations now fail before discovery. (They fail during the initial "capabilities" request.) This change fixes a security issue where built-in wire protocol commands would return repository data even if the web config were configured to deny access to that data. I'm on the fence as to whether we should HTTP 401 the capabilities request. On one hand, it can expose repository metadata and can tell callers things like what version of Mercurial the server is running. On the other hand, a client may need to know the capabilities in order to authenticate in a follow-up request. It appears that Mercurial clients handle the HTTP 401 on *any* protocol request, so we should be OK sending a 401 for "capabilities." But if this causes problems, it should be possible to allow "capabilities" to always work. .. bc:: Various read-only wire protocol commands now return HTTP 401 Unauthorized if the hgweb configuration denies read/pull access to the repository. Previously, various wire protocol commands would still work and return data if read access was disabled.
Tue, 20 Feb 2018 18:53:39 -0800 wireproto: move command permissions dict out of hgweb_mod stable
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 20 Feb 2018 18:53:39 -0800] rev 36753
wireproto: move command permissions dict out of hgweb_mod The operation type associated with wire protocol commands is supposed to be defined in a dictionary so it can be used for permissions checking. Since this metadata is closely associated with wire protocol commands themselves, it makes sense to define it in the same module where wire protocol commands are defined. This commit moves hgweb_mod.perms to wireproto.PERMISSIONS and updates most references in the code to use the new home. The old symbol remains an alias for the new symbol. Tests pass with the code pointing at the old symbol. So this should be API compatible for extensions. As part of the code move, we split up the assignment to the dict so it is next to the @wireprotocommand. This reinforces that a @wireprotocommand should have an entry in this dict. In the future, we'll want to declare permissions as part of the @wireprotocommand decorator. But this isn't appropriate for the stable branch.
Tue, 20 Feb 2018 19:09:01 -0800 tests: comprehensively test HTTP server permissions checking stable
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 20 Feb 2018 19:09:01 -0800] rev 36752
tests: comprehensively test HTTP server permissions checking We didn't have test coverage for numerous web.* config options. We add that test coverage. Included in the tests are tests for custom commands. We have commands that are supposedly read-only and perform writes and a variation of each that does and does not define its operation type in hgweb_mod.perms. The tests reveal a handful of security bugs related to permissions checking. Subsequent commits will address these security bugs.
Sun, 18 Feb 2018 10:40:49 -0800 tests: extract HTTP permissions tests to own test file stable
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 18 Feb 2018 10:40:49 -0800] rev 36751
tests: extract HTTP permissions tests to own test file We're about to implement a lot more coverage of the permissions mechanism. In preparation for that, establish a new test file to hold permissions checks. As part of this, we inline the important parts of the "req" helper function.
Tue, 06 Mar 2018 13:08:00 -0600 Added signature for changeset 369aadf7a326 stable
Kevin Bullock <kbullock@ringworld.org> [Tue, 06 Mar 2018 13:08:00 -0600] rev 36750
Added signature for changeset 369aadf7a326
Tue, 06 Mar 2018 13:07:58 -0600 Added tag 4.5.1 for changeset 369aadf7a326 stable
Kevin Bullock <kbullock@ringworld.org> [Tue, 06 Mar 2018 13:07:58 -0600] rev 36749
Added tag 4.5.1 for changeset 369aadf7a326
Tue, 13 Feb 2018 11:35:32 -0800 revlog: resolve lfs rawtext to vanilla rawtext before applying delta stable 4.5.1
Jun Wu <quark@fb.com> [Tue, 13 Feb 2018 11:35:32 -0800] rev 36748
revlog: resolve lfs rawtext to vanilla rawtext before applying delta This happens when a LFS delta base gets a non-LFS delta from another client. In that case, the LFS delta base needs to be converted to non-LFS version before applying the delta. Differential Revision: https://phab.mercurial-scm.org/D2069
Sun, 04 Mar 2018 21:16:36 -0500 sslutil: some more forcebytes() on some exception messages
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 21:16:36 -0500] rev 36747
sslutil: some more forcebytes() on some exception messages At this point, test-https.t no longer dumps tracebacks everywhere. Instead, we get some results that look like we're not adequately finding things in hg's configuration, which should be manageable (if somewhat annoying to find and fix.) Differential Revision: https://phab.mercurial-scm.org/D2690
Sun, 04 Mar 2018 21:15:37 -0500 sslutil: sslcontext needs the cipher name as a sysstr
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 21:15:37 -0500] rev 36746
sslutil: sslcontext needs the cipher name as a sysstr Differential Revision: https://phab.mercurial-scm.org/D2689
Sun, 04 Mar 2018 18:03:55 -0500 sslutil: lots of unicode/bytes cleanup
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 18:03:55 -0500] rev 36745
sslutil: lots of unicode/bytes cleanup In general, we handle hostnames as bytes, except where Python forces them to be unicodes. This fixes all the tracebacks I was seeing in test-https.t, but there's still some ECONNRESET weirdness that I can't hunt down... Differential Revision: https://phab.mercurial-scm.org/D2687
Tue, 13 Feb 2018 11:35:32 -0800 revlog: do not use delta for lfs revisions stable
Jun Wu <quark@fb.com> [Tue, 13 Feb 2018 11:35:32 -0800] rev 36744
revlog: do not use delta for lfs revisions This is similar to what we have done for changegroups. It is needed to make sure the delta application code path can assume deltas are always against vanilla (ex. non-LFS) rawtext so the next fix becomes possible. Differential Revision: https://phab.mercurial-scm.org/D2068
Tue, 06 Feb 2018 19:08:25 -0800 changegroup: do not delta lfs revisions stable
Jun Wu <quark@fb.com> [Tue, 06 Feb 2018 19:08:25 -0800] rev 36743
changegroup: do not delta lfs revisions There is no way to distinguish whether a delta base is LFS or non-LFS. If the delta is against LFS rawtext, and the client trying to apply it has the base revision stored as fulltext, the delta (aka. bundle) will fail to apply. This patch forbids using delta for LFS revisions in changegroup so bad deltas won't be transmitted. Note: this does not solve the problem entirely. It solves LFS delta applying to non-LFS base. But the other direction: non-LFS delta applying to LFS base is not solved yet. Differential Revision: https://phab.mercurial-scm.org/D2067
Tue, 06 Feb 2018 16:08:57 -0800 lfs: add a test showing bundle application could be broken stable
Jun Wu <quark@fb.com> [Tue, 06 Feb 2018 16:08:57 -0800] rev 36742
lfs: add a test showing bundle application could be broken When a bundle containing LFS delta uses non-LFS delta-base, or vice-versa, the bundle will fail to apply. Differential Revision: https://phab.mercurial-scm.org/D2066
Mon, 05 Mar 2018 20:22:34 -0500 debugwireproto: handle unimplemented util.poll() for Windows
Matt Harbison <matt_harbison@yahoo.com> [Mon, 05 Mar 2018 20:22:34 -0500] rev 36741
debugwireproto: handle unimplemented util.poll() for Windows This is the same logic used in sshpeer.doublepipe. It doesn't completely fix test-ssh-proto{,-unbundle}.t ("read(-1) -> X" is changed to "read(X) -> X", the order of some lines are changed, and abort messages seem to be missing), but it cuts down a ton on the failure spew.
Sun, 04 Mar 2018 16:55:51 -0500 py3: byte-stringify test-blackbox.t
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 16:55:51 -0500] rev 36740
py3: byte-stringify test-blackbox.t
Sun, 04 Mar 2018 16:54:14 -0500 py3: byte-stringify blackbox-readonly-dispatch.py
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 16:54:14 -0500] rev 36739
py3: byte-stringify blackbox-readonly-dispatch.py # skip-blame because just adding some b''
Sun, 04 Mar 2018 16:50:35 -0500 py3: make blackbox-readonly-dispatch.py use ui instead of print()
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 16:50:35 -0500] rev 36738
py3: make blackbox-readonly-dispatch.py use ui instead of print()
Sun, 04 Mar 2018 16:48:17 -0500 py3: fix int formatting of "incoming changes" log
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 16:48:17 -0500] rev 36737
py3: fix int formatting of "incoming changes" log
Sun, 04 Mar 2018 16:48:01 -0500 largefiles: use %d instead of %s to process ints
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:48:01 -0500] rev 36736
largefiles: use %d instead of %s to process ints Differential Revision: https://phab.mercurial-scm.org/D2677
Sun, 04 Mar 2018 16:11:15 -0500 transaction: fix an error string with bytestr() on a repr()d value
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:11:15 -0500] rev 36735
transaction: fix an error string with bytestr() on a repr()d value Fixes test-rollback.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2674
Sun, 04 Mar 2018 16:23:10 -0500 py3: work around comparison between int and None in tagmerge
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 16:23:10 -0500] rev 36734
py3: work around comparison between int and None in tagmerge
Sun, 04 Mar 2018 16:13:46 -0500 py3: do not mutate dict while iterating in tagmerge
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 16:13:46 -0500] rev 36733
py3: do not mutate dict while iterating in tagmerge
Sun, 04 Mar 2018 16:01:18 -0500 py3: fix type of ui.configitems(ignoresub=True) result
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 16:01:18 -0500] rev 36732
py3: fix type of ui.configitems(ignoresub=True) result
Sun, 04 Mar 2018 15:53:10 -0500 py3: don't use str() to stringify pushloc
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 15:53:10 -0500] rev 36731
py3: don't use str() to stringify pushloc
Sun, 04 Mar 2018 15:26:26 -0500 py3: byte-stringify test-config.t and test-config-env.py
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 15:26:26 -0500] rev 36730
py3: byte-stringify test-config.t and test-config-env.py
Sun, 04 Mar 2018 15:24:45 -0500 py3: use startswith() instead of slicing to detect leading whitespace
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Mar 2018 15:24:45 -0500] rev 36729
py3: use startswith() instead of slicing to detect leading whitespace
Sun, 04 Mar 2018 16:06:47 -0500 archival: use py3 friendly replacements for chr() and long()
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:06:47 -0500] rev 36728
archival: use py3 friendly replacements for chr() and long() Differential Revision: https://phab.mercurial-scm.org/D2673
Sun, 04 Mar 2018 16:06:27 -0500 archival: ensure file mode for gzipfile is sysstr
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:06:27 -0500] rev 36727
archival: ensure file mode for gzipfile is sysstr Differential Revision: https://phab.mercurial-scm.org/D2672
Sun, 04 Mar 2018 16:06:10 -0500 archival: fix a missing r'' on a kwargs check
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:06:10 -0500] rev 36726
archival: fix a missing r'' on a kwargs check # skip-blame just an r prefix Differential Revision: https://phab.mercurial-scm.org/D2671
Sun, 04 Mar 2018 16:05:44 -0500 py3: more passing tests (ten this time)
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 16:05:44 -0500] rev 36725
py3: more passing tests (ten this time) Differential Revision: https://phab.mercurial-scm.org/D2670
Sun, 04 Mar 2018 15:55:55 -0500 util: fix unsafe url abort with bytestr() on url
Augie Fackler <augie@google.com> [Sun, 04 Mar 2018 15:55:55 -0500] rev 36724
util: fix unsafe url abort with bytestr() on url Differential Revision: https://phab.mercurial-scm.org/D2669
Sun, 04 Mar 2018 15:16:42 -0500 xdiff: fix builds on Windows
Matt Harbison <matt_harbison@yahoo.com> [Sun, 04 Mar 2018 15:16:42 -0500] rev 36723
xdiff: fix builds on Windows This works on my ancient Fedora system too, without warnings. There are, however, warnings about various 64 to 32 bit conversions on Windows that need to be examined.
(0) -30000 -10000 -3000 -1000 -120 +120 +1000 +3000 +10000 tip