hgweb: use computed base URL from parsed request Let's not reinvent URL construction in a function that runs the templater. Differential Revision: https://phab.mercurial-scm.org/D2735

hgweb: parse WSGI request into a data structure Currently, our WSGI applications (hgweb_mod and hgwebdir_mod) process the raw WSGI request instance themselves. This means they have to talk in terms of system strings. And they need to know details about what's in the WSGI request. And in the case of hgweb_mod, it is doing some very funky things with URL parsing to impact dispatching. The code is difficult to read and maintain. This commit introduces parsing of the WSGI request into a higher-level and easier-to-reason-about data structure. To prove it works, we hook it up to hgweb_mod and use it for populating the relative URL on the request instance. We hold off on using it in more places because the logic in hgweb_mod is crazy and I don't want to involve those changes with review of the parsing code. The URL construction code has variations that use the HTTP: Host header (the canonical WSGI way of reconstructing the URL) and with the use of SERVER_NAME. We need to differentiate because hgweb is currently using SERVER_NAME for URL construction. Differential Revision: https://phab.mercurial-scm.org/D2734

hgweb: always use "?" when writing session vars This code resolves a string to insert in URLs as part of a query string. Essentially, it resolves the {sessionvars} template keyword, which is used by hgweb templates to build a URL as a string. The whole approach here feels wrong because there's no way of knowing when this code runs how the final URL will look. There could be additional URL fragments added before this template keyword that add a query string component. Furthermore, I don't think there's *any* for req.url to have a query string. That's because the code that populates this variable only takes SCRIPT_NAME and REPO_NAME into account. The "?" character it is searching for would only be added if some code attempted to add QUERY_STRING to the URL. Hacking the code up to raise if "?" is present in the URL yields a clean test suite run. I'm not sure if we broke this code or if it has always been broken. Anyway, this commit removes support for emitting "&" as the first character in {sessionvars} and makes it always emit "?", which is what it was always doing before AFAICT. Differential Revision: https://phab.mercurial-scm.org/D2733

hgweb: rename req to wsgireq We will soon introduce a parsed WSGI request object so we don't have to concern ourselves with low-level WSGI matters. Prepare for multiple request objects by renaming the existing one so it is clear it deals with WSGI. We also remove a symbol import to avoid even more naming confusion. # no-check-commit because of some new foo_bar naming that's required Differential Revision: https://phab.mercurial-scm.org/D2732

hgweb: validate WSGI environment dict The wsgiref.validate module contains useful functions for validating that various WSGI data structures are proper. This commit adds validation of the environment dict to our built-in HTTP server, which turns an HTTP request into an environment dict. The check discovered that we weren't always setting QUERY_STRING, which would cause the cgi module to fall back to sys.argv. So we change things to always set QUERY_STRING. The check passes on Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D2731

hgweb: ensure all wsgi environment values are str Previously, we had a few entries that were bytes on Python 3. PEP-0333 states that all entries must be the native str type (bytes on Python 2, str on Python 3). This required a number of changes to hgweb_mod to unbreak things on Python 3. I suspect there still may be some regressions. I'm going to introduce a data structure that represents a parsed WSGI request in upcoming commits. This will hold bytes and will allow us to stop using raw literals throughout the WSGI code. Differential Revision: https://phab.mercurial-scm.org/D2730

wireproto: formalize permissions checking as part of protocol interface Per the inline comment desiring to formalize permissions checking in the protocol interface, we do that. I'm not convinced this is the best way to go about things. I would love for there to e.g. be a better exception for denoting permissions problems. But it does feel strictly better than snipping attributes on the proto instance. Differential Revision: https://phab.mercurial-scm.org/D2719

wireproto: declare permissions requirements in @wireprotocommand (API) With the security patches from 4.5.2 merged into default, we now have a per-command attribute defining what permissions are needed to run that command. We now have a richer @wireprotocommand that can be extended to record additional command metadata. So we port the permissions mechanism to be based on @wireprotocommand. .. api:: hgweb_mod.perms and wireproto.permissions have been removed. Wire protocol commands should declare their required permissions in the @wireprotocommand decorator. Differential Revision: https://phab.mercurial-scm.org/D2718

wireprotoserver: check permissions in main dispatch function The permissions checking code merged from stable is out of place in the refactored hgweb_mod module. This commit moves the main call to wireprotoserver. We still have some lingering code in hgweb_mod. This will get addressed later. Differential Revision: https://phab.mercurial-scm.org/D2717

wireprotoserver: check if command available before calling it The previous behavior was just plain wrong. I have no clue how it landed. My guess is a merge conflict resolution gone wrong on my end a few weeks ago. Differential Revision: https://phab.mercurial-scm.org/D2716

py3: drop encoding.strio() Its buffered nature makes TextIOWrapper unsuitable for temporarily wrapping bytes I/O.

ui: adjust Windows workaround to new _readline() code It's only needed when rawinput() is called. Also made it Py3 compatible.

ui: do not use rawinput() when we have to replace sys.stdin/stdout See the inline comment for why. The current Python3 hack doesn't work if more than one user inputs are expected because TextIOWrapper fills its internal buffer at the first read() request. Maybe we could write an unbuffered TextIOWrapper, but I don't want to make things more complicated. Instead, this patch reinvents raw_input(' ') of no readline support.

ui: do not try readline support if fin/fout aren't standard streams It's unlikely for a non-stdio stream to be a tty. Minimizing readline support makes it much simpler to work around the unicode input() function of Python 3. This also works on chg which duplicates client's tty to stdio fds.

util: add public isstdin/isstdout() functions

ui: add debug commands to test interactive prompt Interactive operations aren't easily covered by tests. So let's add commands to test them manually.

ui: inline util.bytesinput() into ui._readline() Prepares for rework of Python 3 support, which is currently broken due to read-ahead buffer of TextIOWrapper.

hgk: stop using util.bytesinput() to read a single line from stdin Appears that the stdio here is an IPC channel between hg and hgk (tk) processes, which shouldn't need a fancy readline support.

bookmarks: test for exchanging long bookmark names (issue5165) As far as I can tell a test for a long bookmark name never actually got added when this was fixed. Let's document that a 300 byte bookmark is something we're supporting (this patch started out life over a year ago as a way for me to validate the problem, and I recently found it.) I think the nonzero exits from the push operations are only because no new changesets get exchanged. Please correct me if I'm wrong on that. :) Differential Revision: https://phab.mercurial-scm.org/D2727

phabricator: follow-up phab auth improvements with backwards compat mode We'll rip this out before we ship 4.6, but this gives people a window to migrate. Differential Revision: https://phab.mercurial-scm.org/D2703

phabricator: specify API tokens per host, rather than per repo Differential Revision: https://phab.mercurial-scm.org/D1919

py3: drop b'' from generate-working-copy-states.py output I could make everything bytes, but decoding filename seemed easier and we don't have to worry about non-ascii filenames here.

py3: make test-commit-multiple.t byte-safe

py3: fix type of default username

py3: read/write plain lock file in binary mode A lock file shouldn't contain '\n', so this isn't a BC.

util: stop calling os.stat_float_times() It had Python-wide side effects, and it disappears in 3.7.0. As of this change, we're mostly working on 3.7.0b2. There are a few worrying failures, mostly around regular expressions, but we'll have to tackle those separately. Differential Revision: https://phab.mercurial-scm.org/D2697

cleanup: use stat_result[stat.ST_MTIME] instead of stat_result.st_mtime The latter is floating point by default, and we've been doing os.stat_float_times(False). Unfortunately, os.stat_float_times was removed between Python 3.7.0a1 and 3.7.0b2, so we have to stop using it. Differential Revision: https://phab.mercurial-scm.org/D2696

osutil: implement minimal __getitem__ compatibility on our custom listdir type We previously declined to do this, but the removal of the deprecated os.stat_float_times() method in Python 3.7 forces our hand. Differential Revision: https://phab.mercurial-scm.org/D2695

hgweb: adapt to socket._fileobject changes in Python 3 Differential Revision: https://phab.mercurial-scm.org/D2688

debugcommands: fix some %r output with bytestr() wrappers Almost fixes test-merge-tools.t. I think the remaining failure there is due to some overspecified tempfile names. Differential Revision: https://phab.mercurial-scm.org/D2675

tests: add test for issue 5494 but with --collapse This was not fixed, so the test case currently demonstrates the breakage. Differential Revision: https://phab.mercurial-scm.org/D2714

tests: .hg/merge is a directory, so use `test -d` This part of test-rebase-interrupts.t would have passed before the fix in a580b2d65ded (rebase: make sure merge state is cleaned up for no-op rebases (issue5494), 2017-05-18). Differential Revision: https://phab.mercurial-scm.org/D2713

rebase: only store collapse message once The message is determined by the user passing --message or --log when the rebase is started. There's no need to write it to a file for each rebased commit; writing it once at the start of the rebase is enough. Differential Revision: https://phab.mercurial-scm.org/D2712

rebase: collapse two nested if-conditions Also change the order since it feel to me like it's more about --collapse than it is about --keep. Differential Revision: https://phab.mercurial-scm.org/D2711

rebase: reduce scope of "dsguard" variables a bit Differential Revision: https://phab.mercurial-scm.org/D2710

rebase: remove unused argument "state" from rebasenode() Differential Revision: https://phab.mercurial-scm.org/D2709

rebase: delete obsolete internal "keepopen" option The option was apparently introduced for use by the "pbranch" extension, see f2558a8228be (rebase: add option to not commit after a collapsing, 2010-02-07). However, it doesn't seem like it was ever used by that extension (according to `hg grep` in a clone of https://bitbucket.org/parren/hg-pbranch/), so let's delete it. Differential Revision: https://phab.mercurial-scm.org/D2708

releasenotes: allow notes for multiple directives in a single changeset This problem was caught in da91e7309daf8ffc51bf3e6f4b2d8a16ef5af95a. This patch just makes sure there is no warning when we encounter such a case. Differential Revision: https://phab.mercurial-scm.org/D2254

releasenotes: mention changeset with warning and abort Output the changeset hash with the warning/abort message just to know where things messed up. Differential Revision: https://phab.mercurial-scm.org/D2253

releasenotes: replace abort with warning while parsing (issue5775) During the 4.5 development cycle, the extension broke on two different changesets. This fixes the issue by ensuring that it just throws a warning when it encounters unexpected behaviour, instead of aborting. Differential Revision: https://phab.mercurial-scm.org/D2255

archival: fileit should not use atomictemp, causes performance regression Differential Revision: https://phab.mercurial-scm.org/D2704

perf: teach perfbdiff to call blocks() and to use xdiff Differential Revision: https://phab.mercurial-scm.org/D2624

fuzz: fix xdiff build Recent xdiff code cleanups removed some files and changed some structures. Update fuzz code. Differential Revision: https://phab.mercurial-scm.org/D2707

xdiff: remove xmerge related logic hg has its own merge algorithm with flexible config options. Differential Revision: https://phab.mercurial-scm.org/D2706

xdiff: remove xemit related logic xemit handles "diff formatting and output" with options like context lines, whether show function names, etc. That is handled more cleanly at a higher level in hg. Removing context line parameters would also make the trimming logic (D2686) cleaner and more confident. See [1]. [1]: https://github.com/git/git/commit/d2f82950a9226ae1102a7a97f03440a4bf8c6c09 Differential Revision: https://phab.mercurial-scm.org/D2705

xdiff: remove unused structure, functions, and constants `bdiffparam_t` is unused. `xdl_fall_back_diff` is no longer used after D2573. `XDL_MMB_READONLY`, `XDL_MMF_ATOMIC` are unused. `XDL_BDOP*` are unused since there is no xdiff binary diff algorithm. `anchors` feature is not used. It's also relatively new in git. Differential Revision: https://phab.mercurial-scm.org/D2684

xdiff: remove whitespace related feature In Mercurial, whitespace related handling are done at a higher level than the low-level diff algorithm so "ignore spaces". So it's not used by mdiff. Some of the upcoming optimizations would be more difficult with whitespace related features kept. So let's remove them. Differential Revision: https://phab.mercurial-scm.org/D2683

merge with stable There were a handful of merge conflicts in the wire protocol code due to significant refactoring in default. When resolving the conflicts, I tried to produce the minimal number of changes to make the incoming security patches work with the new code. I will send some follow-up commits to get the security patches better integrated into default.

Added signature for changeset 8bba684efde7

Added tag 4.5.2 for changeset 8bba684efde7

merge with security patches

hgweb: always perform permissions checks on protocol commands (BC) (SEC) Previously, the HTTP request handling code would only perform permissions checking on a wire protocol command if that wire protocol command defined its permissions / operation type. This meant that commands (possibly provided by extensions) not defining their operation type would bypass permissions check. This could lead to exfiltration of data from servers and mutating repositories that were supposed to be read-only. This security issue has been present since the permissions table was introduced by d3147b4e3e8a in 2008. This commit changes the behavior of the HTTP server to always perform permissions checking for protocol requests. If an explicit permission for a wire protocol command is not defined, the server assumes the command can be used for writing and governs access accordingly. .. bc:: Wire protocol commands not defining their operation type in ``wireproto.PERMISSIONS`` are now assumed to be used for "push" operations and access control to run those commands is now enforced accordingly.

wireproto: check permissions when executing "batch" command (BC) (SEC) For as long as the "batch" command has existed (introduced by bd88561afb4b and first released as part of Mercurial 1.9), that command (like most wire commands introduced after 2008) lacked an entry in the hgweb permissions table. And since we don't verify permissions if an entry is missing from the permissions table, this meant that executing a command via "batch" would bypass all permissions checks. The security implications are significant: a Mercurial HTTP server would allow writes via "batch" wire protocol commands as long as the HTTP request were processed by Mercurial and the process running the Mercurial HTTP server had write access to the repository. The Mercurial defaults of servers being read-only and the various web.* config options to define access control were bypassed. In addition, "batch" could be used to exfiltrate data from servers that were configured to not allow read access. Both forms of permissions bypass could be mitigated to some extent by using HTTP authentication. This would prevent HTTP requests from hitting Mercurial's server logic. However, any authenticated request would still be able to bypass permissions checks via "batch" commands. The easiest exploit was to send "pushkey" commands via "batch" and modify the state of bookmarks, phases, and obsolescence markers. However, I suspect a well-crafted HTTP request could trick the server into running the "unbundle" wire protocol command, effectively performing a full `hg push` to create new changesets on the remote. This commit plugs this gaping security hole by having the "batch" command perform permissions checking on each sub-command that is being batched. We do this by threading a permissions checking callable all the way to the protocol handler. The threading is a bit hacky from a code perspective. But it preserves API compatibility, which is the proper thing to do on the stable branch. One of the subtle things we do is assume that a command with an undefined permission is a "push" command. This is the safest thing to do from a security perspective: we don't want to take chances that a command could perform a write even though the server is configured to not allow writes. As the test changes demonstrate, it is no longer possible to bypass permissions via the "batch" wire protocol command. .. bc:: The "batch" wire protocol command now enforces permissions of each invoked sub-command. Wire protocol commands must define their operation type or the "batch" command will assume they can write data and will prevent their execution on HTTP servers unless the HTTP request method is POST, the server is configured to allow pushes, and the (possibly authenticated) HTTP user is authorized to perform a push.

wireproto: declare operation type for most commands (BC) (SEC) The permissions model of hgweb relies on a dictionary to declare the operation associated with each command - either "pull" or "push." This dictionary was established by d3147b4e3e8a in 2008. Unfortunately, we neglected to update this dictionary as new wire protocol commands were introduced. This commit defines the operations of most wire protocol commands in the permissions dictionary. The "batch" command is omitted because it is special and requires a more complex solution. Since permissions checking is skipped unless a command has an entry in this dictionary (this security issue will be addressed in a subsequent commit), the practical effect of this change is that various wire protocol commands now HTTP 401 if web.deny_read or web.allow-pull, etc are set to deny access. This is reflected by test changes. Note how various `hg pull` and `hg push` operations now fail before discovery. (They fail during the initial "capabilities" request.) This change fixes a security issue where built-in wire protocol commands would return repository data even if the web config were configured to deny access to that data. I'm on the fence as to whether we should HTTP 401 the capabilities request. On one hand, it can expose repository metadata and can tell callers things like what version of Mercurial the server is running. On the other hand, a client may need to know the capabilities in order to authenticate in a follow-up request. It appears that Mercurial clients handle the HTTP 401 on *any* protocol request, so we should be OK sending a 401 for "capabilities." But if this causes problems, it should be possible to allow "capabilities" to always work. .. bc:: Various read-only wire protocol commands now return HTTP 401 Unauthorized if the hgweb configuration denies read/pull access to the repository. Previously, various wire protocol commands would still work and return data if read access was disabled.

wireproto: move command permissions dict out of hgweb_mod The operation type associated with wire protocol commands is supposed to be defined in a dictionary so it can be used for permissions checking. Since this metadata is closely associated with wire protocol commands themselves, it makes sense to define it in the same module where wire protocol commands are defined. This commit moves hgweb_mod.perms to wireproto.PERMISSIONS and updates most references in the code to use the new home. The old symbol remains an alias for the new symbol. Tests pass with the code pointing at the old symbol. So this should be API compatible for extensions. As part of the code move, we split up the assignment to the dict so it is next to the @wireprotocommand. This reinforces that a @wireprotocommand should have an entry in this dict. In the future, we'll want to declare permissions as part of the @wireprotocommand decorator. But this isn't appropriate for the stable branch.

tests: comprehensively test HTTP server permissions checking We didn't have test coverage for numerous web.* config options. We add that test coverage. Included in the tests are tests for custom commands. We have commands that are supposedly read-only and perform writes and a variation of each that does and does not define its operation type in hgweb_mod.perms. The tests reveal a handful of security bugs related to permissions checking. Subsequent commits will address these security bugs.

tests: extract HTTP permissions tests to own test file We're about to implement a lot more coverage of the permissions mechanism. In preparation for that, establish a new test file to hold permissions checks. As part of this, we inline the important parts of the "req" helper function.

Added signature for changeset 369aadf7a326

Added tag 4.5.1 for changeset 369aadf7a326

revlog: resolve lfs rawtext to vanilla rawtext before applying delta This happens when a LFS delta base gets a non-LFS delta from another client. In that case, the LFS delta base needs to be converted to non-LFS version before applying the delta. Differential Revision: https://phab.mercurial-scm.org/D2069