hgweb: move readallowed to a standalone function hgwebdir s kind of large. Let's make the class smaller by moving things that don't need to be there. Differential Revision: https://phab.mercurial-scm.org/D2812

hgweb: remove some use of wsgireq in hgwebdir While we're here, rename a method so abide by our style policy, since otherwise check-commit would complain. Differential Revision: https://phab.mercurial-scm.org/D2805

hgweb: fix a bug due to variable name typo It looks like the "sort" query string parameter was not being honored properly. Differential Revision: https://phab.mercurial-scm.org/D2804

hgweb: stop passing req and tmpl into @webcommand functions (API) We have effectively removed all consumers of the old wsgirequest type. The templater can be accessed on the requestcontext passed into the @webcommand function. For the most part, these arguments are unused. They only exist to provide backwards compatibility. And in the case of wsgirequest, use of that object could actively interfere with the new request object. So let's stop passing these objects to @webcommand functions. With this commit, wsgirequest is practically dead from the hgweb WSGI application. There are still some uses in hgwebdir though... .. api:: @webcommand functions now only receive a single argument. The request and templater instances can be accessed via the ``req`` and ``templater`` attributes of the first argument. Note that the request object is different from previous Mercurial releases and consumers of the previous ``req`` 2nd argument will need updating to use the new API. Differential Revision: https://phab.mercurial-scm.org/D2803

hgweb: pass modern request type into various webutil functions (API) Our march towards killing wsgirequest continues. .. api:: Various functions in hgweb.webutil now take a modern request object instead of ``wsgirequest``. Differential Revision: https://phab.mercurial-scm.org/D2802

hgweb: don't redundantly pass templater with requestcontext (API) The requestcontenxt has a ``tmpl`` attribute to access the templater. We don't need to pass the templater explicitly when passing a requestcontext instance. .. api:: Various helper functions in hgweb.webutil no longer accept a templater instance. Access the templater through the ``web`` argument instead. Differential Revision: https://phab.mercurial-scm.org/D2801

hgweb: use templater on requestcontext instance After this commit, all @webcommand function no longer use their "tmpl" argument. Instead, they use the templater attached to the requestcontext. This is the same exact object. So there should be no difference in behavior. Differential Revision: https://phab.mercurial-scm.org/D2800

hgweb: add a sendtemplate() helper function This pattern is common. Let's make a helper function to reduce boilerplate. We store the "global" template on the requestcontext instance and use it. The templater used by the helper function is the same templater that's passed in as an argument to the @webcommand functions. It needs to be this way because various commands are accessing and mutating the defaults on the templater instance. Differential Revision: https://phab.mercurial-scm.org/D2799

hgweb: use web.req instead of req.req We now have access to the modern request type on the requestcontext instance. Let's access it from there. While we're here, remove an unused argument from _search(). Differential Revision: https://phab.mercurial-scm.org/D2798

hgweb: stop setting headers on wsgirequest All commands now go through the new response API. This is dead code. Differential Revision: https://phab.mercurial-scm.org/D2797

hgweb: always return iterable from @webcommand functions (API) We had to hack up this function to support our transition to the new response API. Now that we're done with the transition (!!), we can return to returning an iterator of content chunks from these functions. It is tempting to return a normal object and not a generator. However, as the keyword extension demonstrates, extensions may wish to wrap commands and have a try..finally block around execution. Since there is a generator producing content and that generator could be executing code, the try..finally needs to live for as long as the generator is running. That means we have to return a generator so wrappers can consume the generator inside a try..finally. .. api:: hgweb @webcommand functions must use the new response object passed in via ``web.res`` to initiate sending of a response. The hgweb WSGI application will no longer start sending the response automatically. Differential Revision: https://phab.mercurial-scm.org/D2796

hgweb: send errors using new response API Our slow march off of wsgirequest continues. Differential Revision: https://phab.mercurial-scm.org/D2795

hgweb: refactor 304 handling code We had generic code in wsgirequest for handling HTTP 304 responses. We also had a special case for it in the catch all exception handler in the WSGI application. We only ever raise 304 in one place. So, we don't need to treat it specially in the catch all exception handler. But it is useful to validate behavior of 304 responses. We port the code that sends a 304 to use the new response API. We then move the code for screening 304 sanity into the new response API. As part of doing so, we discovered that we would send Content-Length: 0. This is not allowed. So, we fix our response code to not emit that header for empty response bodies. Differential Revision: https://phab.mercurial-scm.org/D2794

hgweb: transition permissions hooks to modern request type (API) We're trying to remove ``wsgirequest``. The permissions hooks don't do anything they can't do with our new request type. So let's pass that in. This was the last use of ``wsgirequest`` in the wire protocol code! .. api:: hgweb.hgweb_mod.permhooks no longer take a ``wsgirequest`` instance as an argument. Differential Revision: https://phab.mercurial-scm.org/D2793

hgweb: port archive command to modern response API Well, I tried to go with PEP 3333's recommendations and only allow our WSGI application to emit data via a response generator. Unfortunately, the "archive" command calls into the zipfile and tarfile modules and these operator on file objects and must send their data to an object with write(). There's no easy way turn these write() calls into a generator. So, we teach our response type how to expose a file object like object that can be used to write() output. We try to keep the API consistent with how things work currently: callers must call a setbody*(), then sendresponse() to trigger sending of headers, and only then can they get a handle on the object to perform writing. This required overloading the return value of @webcommand functions even more. Fortunately, we're almost completely ported off the legacy API. So we should be able to simplify matters in the near future. A test relying on this functionality has also been updated to use the new API. Differential Revision: https://phab.mercurial-scm.org/D2792

hgweb: refactor fake file object proxy for archiving Python's zip file writer operates on a file object. When doing work, it periodically calls write(), flush(), and tell() on that object. In WSGI contexts, the start_response function returns a write() function. That's a function to write data, not a full file object. So, when the archival code was first introduced in 2b03c6733efa in 2006, someone invented a proxy "tellable" type that wrapped a file object like object and kept track of write count so it could implement tell() and satisfy zipfile's needs. When our archival code runs, it attempts to tell() the destination and if that fails, converts it to a "tellable" instance. Our WSGI application passes the "wsgirequest" instance to the archival function. It fails the tell() test and is converted to a "tellable." It's worth noting that "wsgirequest" implements flush(), so "tellable" doesn't. This hackery all seems very specific to the WSGI code. So this commit moves the "tellable" type and the conversion of the destination file object into the WSGI code. There's a chance some other caller may be passing a file object like object that doesn't implement tell(). But I doubt it. As part of the refactor, our new type implements flush() and doesn't implement __getattr__. Given the intended limited use of this type, I want things to fail fast if there is an attempt to access attributes because I think it is important to document which attributes are being used for what purposes. Differential Revision: https://phab.mercurial-scm.org/D2791

tests: additional test coverage of archive web command This command is special in a few ways. First, it is the only command using the write() function from WSGI's start_response() function. Second, it is setting a custom content-disposition header. We change the test so it prints out full details of the HTTP response. We also save the response body to a file so we can verify its size and hash. The hash check will help ensure that archive generation is deterministic. Differential Revision: https://phab.mercurial-scm.org/D2790

hgweb: port static file handling to new response API hgwebdir_mod hasn't received as much porting effort. So we had to do some minor plumbing to get it to match hgweb_mod and to support the new response object. Differential Revision: https://phab.mercurial-scm.org/D2789

hgweb: remove one-off routing for file?style=raw Now that both functions are using the same API, we can unify how the command is called and perform command-specific behavior in the command itself instead of in the high-level router. Differential Revision: https://phab.mercurial-scm.org/D2788

hgweb: port most @webcommand to use modern response type This only focused on porting the return value. raw file requests are wonky because they go through a separate code path at the dispatch layer. Now that everyone is using the same API, we could clean this up. It's worth noting that wsgirequest.respond() allows sending the Content-Disposition header, but the only user of that feature was removed as part of this change (with the setting of the header now being performed inline). A few @webcommand are not as straightforward as the others and they have not been ported yet. Differential Revision: https://phab.mercurial-scm.org/D2787

hgweb: support using new response object for web commands We have a "requestcontext" type for holding state for the current request. Why we pass in the wsgirequest and templater instance to @webcommand functions, I don't know. I like the idea of standardizing on using "requestcontext" for passing all state to @webcommand functions because that scales well without API changes every time you want to pass a new piece of data. So, we add our new request and response instances to "requestcontext" so @webcommand functions can access them. We also teach our command dispatcher to recognize a new calling convention. Instead of returning content from the @webcommand function, we return our response object. This signals that this response object is to be used for sending output. The keyword extension was wrapping various @webcommand and assuming the output was iterable, so we had to teach it about the new calling convention. To prove everything works, we convert the "filelog" @webcommand to use the new convention. The new calling convention is a bit wonky. I intend to improve this once all commands are ported to use the new response object. Differential Revision: https://phab.mercurial-scm.org/D2786

hgweb: inline caching() and port to modern mechanisms We only had one consumer of this simple function. While it could be a generic function, let's not over abstract the code. As part of inlining, we port it off wsgirequest, fix some Python 3 issues, and set a response header on our new response object so it is ready once we start using it to send responses. Differential Revision: https://phab.mercurial-scm.org/D2785

hgweb: expose repo name on parsedrequest I'm not a fan of doing this because I want to find a better solution to the REPO_NAME hack. But this change gets us a few steps closer to eliminating use of wsgirequest. We can worry about fixing REPO_NAME once wsgirequest is gone. Differential Revision: https://phab.mercurial-scm.org/D2784

hgweb: expose URL scheme and REMOTE_* attributes These are consulted by the HTTP wire protocol handler by reading from the env dict. Let's expose them as attributes instead. With the wire protocol handler updates to use the new attributes, we no longer have any consumers of the legacy wsgirequest type in the wire protocol code (outside of a proxied call to the permissions checker). So, we remove most references to it. Differential Revision: https://phab.mercurial-scm.org/D2783

hgweb: remove wsgirequest.form (API) Now that everything is ported to consume from parsedrequest.qsparams, we no longer have a need for wsgirequest.form. Let's remove all references to it. .. api:: The WSGI request object no longer exposes a ``form`` attribute containing parsed query string data. Use the ``qsparams`` attribute instead. Differential Revision: https://phab.mercurial-scm.org/D2782

hgweb: perform all parameter lookup via qsparams I think I managed to update all call sites using wsgirequest.form to use parsedrequest.qsparams. Since behavior of qsparams is to retrieve last value, behavior will change if a parameter was specified multiple times. But I think this is acceptable. I'm not a fan of the `req.req.qsparams` pattern. And some of the modified code could be written better. But I was aiming for a straight port with this change. Cleanup can come later. Differential Revision: https://phab.mercurial-scm.org/D2781

hgweb: set variables in qsparams We currently mutate wsgireq.form in a few places. Since it is independent from req.qsparams, we will need to make changes on req.qsparams as well before consumers can use qsparams. So let's do that. Eventually, we'll delete wsgireq.form and all references to it. Differential Revision: https://phab.mercurial-scm.org/D2780

hgweb: use our new request object for "style" parameter The "style" parameter is kind of wonky because it is explicitly set and has lookups in random locations. Let's port it to qsparams first because it isn't straightforward. There is subtle change in behavior. But I don't think it is worth calling out in a BC. Our multidict's __getitem__ returns the last set value for a key, not the first. So if the query string set a variable multiple times, before we would get the first value and now we would get the last value. It makes no sense to specify these things multiple times. And I think last write wins is more sensible than first write wins. Differential Revision: https://phab.mercurial-scm.org/D2779

hgweb: use a multidict for holding query string parameters My intention with refactoring the WSGI code was to make it easier to read. I initially wanted to vendor and use WebOb, because it seems to be a pretty reasonable abstraction layer for WSGI. However, it isn't using relative imports and I didn't want to deal with the hassle of patching it. But that doesn't mean we can't use good ideas from WebOb. WebOb has a "multidict" data structure for holding parsed query string and POST form data. It quacks like a dict but allows you to store multiple values for each key. It offers mechanisms to return just one value, all values, or return 1 value asserting that only 1 value is set. I quite like its API. This commit implements a read-only "multidict" in the spirit of WebOb's multidict. We replace the query string attributes of our parsed request with an instance of it. Differential Revision: https://phab.mercurial-scm.org/D2776

hgweb: create dedicated type for WSGI responses We have refactored the request side of WSGI processing into a dedicated type. Now let's do the same thing for the response side. We invent a ``wsgiresponse`` type. It takes an instance of a request (for consulation) and the WSGI application's "start_response" handler. The type basically allows setting the HTTP status line, response headers, and the response body. The WSGI application calls sendresponse() to start sending output. Output is emitted as a generator to be fed through the WSGI application. According to PEP 3333, this is the preferred way for output to be transmitted. (Our legacy ``wsgirequest`` exposed a write() to send data. We do not wish to support this API because it isn't recommended by PEP 3333.) The wire protocol code has been ported to use the new API. Differential Revision: https://phab.mercurial-scm.org/D2775