support for request targets that can't be specified as URLs

[njsmith]

There's a weird special-case in HTTP: the OPTIONS method (only!) can be used with a request-target of * to request general info about the server, rather than any particular file on the server (details). What makes this weird is that there's no way to specify this target as a URL: if you do hip.request("OPTIONS", "https://github.com"), that should generate a request-target of /. OPTIONS * HTTP/1.1 and OPTIONS / HTTP/1.1 are both meaningful things, and they're different from each other, so we need some other way to get a * target.

There's another weird special case: the CONNECT method (only!) doesn't take a destination URL, but host + port.

These aren't necessary for an MVP, but we should support them eventually. What are the API implications?

One approach that occurred to me was to offer a low-level API that lets you directly specify the HTTP "request-target" (the thing in-between the method and HTTP/1.1). Normally this is determined by a combination of the target URL + proxy setup, but conceivably people might need to do some wacky stuff to interoperate with custom almost-HTTP servers. But, I don't think this is the best approach here, because it's not actually sufficient for either the OPTIONS * or CONNECT cases.

For OPTIONS *, you normally set the request target to *... but if you're going through a proxy, then you instead use a URL without a trailing slash, like OPTIONS https://github.com HTTP/1.1. (This also means that if the user does hip.request("OPTIONS", "https://github.com"), and is using a proxy, then we need to convert that into OPTIONS https://github.com/ HTTP/1.1, i.e. make sure to add the trailing slash). So this will need special case handling; we can't just cover it with a generic "set the request target" thing.

And for CONNECT you definitely need special-case handling, because instead of a regular response body you get an arbitrary bidirectional byte stream (at least if the request succeeds).

So, maybe we want the core public API to be something like:

# hip.get, hip.post, hip.options, etc. are trivial wrappers around this:
hip.request(method, url, ...)
# These are special and can't be implemented in terms of hip.request
hip.options_asterisk(host, ...)
hip.connect(target, ...)

(Of course internally these would probably all immediately call into the same _real_request interface. But the point is that an interface that can handle all these weird cases at once is maybe too weird and error-prone to make public.)

A few more notes that aren't exactly on topic for this issue, but are kind of adjacent and worth writing down:

• In Go apparently they sometimes use a form like CONNECT /some/path HTTP/1.1, as part of an ad hoc RPC system: https://golang.org/src/net/http/request.go#L1035. Not sure we would ever care about this, and it's semi-deprecated anyway, but hey, if we take a target string like hostname:port, then we can automatically support this too? Or maybe it's better to ignore this weird edge case and make the signature hip.connect(host, port, ...)

• The HTTP/1.1 Upgrade header is another mechanism where you do an HTTP request and then it turns into a bidirectional bytestream. But it's a bit different from CONNECT, because it takes a proper URL. HOWEVER, there is one complicated case where they interact: HTTP/2 doesn't have an Upgrade header; instead, you do a special variant of CONNECT that does take a full URL. If we want to support this (and we probably do eventually, for e.g. websockets), then we can't just expose Upgrade: and RFC-8441-extended-CONNECT, because you have to pick between them based on which protocol you're using, and we don't know that until we've already made the connection to the host. So we probably also need a top-level hip.upgrade(...) function that exposes the common subset of Upgrade: and RFC-8441-extended-CONNECT, and internally it picks between them. This would be another top-level function that can't be implemented in terms of hip.request.