@nsaunders I didn't realize you had requested reviews from others on this yet, so I thought it had gone unnoticed. Thus, I also didn't realize that the CI build failed to pass. Sorry for not updating this PR to fix the build.

This build is failing because there hasn't been a new package set release that includes the updated indexed-monads release that includes the file for do notation. The next release will include that package.

Would it be possible for you to split out the formatting/Ix.do changes into a separate PR? It's a big chunk of the PR that seems unrelated to the kinds change. Would make it more approachable for reviewing. Thanks!

I ported the do notation change commits to the other PR. Is that all? Or were you referring to other formatting commits, too?

@owickstrom I'm guessing you've been unable to get to this PR. So, I'm following up again. Were there any other formatting changes I needed to do?

In this process, I also had to create a WriterResponse newtype to implement getWriter.

On reflection, I'm pretty sure this could change is not needed if the reqState and resState phantom types were only used in Conn instead of the Request type or Response type.

Sorry for the unresponsiveness. I'll try to get to it this weekend.

As I was using this in my own project, I came across an annoyance... A fully fleshed out Middleware type...

Middleware 
  m 
  (Conn req reqStateFrom res resStateFrom compFrom)
  (Conn req reqStateTo res resStateTo compTo)
  a

...can have 8 different transitions

• request state change
• response state change
• component type change

In these recent commits, I moved all of these possible transitions into the Hyper.Conn module. Most of the time, we're only focusing on the request or response state transitions. However, there are times when we want to modify the component type. Thus, I named the below transitions to emphasize the request/response state transitions. If the component was changed, the name includes a ' at the end. If not, it does not appear.

After making that change, I updated the rest of the files in the src, test, and examples directories.

One issue with this PR as it stands is that one could theoretically transition from one state to an invalid one if one defined an invalid type signature. For example

foo :: ResponseTransition m req BodyRead BodyUnread res resState comp a
foo = -- implementation

I'm not sure what the best way to handle this is because we'd need to use type-level functions to validate a transition. For example,

import Prim.TypeError (class Fail, Text)

class ValidRequestTransition (from :: RequestState) (to :: RequestState)
instance validRequestTransitionUnreadToRead 
  :: ValidRequestTransition BodyUnread BodyRead
else
instance invalidTransition 
  :: (Fail (Text "Invalid request state transition")) 
  => ValidRequestTransition from to

class ValidResponseTransition (from :: ResponseState) (to :: ResponseState)
instance      validResponseTransitionStatesLineToHeadersOpen    
  :: ValidResponseTransition StatusLineOpen HeadersOpen
else instance validResponseTransitionStatesLineToBodyOpen       
  :: ValidResponseTransition StatusLineOpen BodyOpen
else instance validResponseTransitionStatesLineToResponseEnded  
  :: ValidResponseTransition StatusLineOpen ResponseEnded
else instance validResponseTransitionHeadersOpenToHeadersOpen   
  :: ValidResponseTransition HeadersOpen HeadersOpen
else instance validResponseTransitionHeadersOpenToBodyOpen      
  :: ValidResponseTransition HeadersOpen BodyOpen
else instance validResponseTransitionHeadersOpenToResponseEnded 
  :: ValidResponseTransition HeadersOpen ResponseEnded
else instance validResponseTransitionBodyOpenToBodyOpen         
  :: ValidResponseTransition BodyOpen BodyOpen
else instance validResponseTransitionBodyOpenToResponseEnded    
  :: ValidResponseTransition BodyOpen ResponseEnded
else instance invalidResponseTransition 
  :: (Fail (Text "Invalid response state transition")) 
  => ValidResponseTransition from to

-- | Alias for easily defining both the request and response state transitions,
-- | including a change in the component type.
type ConnTransition'
      m
      (request :: RequestState -> Type)
      (fromRequestState :: RequestState)
      (toRequestState :: RequestState)
      (response :: ResponseState -> Type)
      (fromResponseState :: ResponseState)
      (toResponseState :: ResponseState)
      fromComp
      toComp
      a
  = ValidRequestTransition fromRequestState toRequestState =>
    ValidResponseTransition fromResponseState toResponseState =>
    Middleware
      m
      (Conn request fromRequestState response fromResponseState fromComp)
      (Conn request toRequestState   response toResponseState   toComp)
      a

Since empty class dictionaries are removed in the latest PS release, this shouldn't incur a performance penalty during runtime.

However, since one can define a Middleware outside of the transition type aliases, one could still create a Middleware that permits an invalid state transition.

Sorry, my time is very limited at the moment to work on this stuff, and it was some time ago so I'm a bit disoriented in the codebase. Pardon my slow response.

But from what I can gather, I find this to be a very large change, touching the API a lot and adding a bunch of type parameters and aliases. I think the gained type safety and constraint (with the custom kinds) isn't really worth it, frankly. The old type signatures were simpler and more readable to me. But, importantly, people using this API should weigh in. I'm very much out of the loop on this.

Also, I realize this PR might be partially based on my old comment, so apologies if it feels like misleading. But in hindsight, maybe that level of type encoding isn't worth. I dunno. Does this make sense?

Thanks for spending time looking at this.

I don't blame you regarding the 'whether this level of type encoding is worth it or not' comment. I've been partially wondering that myself as I've dealt with compiler errors and whatnot. Moreover, I wasn't able to figure out how to ensure that a request/response state transition is valid. I'm not sure whether that can be done by indexed monads, at least how they are defined right now, due to what I stated in my above comment.

Still, you use phantom types for the response type, but don't seem to want to use it for the request type. If so, then why use phantom types for the response type at all? Wouldn't it be better to use MonadState for the response type instead, or the same idea but as a different type class name so that a user could use their own MonadState on top of that?
I'd guess the response's phantom types are only necessary due to response streaming.

Still, you use phantom types for the response type, but don't seem to want to use it for the request type. If so, then why use phantom types for the response type at all? Wouldn't it be better to use MonadState for the response type instead, or the same idea but as a different type class name so that a user could use their own MonadState on top of that?
I'd guess the response's phantom types are only necessary due to response streaming.

Fair point. I there might be two separate things going on:

1. Using phantom types for the request
2. Parametizing Conn with both the type constructors for request and response, and the custom-kinded states

I'm unsure wether (1) is worth it, due to the user mostly dealing with getting the response right (the web server does most of the request processing). The exception being streaming the request body, which could be the thing tipping the scales towards using phantom types (or something else.)

Regarding (2), that's what I referred to specifically when talking about the custom kinds and API changes. Sure, you can't then construct a Conn parameterized with any types, e.g. Conn String Int, but I don't think it buys much safety in practice, and that it adds a lot more verbosity. Just scanning through the PR, I found the type signatures less intuitive with the parameters and various type aliases.

Just to be clear, I'm not saying this design is better or worse than the existing API; it just makes different tradeoffs. Hyper isn't exactly used heavily (I don't know of any "production" apps using it) so the API mustn't be super stable in that regard, but I think this kind of change isnt't worth the API compatibility problems.

Thanks for taking the time to work on this!

I'm unsure wether (1) is worth it, due to the user mostly dealing with getting the response right (the web server does most of the request processing). The exception being streaming the request body, which could be the thing tipping the scales towards using phantom types (or something else.)

Yeah, that makes sense. It seems that this (request body streaming) could either be supported out-of-box (use request kinds), or an end-user could opt-in to such a feature if they wanted to support that on their server (define a different Conn type that supports this and allow end-user to reimplement the parts of this library they want).

Regarding (2), that's what I referred to specifically when talking about the custom kinds and API changes. Sure, you can't then construct a Conn parameterized with any types, e.g. Conn String Int, but I don't think it buys much safety in practice, and that it adds a lot more verbosity. Just scanning through the PR, I found the type signatures less intuitive with the parameters and various type aliases.

I thought the change actually made things clearer. When I was first looking at this repo and looked at the type signature for things, it didn't immediately stand out to me that res had kind ResponseState -> Type. I had to get a few compiler errors, look into it more, and then say, "Oh! That's what is meant. Well, why didn't they just use a kind for that to make it more obvious?"

Still, I see the tradeoff of the verbosity. While the kind annotation in functions could be dropped (e.g. forall (resState :: ResponseState) ... -> forall resState) since that can be specified in the Conn type to reduce verbosity, it would still be verbose and wouldn't fix the 'new learner' problem I experienced above. In some ways, that might be better handled in documentation.

Just to be clear, I'm not saying this design is better or worse than the existing API; it just makes different tradeoffs. Hyper isn't exactly used heavily (I don't know of any "production" apps using it) so the API mustn't be super stable in that regard, but I think this kind of change isnt't worth the API compatibility problems.

Yup! Looks like the tradeoff is more verbosity for higher type-safety. You do say this library is an experimental approach, so that might also be why it's not used in production AFAWK. And yeah, this would be a breaking change.

I've spent the last few weeks looking at the different approaches one can use in PureScript for a web server. httpure uses the function composition approach (Request -> Aff Response) whereas this one used the state monad approach StateT Conn m a. One could add phantom types to the function approach to increase type safety, but I'm not sure whether that would be worth it. Adding phantom types to state monad approach could be done using indexed monads (what you're currently trying) but it leads to this verbosity when taken to its logical conclusions (and even then a user can define an invalid state transition).
Another way to define it is using a modified variant of the state monad State i o m a | m i -> o so that you can use phantom types to model state transitions. Still, this doesn't fix the verbosity problem.
On the other hand, dropping all of this type safety reduces verbosity at the cost of possible runtime errors or invalid program states.

Personally, I think an even simpler implementation for a web server should be built. Then, one can choose their design tradeoffs by choosing frameworks that build upon that base implementation (e.g. choose the approach of httpure or hyper or some variant of the two for modeling the server).