Refactor invoker to be an "object" and not an actor anymore

[slinkydeveloper]

The invoker was meant to be an actor in order to share it among partition processors on the same node. This is not the case anymore, and refactoring the invoker to simply be an object means we can simplify a whole lot of code (see #2335 for a concrete situation where the actor approach is a problem)

[AhmedSoliman]

For clarity, we have not made the decision on whether invoker(s) will continue to be one per partition or not in the future. The decision to split them stemmed as a quick fix for tokio stall issues. There is a few down sides to our current approach that I'd like for us to consider if we'll continue with this design, namely resource budgeting across partitions, and throttling connections/requests to user deployments.

[slinkydeveloper]

What we call the invoker right now is three components effectively:

• Service client, the actual HTTP/lambda client opening connections, doing all the pooling, etc.
• State machine, per invocation, dealing with retries and managing the invocation task
• Invocation task, per invocation attempt, using the Service client to open the connection, interacting with the bidi-stream and parsing service protocol effectively.

There is a few down sides to our current approach that I'd like for us to consider if we'll continue with this design, namely resource budgeting across partitions, and throttling connections/requests to user deployments.

I wonder if down the road this is something that fits in the overall "invoker" or are just concerns we push inside the service client?

[tillrohrmann]

One other resource consumer (in terms of memory) is the SegmentQueue which we instantiate once per Invoker. A single segment queue can allocate up to ~321 MB of memory with the default settings. See #2402 for more details.