Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 GOAWAY frames not handled correctly with http2 #1273

Open
zemaj opened this issue Jun 13, 2024 · 0 comments · May be fixed by #1274
Open

🐛 GOAWAY frames not handled correctly with http2 #1273

zemaj opened this issue Jun 13, 2024 · 0 comments · May be fixed by #1274
Labels
Priority: Normal Minor issue impacting one or more users Type: Bug Something isn't working

Comments

@zemaj
Copy link

zemaj commented Jun 13, 2024

Describe the bug
When connecting to nginx with https using http2 the following error appears in the logs intermittently and results in client facing 502 errors;

Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

To Reproduce
Steps to reproduce the behavior:

  1. Install nginx container (such as nginx:alpine) and cloudflared:latest
  2. Use an nginx config file with http2 on (Using a short timeout or request limit triggers GOAWAY frames faster);
server {
    listen      443 ssl default_server;
    http2       on;
    keepalive_timeout           1s;
    keepalive_requests         1;
}
  1. Use a cloudflared config file with http2 on;
    ingress:
      - service: https://nginx:443
        originRequest:
          http2Origin: true
  1. Start sending requests through cloudflared and view logs. You will see the error;
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
  1. Setting http2Origin: false resolves the issue.

Expected behavior
Cloudflared should see the GOAWAY frame and retry the request.

Environment and versions

  • Cloudflared Version: 2024.6.0

Additional context
Appears to be an issue with http.Request missing a no-op GetBody() method.

The GOAWAY frame (type=0x7) is used to initiate graceful shutdown of a connection by a server. Nginx sends GOAWAY frames when a keepalive connection needs to end (e.g. has hit keepalive_timeout or keepalive_requests limits). This happens regularly with default nginx http2 configs.

@zemaj zemaj added Priority: Normal Minor issue impacting one or more users Type: Bug Something isn't working labels Jun 13, 2024
zemaj added a commit to zemaj/cloudflared that referenced this issue Jun 13, 2024
zemaj added a commit to zemaj/cloudflared that referenced this issue Jun 13, 2024
@zemaj zemaj linked a pull request Jun 13, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: Normal Minor issue impacting one or more users Type: Bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant