-
Notifications
You must be signed in to change notification settings - Fork 40.7k
[WIP] fix: watch client errors #131339
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
[WIP] fix: watch client errors #131339
Conversation
Please note that we're already in Test Freeze for the Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Thu Apr 17 01:31:59 UTC 2025. |
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
5dc32b5
to
7e40f69
Compare
7e40f69
to
bbc5503
Compare
bbc5503
to
9e175a5
Compare
9e175a5
to
501cb1f
Compare
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: karlkfi The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
501cb1f
to
ee2d421
Compare
This PR seemed a bit large, so I've pulled out the test changes without the fix, to prove the bugs exist: |
ee2d421
to
e145080
Compare
- Add channel test helpers to k8s.io/client-go/util/testing - Use the new channel test helpers in the client and server watch tests to validate that the result channels closes without error when the client stops the watcher. - Use the new channel test helpers in the client watcher decoder tests to validate that encoded watch events can be decoded and that the decoder errors with EOF when stopped asynchronously. These new tests uncovered existing errors (added TODOs): 1. The watch client doesn't close the response body when it encounters a NegotiateError. 2. The watch server has a race condition that sometimes sends a watch error on the result channel after the client watcher has been stopped.
- Fix a bug in client-go Watch and WatchList that was keeping the response body open when a NegotiateError was encountered by Request.newStreamWatcher. This was causing the server to keep the storage watcher and timeout channel open until server-side timeout. - Fix a bug in client-go Watch and WatchList that was sometimes sending a "http: read on closed response body" error from the decoder to the result channel after the client watcher had been closed, which closes the http response body. The watcher is suppossed to be closed by the client when done reading from the result channel, so the impact was minimal, but this helps avoid needing to drain the result channel before closing it.
e145080
to
e3cb600
Compare
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What type of PR is this?
/kind bug
/kind cleanup
What this PR does / why we need it
response body open when a NegotiateError was encountered by
Request.newStreamWatcher. This was causing the server to keep the
storage watcher and timeout channel open until server-side timeout.
a "http: read on closed response body" error from the decoder to the
result channel after the client watcher had been closed, which closes
the http response body. The watcher is suppossed to be closed by the
client when done reading from the result channel, so the impact was
minimal, but this helps
Note: This change makes the watch client code a bit harder to read, but it seemed easier to review this way. I can follow up with a refactor to extract a function or two later.
Does this PR introduce a user-facing change?
Dependencies