Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

[WIP] fix: watch client errors #131339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
Loading
from

Conversation

karlkfi
Copy link
Contributor

@karlkfi karlkfi commented Apr 17, 2025

What type of PR is this?

/kind bug
/kind cleanup

What this PR does / why we need it

  • Fix a bug in client-go Watch and WatchList that was keeping the
    response body open when a NegotiateError was encountered by
    Request.newStreamWatcher. This was causing the server to keep the
    storage watcher and timeout channel open until server-side timeout.
  • Fix a bug in client-go Watch and WatchList that was sometimes sending
    a "http: read on closed response body" error from the decoder to the
    result channel after the client watcher had been closed, which closes
    the http response body. The watcher is suppossed to be closed by the
    client when done reading from the result channel, so the impact was
    minimal, but this helps

Note: This change makes the watch client code a bit harder to read, but it seemed easier to review this way. I can follow up with a refactor to extract a function or two later.

Does this PR introduce a user-facing change?

Fixed a bug in the watch client that was failing to close the response body after media type negotiation failure.
Fixed a bug in the watch client that was sometimes sending an error to the result channel after the client watcher had been closed.

Dependencies

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 17, 2025
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.33 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.33.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Thu Apr 17 01:31:59 UTC 2025.

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 17, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Apr 17, 2025
@karlkfi karlkfi changed the title Fix watch client to close response body on error [WIP] Fix watch client to close response body on error Apr 17, 2025
@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 17, 2025
@karlkfi karlkfi force-pushed the karl-watch-client-close branch from 5dc32b5 to 7e40f69 Compare April 17, 2025 19:00
@karlkfi karlkfi changed the title [WIP] Fix watch client to close response body on error [WIP] Fix watch client to cancel the request on error Apr 17, 2025
@karlkfi karlkfi force-pushed the karl-watch-client-close branch from 7e40f69 to bbc5503 Compare April 17, 2025 22:13
@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 17, 2025
@karlkfi karlkfi force-pushed the karl-watch-client-close branch from bbc5503 to 9e175a5 Compare April 17, 2025 23:20
@wojtek-t wojtek-t self-assigned this Apr 23, 2025
@karlkfi karlkfi force-pushed the karl-watch-client-close branch from 9e175a5 to 501cb1f Compare April 23, 2025 17:43
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: karlkfi
Once this PR has been reviewed and has the lgtm label, please ask for approval from wojtek-t. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 24, 2025
@karlkfi karlkfi force-pushed the karl-watch-client-close branch from 501cb1f to ee2d421 Compare April 24, 2025 18:02
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Apr 24, 2025
@karlkfi karlkfi changed the title [WIP] Fix watch client to cancel the request on error fix: watch request not cancelled after negotiation error Apr 24, 2025
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 24, 2025
@karlkfi
Copy link
Contributor Author

karlkfi commented Apr 28, 2025

This PR seemed a bit large, so I've pulled out the test changes without the fix, to prove the bugs exist:

@karlkfi karlkfi changed the title fix: watch request not cancelled after negotiation error [WIP] fix: watch request not cancelled after negotiation error Apr 28, 2025
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 28, 2025
@karlkfi karlkfi force-pushed the karl-watch-client-close branch from ee2d421 to e145080 Compare April 28, 2025 04:07
karlkfi added 2 commits April 27, 2025 21:24
- Add channel test helpers to k8s.io/client-go/util/testing
- Use the new channel test helpers in the client and server watch
  tests to validate that the result channels closes without error
  when the client stops the watcher.
- Use the new channel test helpers in the client watcher decoder
  tests to validate that encoded watch events can be decoded and
  that the decoder errors with EOF when stopped asynchronously.

These new tests uncovered existing errors (added TODOs):
1.  The watch client doesn't close the response body when it
    encounters a NegotiateError.
2.  The watch server has a race condition that sometimes sends a
    watch error on the result channel after the client watcher has
    been stopped.
- Fix a bug in client-go Watch and WatchList that was keeping the
  response body open when a NegotiateError was encountered by
  Request.newStreamWatcher. This was causing the server to keep the
  storage watcher and timeout channel open until server-side timeout.
- Fix a bug in client-go Watch and WatchList that was sometimes sending
  a "http: read on closed response body" error from the decoder to the
  result channel after the client watcher had been closed, which closes
  the http response body. The watcher is suppossed to be closed by the
  client when done reading from the result channel, so the impact was
  minimal, but this helps avoid needing to drain the result channel
  before closing it.
@karlkfi karlkfi force-pushed the karl-watch-client-close branch from e145080 to e3cb600 Compare April 28, 2025 04:26
@karlkfi karlkfi changed the title [WIP] fix: watch request not cancelled after negotiation error [WIP] fix: watch client errors Apr 28, 2025
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 10, 2025
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/bug Categorizes issue or PR as related to a bug. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.