What happened?

We have recently noticed an increase in gRPC errors (specifically, use of closed network connection) in apiserver logs.

log sample
I0619 00:22:[31.927056 11](tel:3192705611) http2_client.go:959] "[transport] [client-transport 0xc004144900] Closing: connection error: desc = \"error reading from server: read tcp 10.0.33.210:43716→10.0.32.16:2379: use of closed network connection\"\n"

These errors occurs intermittently every few hours during the lifecycle of apiserver.

Although there have been no reported performance regression yet, the recurring error message is concerning.

What did you expect to happen?

Assume the error may occur when the apiserver is restarted, rather than during its lifecycle.

How can we reproduce it (as minimally and precisely as possible)?

launch apiserver against k8s version v1.28.9

Anything else we need to know?

No response

Kubernetes version

Cloud provider

OS version

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)