Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

[fix] fix ds controller deletes pod when not match RequiredDuringSchedulingIgnoredDuringExecution #129315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
Loading
from
Open

[fix] fix ds controller deletes pod when not match RequiredDuringSchedulingIgnoredDuringExecution #129315

wants to merge 1 commit into from

Conversation

xyz-li
Copy link
Contributor

@xyz-li xyz-li commented Dec 20, 2024

fix ds controller not match RequiredDuringSchedulingIgnoredDuringExecution

What type of PR is this?

/kind bug

What this PR does / why we need it:

IgnoredDuringExecution means that if the node labels change after Kubernetes schedules the Pod, the Pod continues to run.

Which issue(s) this PR fixes:

Fixes #129196

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 20, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot
Copy link
Contributor

Hi @xyz-li. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Dec 20, 2024
@xyz-li xyz-li changed the title [fix] fix ds controller evicts pod when not match RequiredDuringSchedulingIgnoredDuringExecution [fix] fix ds controller deletes pod when not match RequiredDuringSchedulingIgnoredDuringExecution Dec 20, 2024
@k8s-ci-robot k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 20, 2024
@xyz-li xyz-li force-pushed the ds_controller branch 3 times, most recently from 043429f to 0bc2d9e Compare December 20, 2024 03:30
@xyz-li
Copy link
Contributor Author

xyz-li commented Dec 20, 2024

/sig apps

@xyz-li
Copy link
Contributor Author

xyz-li commented Jan 3, 2025

/cc @deads2k

@k8s-ci-robot k8s-ci-robot requested a review from deads2k January 3, 2025 14:20
This was referenced Feb 13, 2025
Merged
Merged
badouralix
badouralix approved these changes Mar 11, 2025
View reviewed changes
Copy link

@badouralix badouralix left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We hit a similar issue where cilium-agent pods were evicted from running nodes when the nodes labels were changed after the pods were scheduled
We tested this patch on clusters of various size and various load and can confirm it fixes the issue without encountering side effects
Any chance we can get this into upstream ? Happy to provide more insights or test it more if it can help moving forward

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: badouralix, pl-jankowskimichal, xyz-li
Once this PR has been reviewed and has the lgtm label, please assign kow3ns for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@xyz-li
Copy link
Contributor Author

xyz-li commented Mar 12, 2025

/cc @andrewsykim
/cc @deads2k
/cc @cheftako

@k8s-ci-robot k8s-ci-robot requested a review from cheftako March 12, 2025 14:15
@xyz-li
Copy link
Contributor Author

xyz-li commented Mar 19, 2025

@dims

@xyz-li
Copy link
Contributor Author

xyz-li commented Mar 27, 2025

@wojtek-t Please help review this PR.

@xyz-li
Copy link
Contributor Author

xyz-li commented Apr 26, 2025

/cc @janetkuo
/cc @kow3ns

@k8s-ci-robot k8s-ci-robot requested review from janetkuo and kow3ns April 26, 2025 02:19
@xyz-li
Copy link
Contributor Author

xyz-li commented May 7, 2025

/cc @mimowo

@k8s-ci-robot k8s-ci-robot requested a review from mimowo May 7, 2025 12:15
janetkuo
janetkuo reviewed May 9, 2025
View reviewed changes
pkg/controller/daemon/daemon_controller.go Show resolved Hide resolved
@xyz-li
[fix] fix ds controller deletes pod when not match RequiredDuringSche…
b71d060 
…dulingIgnoredDuringExecution

Signed-off-by: xyz-li <hui0787411@163.com>
@lojies
Copy link
Contributor

lojies commented May 13, 2025

"requiredDuringSchedulingIgnoredDuringExecution evicts DaemonSet when node label is removed — is this expected behavior? We've encountered the same issue. Is there any ongoing discussion or plans around this?"

@lojies lojies moved this from Needs Triage to In Progress in SIG Apps May 13, 2025
@lojies lojies moved this from In Progress to Needs Triage in SIG Apps May 13, 2025
@xyz-li
Copy link
Contributor Author

xyz-li commented May 13, 2025 •
edited
Loading

kubectl explain ds.spec.template.spec.affinity.nodeAffinity

  requiredDuringSchedulingIgnoredDuringExecution	<NodeSelector>
    If the affinity requirements specified by this field are not met at
    scheduling time, the pod will not be scheduled onto the node. If the
    affinity requirements specified by this field cease to be met at some point
    during pod execution (e.g. due to an update), the system may or may not try
    to eventually evict the pod from its node.

@xyz-li
Copy link
Contributor Author

xyz-li commented May 14, 2025

"requiredDuringSchedulingIgnoredDuringExecution evicts DaemonSet when node label is removed — is this expected behavior? We've encountered the same issue. Is there any ongoing discussion or plans around this?"

requiredDuringSchedulingIgnoredDuringExecution will not evict Deployment when node label is removed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@janetkuo janetkuo janetkuo left review comments

@badouralix badouralix badouralix approved these changes

@atiratree atiratree Awaiting requested review from atiratree

@soltysh soltysh Awaiting requested review from soltysh

@deads2k deads2k Awaiting requested review from deads2k

@pl-jankowskimichal pl-jankowskimichal Awaiting requested review from pl-jankowskimichal

@andrewsykim andrewsykim Awaiting requested review from andrewsykim

@cheftako cheftako Awaiting requested review from cheftako

@kow3ns kow3ns Awaiting requested review from kow3ns

@mimowo mimowo Awaiting requested review from mimowo

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
SIG Apps
Status: Needs Triage
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

'requiredDuringSchedulingIgnoredDuringExecution' evicts Daemonset when node label removed
6 participants
@xyz-li @k8s-ci-robot @lojies @janetkuo @pl-jankowskimichal @badouralix
Morty Proxy This is a proxified and sanitized view of the page, visit original site.