Reduce the risk of waiting on the scheduling queue Pop(), despite having pods in backoffQ

With SchedulerPopFromBackoffQ feature enabled, if activeQ is empty, but there is a pod in backoffQ, we try to schedule the pod from there. Because the backoffQ is locked with a different lock than activeQ, it is possible that between a check of backoffQ emptiness (line 260 below) and waiting on a condition (line 270), the pod might be added to the backoffQ in the meantime, leading to unnecessary waiting on condition (see comment). It doesn't regress the scenario of popping pods from the activeQ, but might affect popping from the backoffQ.

kubernetes/pkg/scheduler/backend/queue/active_queue.go

Lines 255 to 271 in 47a61c5

    
           func (aq *activeQueue) unlockedPop(logger klog.Logger) (*framework.QueuedPodInfo, error) { 
        
           	var pInfo *framework.QueuedPodInfo 
        
           	for aq.queue.Len() == 0 { 
        
           		// backoffQPopper is non-nil only if SchedulerPopFromBackoffQ feature is enabled. 
        
           		// In case of non-empty backoffQ, try popping from there. 
        
           		if aq.backoffQPopper != nil && aq.backoffQPopper.lenBackoff() != 0 { 
        
           			break 
        
           		} 
        
           		// When the queue is empty, invocation of Pop() is blocked until new item is enqueued. 
        
           		// When Close() is called, the p.closed is set and the condition is broadcast, 
        
           		// which causes this loop to continue and return from the Pop(). 
        
           		if aq.closed { 
        
           			logger.V(2).Info("Scheduling queue is closed") 
        
           			return nil, nil 
        
           		} 
        
           		aq.cond.Wait() 
        
           	}

/sig scheduling
/kind cleanup

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reduce the risk of waiting on the scheduling queue Pop(), despite having pods in backoffQ #130976

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	func (aq activeQueue) unlockedPop(logger klog.Logger) (framework.QueuedPodInfo, error) {
	var pInfo *framework.QueuedPodInfo
	for aq.queue.Len() == 0 {
	// backoffQPopper is non-nil only if SchedulerPopFromBackoffQ feature is enabled.
	// In case of non-empty backoffQ, try popping from there.
	if aq.backoffQPopper != nil && aq.backoffQPopper.lenBackoff() != 0 {
	break
	}
	// When the queue is empty, invocation of Pop() is blocked until new item is enqueued.
	// When Close() is called, the p.closed is set and the condition is broadcast,
	// which causes this loop to continue and return from the Pop().
	if aq.closed {
	logger.V(2).Info("Scheduling queue is closed")
	return nil, nil
	}
	aq.cond.Wait()
	}

Search code, repositories, users, issues, pull requests...

Reduce the risk of waiting on the scheduling queue Pop(), despite having pods in backoffQ #130976

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions