Description

The analysis ONNX sessions size their intra-op thread pool from the host's core count, not from the container's CPU limit. In a containerized deployment with a CPU cap, every worker replica spawns far more inference threads than it has CPU to run them on. They stay runnable-but-throttled by the CFS quota, so the symptom is a very high load average with an idle CPU and no I/O wait — nothing errors, nothing crashes, and throughput just quietly falls short of what the replica count suggests.

The CLAP path already has a knob for exactly this. The MusiCNN/analysis path does not.

Environment

The asymmetry

tasks/clap_analyzer.py deliberately constrains the pool:

def _clap_session_options(label):
    sess_options = ort.SessionOptions()
    ...
    if config.CLAP_PYTHON_MULTITHREADS:
        sess_options.intra_op_num_threads = 1
        sess_options.inter_op_num_threads = 1
        logger.info("CLAP %s: Using Python threading, ONNX single-threaded", label)
    else:
        logger.info("CLAP %s: Using ONNX Runtime automatic thread management", label)

tasks/analysis/song.py never does:

def _default_sess_options():
    opts = ort.SessionOptions()
    opts.enable_cpu_mem_arena = False
    opts.enable_mem_pattern = False
    return opts

intra_op_num_threads stays 0 (auto), and ORT sizes that pool from the detected core count.

Why the container limit doesn't help

deploy.resources.limits.cpus is a CFS bandwidth quota, not an affinity mask, so core detection inside the container still reports the host:

$ nproc                                     -> 12
$ python3 -c 'import os; print(len(os.sched_getaffinity(0)), os.cpu_count())'
12 12

Measured thread count of a single bare InferenceSession on /app/model/musicnn_embedding.onnx:

The 10-thread delta is the intra-op pool going 12 -> 2.

Host-level symptom

With 3 replicas x 2 CPU on a 12-core host, during analysis:

load average: 28.56, 16.61, 8.66
%Cpu(s): 31.1 us, 9.1 sy, 59.8 id, 0.0 wa

runnable (R) threads: 35
uninterruptible (D) threads: 0

Load at 2.4x the core count while the CPU is 60% idle with zero I/O wait — the threads are runnable and throttled, not blocked. Live analysis child processes carried 33-44 threads each against a 2-CPU budget.

What does not work as a workaround

I tried the obvious two before opening this, and both failed in the real worker — worth recording so nobody else spends the time:

• OMP_NUM_THREADS=2 shrinks the synthetic single-session case (23 -> 13, above), but deployed to a live worker it changed nothing measurable. Thread counts on the treated host were 44/44/11 against 33/33/44 on an identical untreated host, and throughput over a 10-minute window was 90 tracks vs 92 before the change — noise. ORT 1.19 doesn't route its intra-op pool through OpenMP, and the analysis child evidently builds several pools that this doesn't reach.
• --cpuset-cpus does make detection correct, but it is no better than the env var in the synthetic test, and it forces you to abandon deploy.replicas for individually pinned services.

So there is no deployment-side fix available; it has to come from the session options.

Suggested fix

Give the analysis path the same control CLAP already has — have _default_sess_options() accept a thread count, sourced from config/env, e.g.:

def _default_sess_options(num_threads=None):
    opts = ort.SessionOptions()
    opts.enable_cpu_mem_arena = False
    opts.enable_mem_pattern = False
    n = num_threads or config.ANALYSIS_ONNX_THREADS   # 0 = keep current auto behaviour
    if n and int(n) > 0:
        opts.intra_op_num_threads = int(n)
        opts.inter_op_num_threads = 1
    return opts

Defaulting it to 0 preserves today's behaviour for anyone running uncontainerized or uncapped, while letting container deployments match the pool to the quota they actually have.

Happy to test a patch against this setup — I have two identical 12-core hosts on the same queue and can A/B it.