Description

Under sustained load on 3008.x (thousands of events/sec, rapid-fire netapi logins, highstate loops), the master, salt-api, and minion processes leak processes, file descriptors, and memory. Master RSS balloons past 7 GB and eventually OOMs. Investigation on 3006.x reproduced the same root causes; they are present in 3008.x head.

Filing as one tracking issue for the whole set — each sub-item has an independent fix.

Reproduction

Stress harness in tests/monitoring/ (docker-compose master + minions + salt-api + Prometheus/Grafana). Drivers:

• flood_events.py — fire salt events at ~5,000/sec into the event bus.
• stress_test.sh — parallel state.apply / test.ping / manage.status loops.
• stress_api.sh — CherryPy netapi login + client=local / client=runner in a tight loop.

Observed within minutes: MWorker RSS climbs unbounded, master FD count climbs, zombie children accumulate. Within an hour master hits multi-GB RSS.

Leak inventory (root causes identified)

Process / FD leaks

1. salt/utils/timed_subprocess.py — TimedProc.kill() does not wait(), orphaning zombies to PID 1 after timeout.
2. salt/utils/event.py — SaltEvent/MasterEvent have no context-manager protocol and no full teardown; ZMQ sockets and event-bus connections outlive their callers.
3. salt/netapi/__init__.py — NetapiClient instantiates LocalClient / RunnerClient / WheelClient / SSHClient per request without teardown. Primary source of the CherryPy TCP/RAM leak.
4. salt/runner.py, salt/wheel/__init__.py — no destroy(); callers using bare instantiation leak per-call.

Memory leaks / cache bloat

1. salt/loader/lazy.py — LazyLoader never purges _dict, loaded_modules, or sys.modules entries. Every reload leaks the module set → "module creep" on both master and minion.
2. salt/utils/context.py, salt/utils/ctx.py — ContextDict / RequestContext.current_request not cleared on teardown; per-request state pinned indefinitely.
3. salt/master.py, salt/daemons/masterapi.py — AESFuncs / RemoteFuncs bound methods (e.g. _serve_file) create refcycles that pin Fileserver and friends past worker recycle.
4. salt/utils/args.py — get_function_argspec() defined _ArgSpec = namedtuple(...) at function scope, dynamically compiling a fresh class (C-level struct) on every module call. At high call rates this is a fast, unbounded leak.
5. Worker recycling + PyZMQ asyncio.Task: closing ZMQStream sockets mid-flight while _run_callback has wrapped coroutines via asyncio.ensure_future permanently leaks the Task objects into the running loop. Aggressive worker_resource_backcount recycling made this worse, not better.
6. Destructor cascades missing on the loader-owning classes — Fileserver, Cache, Roster, MasterMinion, RemoteFuncs/LocalFuncs, NetapiClient, LoadAuth, Runner. Even when callers do the right thing, the owned LazyLoaders stay in sys.modules.

Scalability

1. salt/tokens/localfs.py — list_tokens uses os.walk and returns a full list; on token dirs with 1M+ entries this is a GB-scale spike during clean_expired_tokens. Same code path in salt/daemons/masterapi.py Maintenance.

IPC

1. EventPublisher inherits Tornado's default ipc_write_buffer (~27 GB effective). A slow consumer causes unbounded RAM growth on the publisher instead of a disconnect.

Fix status (staged locally on 3006.x branch dwoz/3006leak; forward-porting to 3008.x)

• TimedProc.wait() after kill().
• SaltEvent/MasterEvent context managers + full destroy().
• NetapiClient uses with for all sub-clients; Runner/Wheel gain destroy() and context-manager use.
• LazyLoader.destroy() explicitly purges _dict, loaded_modules, and matching sys.modules keys.
• ContextDict.destroy(); RequestContext clears current_request.
• AESFuncs.destroy() / RemoteFuncs.destroy() nullify bound-method attributes (self._serve_file = None, etc.) to break the refcycle.
• _ArgSpec namedtuple hoisted to module scope in salt/utils/args.py.
• worker_resource_backcount recycle path removed — workers cache execution context; saves ~600 MB globally and eliminates the PyZMQ Task leak. (If a bounded reset is still wanted, it must first drain outstanding ZMQStream callbacks.)
• Destructor cascades wired into Fileserver, Cache, Roster, MasterMinion, RemoteFuncs, LocalFuncs, NetapiClient, LoadAuth, Runner.
• list_tokens converted to os.scandir-based generator; Maintenance.clean_expired_tokens consumes the generator (constant memory).
• Recommend a sane default cap for ipc_write_buffer on EventPublisher (or hard cap + disconnect on overflow).

Post-fix validation

• Master baseline ~70 MB / peak ~155 MB RSS (was 7 GB+ before OOM).
• Minion ~108 MB stable (was 1.3 GB).
• salt-api ~53 MB stable.
• Master FDs stable at ~1,300 under ~5,700 events/sec.

Remaining slow leak (unresolved)

After the above, MWorker still creeps ~25 MB/hr under sustained stress; MWorkerQueue occasionally accumulates. Profiler shows ReferenceType, Context, generator, and Task counts ticking upward. Suspects (unconfirmed):

• Residual salt.ext.tornado.zmq.eventloop.zmqstream future retention.
• Unbounded cache inside a hot execution module (test.ping, manage.status, grains.items) or in CherryPy session handling.
• IOLoop.add_future callbacks not clearing on external-auth paths.

Will file a follow-up narrowly-scoped issue once tracemalloc/objgraph isolate it.

Target

3008.x. 3006.x carries the same code paths (verified) but is LTS — separate backport scope.

salt --versions-report

3008.x head (branch tip at time of filing). Reproduction environment is Ubuntu 24.04 containers under docker-compose (see tests/monitoring/).