Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

warm: generated warmer wrapper omits callbackWaitsForEmptyEventLoop=false — warm pings on streaming SSR functions silently burn the full function timeout #6951

Copy link
Copy link

Description

@UzairM
Issue body actions

Summary

When warm: is set on an SSR site with a streaming server function (e.g. sst.aws.TanStackStart, Node runtime), the generated server-index.mjs wrapper's warmer branch ends the response stream and returns without setting context.callbackWaitsForEmptyEventLoop = false. The Lambda runtime then waits post-return for the event loop to drain. If the warmed sandbox holds any pending handles from earlier real invocations (keep-alive upstream sockets, timers, telemetry exporters), the warm-ping invocation silently burns the full function timeout (30s in our case) and is billed for it.

Mechanism

useServerWarmingInjection (platform/src/components/aws/ssr-site.ts) emits, for the streaming case:

if (event.type === "warmer") {
  const p = new Promise((resolve) => {
    setTimeout(() => {
      resolve({ serverId: "server-" + Math.random().toString(36).slice(2, 8) });
    }, event.delay);
  });
  const response = await p;
  responseStream.write(JSON.stringify(response));
  responseStream.end();
  return;   // <-- callbackWaitsForEmptyEventLoop never set to false
}

For awslambda.streamifyResponse handlers, responseStream.end() completes the Invoke response (so the warmer sees a success and logs uniqueServersWarmed normally), but the invocation itself only ends when the event loop drains — unless context.callbackWaitsForEmptyEventLoop = false is set. Same post-return drain-freeze as opennextjs/aws#487, just on the warming path.

Because the wrapper short-circuits before await import("./index.mjs"), the app handler can never set the flag itself for warmer events, and no user code runs — the timeouts are completely silent (log shows only START/END/REPORT with Status: timeout on the REPORT line; note streaming functions never log "Task timed out after").

Observed impact (production, sst 3.17.25, nodejs22.x, warm: 3, rate(5 minutes))

  • 272–478 warm-ping invocations/day dying at exactly 30,000ms, phase-locked to the warmer cron (start offsets identical mod 300s), frequently as same-millisecond triples (all 3 concurrent pings of one cycle).
  • Each hit: full timeout billed (30s × function memory), a concurrency slot held 30s, the Lambda Errors metric polluted (false alarms), and the warm sandbox reset — forcing extra user-facing cold starts, the opposite of what warm is for.
  • The warmer is structurally blind to it: the stream ends in ~75–80ms so success counts it, and the random per-ping serverId makes uniqueServersWarmed === success by construction.

Repro

  1. Deploy a streaming SSR site (warm: 1 is enough) whose app makes ordinary keep-alive HTTP calls to any upstream during real requests (i.e. leaves sockets in the agent pool).
  2. Send a real request to a sandbox, then let the warmer ping that sandbox.
  3. Observe the ping's REPORT line: Duration: 30000.00 ms ... Status: timeout, while the warmer function logs success for the same cycle.

Fix

Emit context.callbackWaitsForEmptyEventLoop = false; as the first line of the warming injection (both streaming and non-streaming branches are safe — context is in scope in both generated wrapper signatures). Verified in production: prepending exactly that line to the wrapper injections eliminates the timeouts while warm-ping durations stay ~15–80ms.

Happy to send a PR if useful.

Reactions are currently unavailable

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.