Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Drain crashed testhost stderr without blocking a thread-pool thread - #16191

#16191
Open
nohwnd wants to merge 6 commits into
microsoft:mainmicrosoft/vstest:mainfrom
nohwnd:nohwnd-flaky-test-investigationnohwnd/vstest:nohwnd-flaky-test-investigationCopy head branch name to clipboard
Open

Drain crashed testhost stderr without blocking a thread-pool thread#16191
nohwnd wants to merge 6 commits into
microsoft:mainmicrosoft/vstest:mainfrom
nohwnd:nohwnd-flaky-test-investigationnohwnd/vstest:nohwnd-flaky-test-investigationCopy head branch name to clipboard

Conversation

@nohwnd

@nohwnd nohwnd commented Jun 29, 2026

Copy link
Copy Markdown
Member

RunTestsShouldThrowOnStackOverflowException is still flaky, even after #16128. The abort message comes back as just Process path: ...\testhost.exe with the Stack overflow. line missing.

When a testhost crashes, .NET writes Stack overflow. to its stderr, and we collect that asynchronously through ErrorDataReceived. Under load - many test hosts running in parallel on CI, thread-pool starvation - that callback can fire noticeably late, after the process has already exited. #16128 added a wait for the stderr stream to reach EOF before reading it, but two things kept it flaky:

  • the wait shared a single 500ms budget with the process-exit wait, so by the time we got to draining there was often little of it left, and
  • it blocked a thread-pool thread, competing for the very thread the pending ErrorDataReceived callback needs to deliver EOF.

So under starvation the wait could starve out the thing it was waiting for, time out, and read an empty buffer - dropping the callstack and truncating the abort message.

This decouples the stderr drain from the exit-wait budget and awaits the EOF signal instead of blocking on it. A crash gets a generous bounded budget (5s); a clean exit keeps a short 500ms grace period so the common case never pays. The wait returns the instant EOF arrives, so a process that drains promptly costs almost nothing.

That 5s budget is only right for a genuine crash, though. When we kill a still-running testhost on purpose - aborting or cleaning up a run from an IDE - it exits with a non-zero code that looks exactly like a crash, and if a grandchild process (e.g. a browser driver) holds the stderr pipe open, EOF never comes and the abort would hang for the full budget. So I track the processes we kill in TerminateProcess and treat their exit as an abort with the short drain; a process that exited on its own is never marked, so a real crash still gets the generous budget.

ProcessHelperTests cover the drain primitive (returns on EOF, stays bounded when EOF never comes, no-op when there is nothing to drain) and the budget selection (crash vs abort/clean exit), and RunTestsShouldThrowOnStackOverflowException passes locally. The race doesn't reproduce deterministically here, so the real proof is the CI flake rate.

RunTestsShouldThrowOnStackOverflowException is still flaky. When a testhost crashes, the "Stack overflow." line can reach ErrorDataReceived late under thread-pool starvation (many hosts running in parallel on CI). The exit handler shared one 500ms budget with the process-exit wait and then blocked a thread-pool thread on it, so it both ran low on time and competed for the very thread that delivers the output it was waiting for. The callstack got dropped and the abort message was truncated to just the process path.

Decouple the stderr drain from the exit-wait budget, await the EOF signal instead of blocking it, and only spend the generous budget when the process exited abnormally. Clean exits keep a short grace period and the wait returns as soon as EOF arrives.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 29, 2026 14:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens ProcessHelper’s crash diagnostics by awaiting stderr EOF asynchronously (with separate clean-exit vs crash budgets) so late-delivered crash output like Stack overflow. isn’t truncated under thread-pool starvation.

Changes:

  • Replace the stderr “drain” primitive with an async, non-thread-blocking WaitForErrorStreamToDrainAsync and use longer timeout on abnormal exit.
  • Update ProcessHelper’s stderr EOF signal from ManualResetEventSlim to TaskCompletionSource (with RunContinuationsAsynchronously).
  • Update ProcessHelperTests to cover the new async drain helper.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/Microsoft.TestPlatform.PlatformAbstractions/common/System/ProcessHelper.cs Await stderr EOF without blocking a thread-pool thread; use separate drain timeouts for clean vs crashed exits.
test/vstest.console.UnitTests/ProcessHelperTests.cs Convert drain helper tests to async and validate bounded / no-op behavior.

Comment thread test/vstest.console.UnitTests/ProcessHelperTests.cs
@nohwnd
nohwnd marked this pull request as draft June 29, 2026 14:51
@nohwnd

nohwnd commented Jun 29, 2026

Copy link
Copy Markdown
Member Author

This is about tenth attempt at fixing this, so needs careful review, especially considering that the process needs to exit fast in IDE, especially where there is no crash.

The generous stderr-drain budget is only meant for a genuine crash. When
we kill a still-running test host on purpose - aborting or cleaning up a
run from an IDE - it exits with a non-zero code that is indistinguishable
from a crash, so it would wait the full crash budget. If a grandchild
process (e.g. a browser driver) inherited the stderr handle and keeps the
pipe open, EOF never arrives and the abort hangs for seconds.

Record the processes we kill in TerminateProcess and treat their exit as
an abort (short drain), not a crash. A process that exited on its own is
never recorded, so a real crash still gets the generous budget needed to
capture a late callstack. Added a unit test for the budget selection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The CancellationTokenSource that bounds the post-exit wait is created with a timeout, so it allocates a timer; dispose it via 'using' so we don't leak one per testhost exit when many hosts are spawned.

The stderr-drain test had been reduced to the already-drained fast path, so it could not catch a regression where EOF arrives just after the exit handler starts waiting. Make EOF land ~150ms into the wait so the test exercises waiting for a late ErrorDataReceived delivery and returning promptly, and keep a separate fast-path test for the already-drained case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 30, 2026 13:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

nohwnd and others added 2 commits July 1, 2026 13:29
The late-EOF test from the previous commit signaled EOF from a Task.Run
with a Task.Delay, neither taking a CancellationToken. That trips
MSTEST0049, which is only a warning in Debug but an error in the Release
build CI runs - so the Windows leg failed at compile and every test was
skipped. That is why the build went red without a test actually failing.

Rewrite it without a timed background task: start the wait, assert it is
still in progress, then signal EOF and await it. That exercises the same
"EOF lands after the wait has started" path, but deterministically - no
sleep, no timing assertion that could itself flake - and without the
analyzer-tripping Task.Run/Task.Delay.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the process overruns the 500ms exit budget on the non-Windows
!NET path, the CancellationToken callback force-kills it. That kill was
not recorded via MarkDeliberatelyTerminated, so the exit handler saw a
non-zero exit, treated it as a crash, and waited the generous 5s stderr
drain - the exact abort latency this change avoids. Mark it before the
kill, mirroring TerminateProcess, so the drain uses the short budget.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 1, 2026 12:38
@nohwnd
nohwnd marked this pull request as ready for review July 1, 2026 12:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread test/vstest.console.UnitTests/ProcessHelperTests.cs
Comment thread src/Microsoft.TestPlatform.PlatformAbstractions/common/System/ProcessHelper.cs Outdated
… returns promptly

The already-drained case is the common one on a clean exit; short-circuit before allocating the CancellationTokenSource and Task.Delay timer. Also tighten the late-EOF test so it fails if the wait ever ignored EOF and ran to the full timeout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nohwnd
nohwnd enabled auto-merge (squash) July 3, 2026 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.