Drain crashed testhost stderr without blocking a thread-pool thread - #16191
#16191Open
nohwnd wants to merge 6 commits into
microsoft:mainmicrosoft/vstest:mainfrom
nohwnd:nohwnd-flaky-test-investigationnohwnd/vstest:nohwnd-flaky-test-investigationCopy head branch name to clipboard
Open
Drain crashed testhost stderr without blocking a thread-pool thread#16191nohwnd wants to merge 6 commits intomicrosoft:mainmicrosoft/vstest:mainfrom nohwnd:nohwnd-flaky-test-investigationnohwnd/vstest:nohwnd-flaky-test-investigationCopy head branch name to clipboard
nohwnd wants to merge 6 commits into
microsoft:mainmicrosoft/vstest:mainfrom
nohwnd:nohwnd-flaky-test-investigationnohwnd/vstest:nohwnd-flaky-test-investigationCopy head branch name to clipboard
Conversation
RunTestsShouldThrowOnStackOverflowException is still flaky. When a testhost crashes, the "Stack overflow." line can reach ErrorDataReceived late under thread-pool starvation (many hosts running in parallel on CI). The exit handler shared one 500ms budget with the process-exit wait and then blocked a thread-pool thread on it, so it both ran low on time and competed for the very thread that delivers the output it was waiting for. The callstack got dropped and the abort message was truncated to just the process path. Decouple the stderr drain from the exit-wait budget, await the EOF signal instead of blocking it, and only spend the generous budget when the process exited abnormally. Clean exits keep a short grace period and the wait returns as soon as EOF arrives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens ProcessHelper’s crash diagnostics by awaiting stderr EOF asynchronously (with separate clean-exit vs crash budgets) so late-delivered crash output like Stack overflow. isn’t truncated under thread-pool starvation.
Changes:
- Replace the stderr “drain” primitive with an async, non-thread-blocking
WaitForErrorStreamToDrainAsyncand use longer timeout on abnormal exit. - Update
ProcessHelper’s stderr EOF signal fromManualResetEventSlimtoTaskCompletionSource(withRunContinuationsAsynchronously). - Update
ProcessHelperTeststo cover the new async drain helper.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/Microsoft.TestPlatform.PlatformAbstractions/common/System/ProcessHelper.cs |
Await stderr EOF without blocking a thread-pool thread; use separate drain timeouts for clean vs crashed exits. |
test/vstest.console.UnitTests/ProcessHelperTests.cs |
Convert drain helper tests to async and validate bounded / no-op behavior. |
nohwnd
marked this pull request as draft
June 29, 2026 14:51
Member
Author
|
This is about tenth attempt at fixing this, so needs careful review, especially considering that the process needs to exit fast in IDE, especially where there is no crash. |
The generous stderr-drain budget is only meant for a genuine crash. When we kill a still-running test host on purpose - aborting or cleaning up a run from an IDE - it exits with a non-zero code that is indistinguishable from a crash, so it would wait the full crash budget. If a grandchild process (e.g. a browser driver) inherited the stderr handle and keeps the pipe open, EOF never arrives and the abort hangs for seconds. Record the processes we kill in TerminateProcess and treat their exit as an abort (short drain), not a crash. A process that exited on its own is never recorded, so a real crash still gets the generous budget needed to capture a late callstack. Added a unit test for the budget selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The CancellationTokenSource that bounds the post-exit wait is created with a timeout, so it allocates a timer; dispose it via 'using' so we don't leak one per testhost exit when many hosts are spawned. The stderr-drain test had been reduced to the already-drained fast path, so it could not catch a regression where EOF arrives just after the exit handler starts waiting. Make EOF land ~150ms into the wait so the test exercises waiting for a late ErrorDataReceived delivery and returning promptly, and keep a separate fast-path test for the already-drained case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The late-EOF test from the previous commit signaled EOF from a Task.Run with a Task.Delay, neither taking a CancellationToken. That trips MSTEST0049, which is only a warning in Debug but an error in the Release build CI runs - so the Windows leg failed at compile and every test was skipped. That is why the build went red without a test actually failing. Rewrite it without a timed background task: start the wait, assert it is still in progress, then signal EOF and await it. That exercises the same "EOF lands after the wait has started" path, but deterministically - no sleep, no timing assertion that could itself flake - and without the analyzer-tripping Task.Run/Task.Delay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the process overruns the 500ms exit budget on the non-Windows !NET path, the CancellationToken callback force-kills it. That kill was not recorded via MarkDeliberatelyTerminated, so the exit handler saw a non-zero exit, treated it as a crash, and waited the generous 5s stderr drain - the exact abort latency this change avoids. Mark it before the kill, mirroring TerminateProcess, so the drain uses the short budget. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… returns promptly The already-drained case is the common one on a clean exit; short-circuit before allocating the CancellationTokenSource and Task.Delay timer. Also tighten the late-EOF test so it fails if the wait ever ignored EOF and ran to the full timeout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nohwnd
enabled auto-merge (squash)
July 3, 2026 11:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RunTestsShouldThrowOnStackOverflowExceptionis still flaky, even after #16128. The abort message comes back as justProcess path: ...\testhost.exewith theStack overflow.line missing.When a testhost crashes, .NET writes
Stack overflow.to its stderr, and we collect that asynchronously throughErrorDataReceived. Under load - many test hosts running in parallel on CI, thread-pool starvation - that callback can fire noticeably late, after the process has already exited. #16128 added a wait for the stderr stream to reach EOF before reading it, but two things kept it flaky:ErrorDataReceivedcallback needs to deliver EOF.So under starvation the wait could starve out the thing it was waiting for, time out, and read an empty buffer - dropping the callstack and truncating the abort message.
This decouples the stderr drain from the exit-wait budget and awaits the EOF signal instead of blocking on it. A crash gets a generous bounded budget (5s); a clean exit keeps a short 500ms grace period so the common case never pays. The wait returns the instant EOF arrives, so a process that drains promptly costs almost nothing.
That 5s budget is only right for a genuine crash, though. When we kill a still-running testhost on purpose - aborting or cleaning up a run from an IDE - it exits with a non-zero code that looks exactly like a crash, and if a grandchild process (e.g. a browser driver) holds the stderr pipe open, EOF never comes and the abort would hang for the full budget. So I track the processes we kill in
TerminateProcessand treat their exit as an abort with the short drain; a process that exited on its own is never marked, so a real crash still gets the generous budget.ProcessHelperTestscover the drain primitive (returns on EOF, stays bounded when EOF never comes, no-op when there is nothing to drain) and the budget selection (crash vs abort/clean exit), andRunTestsShouldThrowOnStackOverflowExceptionpasses locally. The race doesn't reproduce deterministically here, so the real proof is the CI flake rate.