Reduce excessive test output in CI/Helix logs - #54782
#54782Reduce excessive test output in CI/Helix logs#54782
Conversation
- Disable xUnit diagnosticMessages and showLiveOutput in xunit.runner.json and dotnet.Tests/app.config to eliminate internal xUnit chatter - Buffer TestCommand stdout/stderr and only dump on failure; on success, log a single summary line with suppressed line counts - Remove -v:diag verbosity escalation on NU3003 in MSBuildCommand; the binlog (already uploaded to Helix) provides full diagnostic data - Raise XunitLoggerProvider default log level from Trace to Information Set DOTNET_SDK_TEST_VERBOSE=1 to restore full real-time output for local debugging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR reduces excessive Helix/CI test log volume by disabling xUnit diagnostic/live output, buffering command output so it’s only emitted on failure by default, and avoiding MSBuild verbosity escalation retries.
Changes:
- Turn off xUnit diagnostic messages and live output to reduce baseline noise.
- Buffer
TestCommandstdout/stderr and only dump it on failures (opt-in verbose streaming viaDOTNET_SDK_TEST_VERBOSE=1). - Stop re-running MSBuild with
-v:diagonNU3003, relying on uploaded binlogs instead; raiseXunitLoggerProviderdefault minimum log level.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/xunit.runner.json | Disables xUnit diagnostic messages and live output globally. |
| test/dotnet.Tests/app.config | Disables xUnit diagnostic messages for dotnet.Tests. |
| test/Microsoft.NET.TestFramework/Commands/TestCommand.cs | Buffers process output and suppresses it on success unless verbose mode is enabled. |
| test/Microsoft.NET.TestFramework/Commands/MSBuildCommand.cs | Removes -v:diag retry escalation on NU3003 and logs guidance to use binlogs. |
| test/Microsoft.NET.TestFramework/XunitLoggerProvider.cs | Raises default ILogger minimum level to Information to reduce logging noise. |
PowerShell script that queries AzDO build timelines to extract Helix job IDs, then uses HTTP HEAD against console log blob URLs to measure log sizes without downloading them. Supports single-build analysis and side-by-side comparison of two builds. Outputs top-N largest work items and optional CSV export. Usage: # Single build analysis .\scripts\helix-log-sizes.ps1 -BaselineBuildId 1465148 # Compare baseline vs PR build .\scripts\helix-log-sizes.ps1 -BaselineBuildId 1465148 -ComparisonBuildId <new> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
I picked a passing build against main and had copilot do a data analysis of the log file size as I suspect that this will not be enough. Either way, we're definitely making it hard for a human to investigate and wasting copilot resources digging through gigantic log files. 
|
This assumes failures only occur when a command fails. This is not always the case, the test can expect the command to produce different results/output. Suppressing the output in these scenarios could make it difficult to diagnose failures without reproducing locally and enabling DOTNET_SDK_TEST_VERBOSE. If this proves to be a common occurrence, we may want to add API support for enabling the verbose output on the TestCommand so the test can opt in. |
|
@marcpopMSFT, @dsplaisted - How do you see this PR integrating with #54795? |
#54795 is against 9.0.1xx, where the test logging infrastructure is slightly different. So probably we could take that for 9.0 and this one for main. |
…t API, clean up comments - Remove duplicate List<string> buffering; use result.StdOut/StdErr (already captured by CaptureStdOut/CaptureStdErr) to dump output on failure, avoiding doubled memory pressure in CI - Add TestCommand.VerboseOutput property so individual tests can opt into real-time output streaming without requiring the global env var - Rewrite NU3003 comment and log message to not reference old behavior - Ensure CommandOutputHandler is still invoked in non-verbose mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Copilot analysis of the improvements from this PR, looks like it provided significant log file reduction. |
String.Split(string) is not available on .NET Framework. Use String.Split(string[], StringSplitOptions) instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Final results for this PR: |
Summary
Reduces the volume of test output in Helix console logs, making it significantly easier to find [fail] markers and reducing costs for both human reviewers and agent-based log analysis.
Problem
Helix console logs for PR builds were excessively large, making test failure investigation difficult. The root cause was not MSBuild diagnostic-level builds (CI defaults to \minimal\ verbosity), but rather the test framework itself logging too aggressively.
Changes
1. Disable xUnit diagnostic output (high impact)
2. Buffer TestCommand output — only dump on failure (highest impact)
3. Remove -v:diag\ retry on NU3003 (medium impact)
4. Raise XunitLoggerProvider minimum log level (moderate impact)
Not included (follow-up)