Support UTF-16 BOM in global.json by Copilot · Pull Request #53141 · dotnet/sdk

Copilot · Feb 24, 2026

global.json is user-authored and can be written with UTF-16 encoding (e.g., PowerShell's Out-File defaults to UTF-16 on Windows). The GlobalJsonReader was using File.OpenRead() feeding raw bytes into Utf8JsonStreamReader, which only strips the UTF-8 BOM — silently failing on UTF-16 encoded files.

Changes

SdkDirectoryWorkloadManifestProvider.GlobalJsonReader.cs: Opens the file with StreamReader (detectEncodingFromByteOrderMarks: true), calls Peek() to trigger BOM detection, then checks CurrentEncoding to select the appropriate read path:
- UTF-8 / no BOM (common case): seeks BaseStream back to position 0 and streams directly into Utf8JsonStreamReader — no memory allocation. Utf8JsonStreamReader handles the UTF-8 BOM itself.
- Other encodings / UTF-16 (rare case): calls StreamReader.ReadToEnd() to transcode the content to a UTF-8 MemoryStream, which is then passed to Utf8JsonStreamReader. This is acceptable given how small global.json files are.
The JSON parsing logic was extracted into a ParseGlobalJson helper to avoid duplicating the ~50 lines of JSON walking code across both paths.
SdkDirectoryWorkloadManifestProviderTests.cs: Added ItUsesWorkloadSetFromGlobalJsonWithUtf16Encoding as a parameterized theory covering both UTF-16 LE and UTF-16 BE.

// Before — only handles UTF-8 BOM, always reads whole file
using var fileStream = File.OpenRead(globalJsonPath);
var reader = new Utf8JsonStreamReader(fileStream, readerOptions);

// After — uses StreamReader for encoding detection, streams for UTF-8 (common), transcodes in-memory for non-UTF-8 (rare)
using var streamReader = new StreamReader(globalJsonPath, detectEncodingFromByteOrderMarks: true);
streamReader.Peek(); // trigger BOM detection without consuming content

if (streamReader.CurrentEncoding is UTF8Encoding)
{
    streamReader.BaseStream.Seek(0, SeekOrigin.Begin);
    var reader = new Utf8JsonStreamReader(streamReader.BaseStream, readerOptions);
    return ParseGlobalJson(ref reader, out shouldUseWorkloadSets);
}
else
{
    var content = streamReader.ReadToEnd();
    using var memStream = new MemoryStream(Encoding.UTF8.GetBytes(content));
    var reader = new Utf8JsonStreamReader(memStream, readerOptions);
    return ParseGlobalJson(ref reader, out shouldUseWorkloadSets);
}

Original prompt

This section details on the original issue you should resolve

<issue_title>Support UTF-16 BOM for global.json</issue_title>
<issue_description>The deps.json, runtimeconfig.json files are build generated, but global.json is user generated. So this can
be encoded in UTF-16.

See https://github.com/dotnet/cli/issues/2159

@gkhanna79 I just ran into this issue with a global.json that has a UTF-16 BOM.

I'm testing .NET on a Windows Nano Server container, so I used PowerShell's Out-File cmdlet to create the global.json file, which appears to use UTF-16 by default.

I was able to work around the issue by removing the BOM by using [IO.File]::WriteAllText. I also verified that using a UTF-8 BOM is another workaround, e.g. by using Out-File -Encoding utf8.

I'm wondering if it would be worth fixing this issue for UTF-16 in addition to UTF-8, because I think people might run into it, especially because the error message does not make it obvious what the problem is.

cc: @dleeapho @MichaelSimons

Actual Behavior

Supports UTF-8 BOM

Expected Behavior

Support UTF-16 BOM
</issue_description>

<agent_instructions>Either use the BCL to probe for encodings on the file to decide if it's safe to read directly as UTF8, or use a slower-but-more-compatible System.Text.Json serialization method to read the global json file contents.</agent_instructions>

Comments on the Issue (you are @copilot in this section)
@baronfel This might be cauesd by [https://github.com/baronfel/sdk/blob/3fe1e18d05b0b40e644a00c83eef4b97833c83b3/src/Resolvers/Microsoft.NET.Sdk.WorkloadManifestReader/SdkDirectoryWorkloadManifestProvider.GlobalJsonReader.cs#L29] where one of our common global.json readers assumes UTF8-ed-ness. We can't do that, so we should make this pathway able to read files of many encodings.

Fixes Support UTF-16 BOM for global.json #12347

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>

…nual BOM probing Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>

Copilot

Pull request overview

Updates the workload manifest reader’s global.json parsing to correctly handle user-authored global.json files encoded with UTF-16 (with BOM), which previously could silently fail due to assuming UTF-8 bytes.

Changes:

Detects file encoding via StreamReader BOM detection and selects an efficient parsing path (direct stream for UTF-8; in-memory transcode for non-UTF-8).
Extracts the JSON-walking logic into a shared ParseGlobalJson helper to avoid duplication.
Adds a theory test covering UTF-16 LE and UTF-16 BE global.json encodings.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/Resolvers/Microsoft.NET.Sdk.WorkloadManifestReader/SdkDirectoryWorkloadManifestProvider.GlobalJsonReader.cs	Adds BOM-based encoding detection and UTF-16-compatible parsing while keeping the UTF-8 fast path.
test/Microsoft.NET.Sdk.WorkloadManifestReader.Tests/SdkDirectoryWorkloadManifestProviderTests.cs	Adds test coverage for UTF-16 LE/BE encoded `global.json` selecting workload sets correctly.

Initial plan

bf05920

Copilot AI assigned Copilot and baronfel Feb 24, 2026

Copilot started work on behalf of baronfel February 24, 2026 22:05 View session

Support UTF-16 BOM for global.json reading

435cdce

Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Support UTF-16 BOM for global.json handling~~ Support UTF-16 BOM in global.json Feb 24, 2026

Copilot finished work on behalf of baronfel February 24, 2026 22:16

Copilot AI requested a review from baronfel February 24, 2026 22:16

baronfel reviewed Feb 25, 2026

View reviewed changes

Comment thread ...soft.NET.Sdk.WorkloadManifestReader/SdkDirectoryWorkloadManifestProvider.GlobalJsonReader.cs Outdated

Copilot started work on behalf of baronfel February 25, 2026 15:15 View session

Use streaming for UTF-8 global.json; only read into memory for UTF-16

c5efb46

Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>

Copilot AI requested a review from baronfel February 25, 2026 15:28

Copilot finished work on behalf of baronfel February 25, 2026 15:28

baronfel reviewed Feb 25, 2026

View reviewed changes

Comment thread ...soft.NET.Sdk.WorkloadManifestReader/SdkDirectoryWorkloadManifestProvider.GlobalJsonReader.cs Outdated

Copilot started work on behalf of baronfel February 25, 2026 15:38 View session

Use StreamReader.CurrentEncoding for encoding detection instead of ma…

309ea32

…nual BOM probing Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>

Copilot AI requested a review from baronfel February 25, 2026 15:47

Copilot finished work on behalf of baronfel February 25, 2026 15:47

baronfel approved these changes Feb 25, 2026

View reviewed changes

baronfel requested a review from a team February 25, 2026 15:48

baronfel added the Area-Workloads label Feb 25, 2026

baronfel marked this pull request as ready for review February 25, 2026 17:06

Copilot AI review requested due to automatic review settings February 25, 2026 17:06

Copilot started reviewing on behalf of baronfel February 25, 2026 17:07 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

nagilson approved these changes Feb 25, 2026

View reviewed changes

MiYanni approved these changes Feb 25, 2026

View reviewed changes

Merge branch 'main' into copilot/support-utf16-bom-global-json

315f663

nagilson enabled auto-merge March 11, 2026 18:13

nagilson merged commit 372de0d into main Mar 11, 2026
27 checks passed

nagilson deleted the copilot/support-utf16-bom-global-json branch March 11, 2026 20:42

dotnet-maestro Bot mentioned this pull request Mar 12, 2026

[main] Source code updates from dotnet/sdk dotnet/dotnet#5402

Merged

mthalman mentioned this pull request Jun 25, 2026

[demo] Support UTF-16 BOM in global.json (repro of dotnet/sdk#53141) mthalman/sdk#6

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support UTF-16 BOM in global.json - #53141

Support UTF-16 BOM in global.json#53141
nagilson merged 5 commits into
maindotnet/sdk:mainfrom
copilot/support-utf16-bom-global-jsondotnet/sdk:copilot/support-utf16-bom-global-jsonCopy head branch name to clipboard

Copilot AI commented Feb 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Search code, repositories, users, issues, pull requests...

Uh oh!

Conversation

Copilot AI commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Actual Behavior

Expected Behavior

Comments on the Issue (you are @copilot in this section)

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Copilot AI commented Feb 24, 2026 •

edited

Loading