Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Add Compliance tests for GB18030-2022 - #118075

#118075
Open
jozkee wants to merge 23 commits into
dotnet:maindotnet/runtime:mainfrom
jozkee:compliance_gb18030jozkee/runtime:compliance_gb18030Copy head branch name to clipboard
Open

Add Compliance tests for GB18030-2022#118075
jozkee wants to merge 23 commits into
dotnet:maindotnet/runtime:mainfrom
jozkee:compliance_gb18030jozkee/runtime:compliance_gb18030Copy head branch name to clipboard

Conversation

@jozkee

@jozkee jozkee commented Jul 25, 2025

Copy link
Copy Markdown
Member

No description provided.

@jozkee
jozkee requested a review from tarekgh July 25, 2025 22:41
@jozkee jozkee self-assigned this Jul 25, 2025
Copilot AI review requested due to automatic review settings July 25, 2025 22:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces comprehensive compliance testing for the GB18030-2022 Chinese character encoding standard by adding a new test project with thorough test coverage across multiple .NET API surfaces.

Key changes:

  • Creates a complete test suite for GB18030 encoding compliance validation
  • Implements tests for string operations, file I/O, directory operations, console I/O, and encoding roundtrips
  • Adds test data file with Chinese character test cases for both short and extended length scenarios

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file
File Description
Compliance.Tests.csproj New test project configuration with GB18030 test files and dependencies
TestHelper.cs Central test helper providing GB18030 encoding, test data parsing, and culture/comparison configurations
StringTests.cs Comprehensive string operation tests including comparison, search, equality, and manipulation methods
FileTests.cs File I/O operation tests for reading/writing text with GB18030 encoding
FileTestBase.cs Abstract base class for file system operation tests with GB18030 filenames
FileInfoTests.cs FileInfo-specific implementation of file system tests
DirectoryTests.cs Directory operation tests using GB18030 directory names
DirectoryTestBase.cs Abstract base class for directory operation tests with nested directory support
DirectoryInfoTests.cs DirectoryInfo-specific implementation with enumeration tests
ConsoleTests.cs Console I/O tests for standard input/output/error streams with GB18030 encoding
EncodingTests.cs Basic encoding roundtrip validation tests
Level3+Amendment_Test_Data_for_Mid_to_High_Volume_cases.txt Test data file containing GB18030 character sequences
Common.Tests.slnx Solution file update to include the new compliance test project
Comments suppressed due to low confidence (1)

src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs:13

  • [nitpick] The constant name 'Dummy' is ambiguous and doesn't clearly indicate its purpose as a replacement character for testing. Consider renaming to 'ReplacementChar' or 'TestReplacementCharacter'.
    private const string Dummy = "\uFFFF";

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs
Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs
Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/Tests/ConsoleTests.cs Outdated
Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs Outdated
Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs Outdated
Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs Outdated
Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs Outdated
@tarekgh

tarekgh commented Jul 28, 2025

Copy link
Copy Markdown
Member

@jozkee I'll wait your changes specifically in the string tests and then will take another look then.

Also, it would be good if you could add a specific net core app test that tests Char, CharUnicodeInfo, and Regex with GB18030 too.

@jozkee

jozkee commented Aug 8, 2025

Copy link
Copy Markdown
Member Author

CharUnicodeInfo

@tarekgh This test looks like is matching all codepoints with their category in UnicodeData.txt, does that suffice?

@tarekgh

tarekgh commented Aug 8, 2025

Copy link
Copy Markdown
Member

This test looks like is matching all codepoints with their category in UnicodeData.txt, does that suffice?

This test currently verifies whatever data is taken from the Unicode file UnicodeData.txt. If we happened to use an older version that doesn’t include the GB18030 characters, the test would still pass. While this is unlikely, I suggest adding checks in your new tests for some of the newly added GB18030 characters. That way, regardless of the Unicode version in use, we can be confident that GB18030 is covered.

Commit aa37d99 accidentally converted src/native/external/libunwind/README.md
from a symlink (git mode 120000, pointing at README) into a regular file with
mode 100644. The blob content was unchanged (a symlink's content is its target
path, "README"), so the change was easy to miss, but the mode flip is unrelated
to the GB18030 compliance tests and diverges from upstream. Restore the symlink.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs
Copilot AI review requested due to automatic review settings July 26, 2026 05:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:127

  • The path-segment splitting logic can yield segments that still exceed MaxPathSegmentName. The check runs before appending the next text element, so it only detects overflow one iteration late and can add an already-too-long segment to the results, potentially causing path-related tests to fail on some inputs/filesystems.
                if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
                {
                    result.Add(current);
                    current = string.Empty;
                }

Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs
@dotnet dotnet deleted a comment from azure-pipelines Bot Jul 26, 2026
@jozkee
jozkee removed request for a team, AaronRobinsonMSFT and janvorli July 26, 2026 18:09
Normalize the embedded test-data resource to LF at read time and parse
records against literal LF delimiters instead of Environment.NewLine, so
results are identical regardless of the OS that built or runs the tests.
This fixes the WASM legs, where a Windows-built (CRLF) resource was parsed
on mono/WASM (LF), throwing from TestHelper's static constructor and
failing every test. Use the char[] Split overload so it also compiles on
.NET Framework.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 744f142e-00de-4b24-a591-1646cbd71a11
Copilot AI review requested due to automatic review settings July 26, 2026 18:50
…ce_CultureInfo

The browser's ICU collation shard treats '0' (U+0030) and the ideographic
zero '\u3007' as equal even under zh-CN, unlike full ICU where zh-CN keeps
them distinct. Relax the existing non-zh-CN special case to also apply on
browser so Replace_CultureInfo doesn't fail on the WASM legs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 744f142e-00de-4b24-a591-1646cbd71a11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs:51

  • This test uses string.Contains(string, StringComparison), which isn't available when compiling for the .NET Framework target in this multi-targeted test project. The [SkipOnTargetFramework] attribute doesn't prevent compilation. Consider compiling this test only for NETCOREAPP.
    public static IEnumerable<object[]> Contains_MemberData() =>
        TestHelper.DecodedTestData.SelectMany(testData =>
        TestHelper.NonOrdinalStringComparisons.Select(comparison => new object[] { testData, comparison  }));

    [Theory]
    [SkipOnTargetFramework(TargetFrameworkMonikers.NetFramework)]
    [MemberData(nameof(Contains_MemberData))]
    public void Contains(string decoded, StringComparison comparison)

src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:145

  • NonExceedingPathNameMaxDecodedTestData only splits after current is already over the byte limit. That can yield path segments longer than MaxPathSegmentName (the check should consider the next element before appending).
            foreach (string element in GetTextElements(data))
            {
                if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
                {
                    result.Add(current);
                    current = string.Empty;
                }
                current += element;
            }
            result.Add(current);

Comment thread src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs
Copilot AI review requested due to automatic review settings July 26, 2026 18:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:140

  • The path segment splitting logic checks GetByteCount(current) > MaxPathSegmentName before appending the next text element, which can yield segments that exceed the max (and can also add an empty segment when the first element pushes it over). It should check whether adding the next element would exceed the limit, then flush the current segment before appending.
            foreach (string element in GetTextElements(data))
            {
                if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
                {
                    result.Add(current);

src/libraries/Common/tests/ComplianceTests/GB18030/Tests/DirectoryTestBase.cs:48

  • Path.Combine(Enumerable.Repeat(...).ToArray()) can produce string.Empty (or potentially throw on older implementations) when recurseLevel is 0, making this behavior depend on framework details. Since the test explicitly asserts the 0-level case equals firstPath, build the nested path deterministically without relying on Path.Combine with an empty array.
    {
        string firstPath = Path.Combine(TempDirectory.FullName, gb18030Line);
        string nestedDirPath = Path.Combine(firstPath, Path.Combine(Enumerable.Repeat(gb18030Line, recurseLevel).ToArray()));
        Assert.True(recurseLevel > 0 || firstPath.Equals(nestedDirPath));

Comment on lines +86 to +90
[Theory]
[MemberData(nameof(NamedBlock_MemberData))]
public async Task NamedBlock_InclusionAsync(string[]characters, RegexNamedBlock namedBlock, RegexEngine engine, CultureInfo culture)
{
Regex r = await RegexHelpers.GetRegexAsync(engine, $@"\p{{{namedBlock}}}", RegexOptions.None, culture);
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 744f142e-00de-4b24-a591-1646cbd71a11
Copilot AI review requested due to automatic review settings July 26, 2026 19:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs:24

  • The fixed (sbyte* p = (sbyte[])(object)encoded) cast will throw InvalidCastException at runtime (a byte[] can't be cast to sbyte[]). You can pin the byte[] and cast the pointer to sbyte* for the string(sbyte*, ...) constructor.
    public unsafe void Ctor(byte[] encoded)
    {
        fixed (sbyte* p = (sbyte[])(object)encoded)
        {
            string s = new string(p, 0, encoded.Length, TestHelper.GB18030Encoding);

src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:142

  • This logic can emit path segments that exceed MaxPathSegmentName: the byte-count check happens before appending the next text element, so current can grow beyond the limit and only be split on the following iteration (after it's already too large). This can make the filesystem tests fail when creating a segment longer than the intended cap.
            foreach (string element in GetTextElements(data))
            {
                if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
                {
                    result.Add(current);

src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexGeneratorHelper.netcoreapp.cs:50

  • Using the null-forgiving operator here will turn a missing directory name into a later ArgumentNullException/NullReferenceException without context. Since this is test infrastructure, it's better to throw an explicit exception if the directory can't be determined.
                MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
                MetadataReference.CreateFromFile(Path.Combine(Path.GetDirectoryName(corelibPath)!, "System.Runtime.dll")),
                MetadataReference.CreateFromFile(typeof(Unsafe).Assembly.Location),

src/libraries/System.Runtime/tests/System.Globalization.Tests/System/Globalization/CharUnicodeInfoTestData.cs:17

  • The null-forgiving operator will cause a later null deref if the embedded resource name changes or isn't embedded correctly. Throwing a descriptive exception here makes failures much easier to diagnose.
            string fileName = "UnicodeData.txt";
            Stream stream = typeof(CharUnicodeInfoTestData).GetTypeInfo().Assembly.GetManifestResourceStream(fileName)!;
            using (StreamReader reader = new StreamReader(stream))

src/libraries/Common/tests/ComplianceTests/GB18030/Tests/RegexTests.cs:89

  • Minor formatting: missing space between the array type and parameter name (string[]characters) reduces readability and doesn't match typical style in nearby code.
    [MemberData(nameof(NamedBlock_MemberData))]
    public async Task NamedBlock_InclusionAsync(string[] characters, RegexNamedBlock namedBlock, RegexEngine engine, CultureInfo culture)
    {

Copilot AI review requested due to automatic review settings July 26, 2026 19:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:148

  • NonExceedingPathNameMaxDecodedTestData splitting is incorrect: it checks the byte count of the current segment after it already exceeded the limit, so it can emit path segments whose encoded byte length is > MaxPathSegmentName. That defeats the purpose of the helper and can still produce overlong path components on some platforms.
            List<string> result = new();
            string current = string.Empty;
            foreach (string element in GetTextElements(data))
            {
                if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
                {
                    result.Add(current);
                    current = string.Empty;
                }
                current += element;
            }
            result.Add(current);
            return result;

Comment on lines +69 to +71
File.AppendAllText(tempFile, s_expectedText, TestHelper.GB18030Encoding);

byte[] expected = TestHelper.GB18030Encoding.GetBytes(initialContent + s_expectedText);
Comment on lines +9 to +10
#pragma warning disable xUnit2009 // Do not use boolean check to check for substrings
#pragma warning disable xUnit2010 // Do not use boolean check to check for string equality
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.