Add Compliance tests for GB18030-2022 - #118075
#118075Add Compliance tests for GB18030-2022#118075jozkee wants to merge 23 commits into
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces comprehensive compliance testing for the GB18030-2022 Chinese character encoding standard by adding a new test project with thorough test coverage across multiple .NET API surfaces.
Key changes:
- Creates a complete test suite for GB18030 encoding compliance validation
- Implements tests for string operations, file I/O, directory operations, console I/O, and encoding roundtrips
- Adds test data file with Chinese character test cases for both short and extended length scenarios
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| Compliance.Tests.csproj | New test project configuration with GB18030 test files and dependencies |
| TestHelper.cs | Central test helper providing GB18030 encoding, test data parsing, and culture/comparison configurations |
| StringTests.cs | Comprehensive string operation tests including comparison, search, equality, and manipulation methods |
| FileTests.cs | File I/O operation tests for reading/writing text with GB18030 encoding |
| FileTestBase.cs | Abstract base class for file system operation tests with GB18030 filenames |
| FileInfoTests.cs | FileInfo-specific implementation of file system tests |
| DirectoryTests.cs | Directory operation tests using GB18030 directory names |
| DirectoryTestBase.cs | Abstract base class for directory operation tests with nested directory support |
| DirectoryInfoTests.cs | DirectoryInfo-specific implementation with enumeration tests |
| ConsoleTests.cs | Console I/O tests for standard input/output/error streams with GB18030 encoding |
| EncodingTests.cs | Basic encoding roundtrip validation tests |
| Level3+Amendment_Test_Data_for_Mid_to_High_Volume_cases.txt | Test data file containing GB18030 character sequences |
| Common.Tests.slnx | Solution file update to include the new compliance test project |
Comments suppressed due to low confidence (1)
src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs:13
- [nitpick] The constant name 'Dummy' is ambiguous and doesn't clearly indicate its purpose as a replacement character for testing. Consider renaming to 'ReplacementChar' or 'TestReplacementCharacter'.
private const string Dummy = "\uFFFF";
|
Tagging subscribers to this area: @dotnet/area-meta |
|
@jozkee I'll wait your changes specifically in the string tests and then will take another look then. Also, it would be good if you could add a specific net core app test that tests |
@tarekgh This test looks like is matching all codepoints with their category in UnicodeData.txt, does that suffice? |
This test currently verifies whatever data is taken from the Unicode file UnicodeData.txt. If we happened to use an older version that doesn’t include the GB18030 characters, the test would still pass. While this is unlikely, I suggest adding checks in your new tests for some of the newly added GB18030 characters. That way, regardless of the Unicode version in use, we can be confident that GB18030 is covered. |
Commit aa37d99 accidentally converted src/native/external/libunwind/README.md from a symlink (git mode 120000, pointing at README) into a regular file with mode 100644. The blob content was unchanged (a symlink's content is its target path, "README"), so the change was easy to miss, but the mode flip is unrelated to the GB18030 compliance tests and diverges from upstream. Restore the symlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:127
- The path-segment splitting logic can yield segments that still exceed MaxPathSegmentName. The check runs before appending the next text element, so it only detects overflow one iteration late and can add an already-too-long segment to the results, potentially causing path-related tests to fail on some inputs/filesystems.
if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
{
result.Add(current);
current = string.Empty;
}
Normalize the embedded test-data resource to LF at read time and parse records against literal LF delimiters instead of Environment.NewLine, so results are identical regardless of the OS that built or runs the tests. This fixes the WASM legs, where a Windows-built (CRLF) resource was parsed on mono/WASM (LF), throwing from TestHelper's static constructor and failing every test. Use the char[] Split overload so it also compiles on .NET Framework. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 744f142e-00de-4b24-a591-1646cbd71a11
…ce_CultureInfo The browser's ICU collation shard treats '0' (U+0030) and the ideographic zero '\u3007' as equal even under zh-CN, unlike full ICU where zh-CN keeps them distinct. Relax the existing non-zh-CN special case to also apply on browser so Replace_CultureInfo doesn't fail on the WASM legs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 744f142e-00de-4b24-a591-1646cbd71a11
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs:51
- This test uses
string.Contains(string, StringComparison), which isn't available when compiling for the .NET Framework target in this multi-targeted test project. The[SkipOnTargetFramework]attribute doesn't prevent compilation. Consider compiling this test only for NETCOREAPP.
public static IEnumerable<object[]> Contains_MemberData() =>
TestHelper.DecodedTestData.SelectMany(testData =>
TestHelper.NonOrdinalStringComparisons.Select(comparison => new object[] { testData, comparison }));
[Theory]
[SkipOnTargetFramework(TargetFrameworkMonikers.NetFramework)]
[MemberData(nameof(Contains_MemberData))]
public void Contains(string decoded, StringComparison comparison)
src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:145
NonExceedingPathNameMaxDecodedTestDataonly splits aftercurrentis already over the byte limit. That can yield path segments longer thanMaxPathSegmentName(the check should consider the next element before appending).
foreach (string element in GetTextElements(data))
{
if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
{
result.Add(current);
current = string.Empty;
}
current += element;
}
result.Add(current);
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:140
- The path segment splitting logic checks
GetByteCount(current) > MaxPathSegmentNamebefore appending the next text element, which can yield segments that exceed the max (and can also add an empty segment when the first element pushes it over). It should check whether adding the next element would exceed the limit, then flush the current segment before appending.
foreach (string element in GetTextElements(data))
{
if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
{
result.Add(current);
src/libraries/Common/tests/ComplianceTests/GB18030/Tests/DirectoryTestBase.cs:48
Path.Combine(Enumerable.Repeat(...).ToArray())can producestring.Empty(or potentially throw on older implementations) whenrecurseLevelis 0, making this behavior depend on framework details. Since the test explicitly asserts the 0-level case equalsfirstPath, build the nested path deterministically without relying onPath.Combinewith an empty array.
{
string firstPath = Path.Combine(TempDirectory.FullName, gb18030Line);
string nestedDirPath = Path.Combine(firstPath, Path.Combine(Enumerable.Repeat(gb18030Line, recurseLevel).ToArray()));
Assert.True(recurseLevel > 0 || firstPath.Equals(nestedDirPath));
| [Theory] | ||
| [MemberData(nameof(NamedBlock_MemberData))] | ||
| public async Task NamedBlock_InclusionAsync(string[]characters, RegexNamedBlock namedBlock, RegexEngine engine, CultureInfo culture) | ||
| { | ||
| Regex r = await RegexHelpers.GetRegexAsync(engine, $@"\p{{{namedBlock}}}", RegexOptions.None, culture); |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 744f142e-00de-4b24-a591-1646cbd71a11
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (5)
src/libraries/Common/tests/ComplianceTests/GB18030/Tests/StringTests.cs:24
- The
fixed (sbyte* p = (sbyte[])(object)encoded)cast will throwInvalidCastExceptionat runtime (abyte[]can't be cast tosbyte[]). You can pin thebyte[]and cast the pointer tosbyte*for thestring(sbyte*, ...)constructor.
public unsafe void Ctor(byte[] encoded)
{
fixed (sbyte* p = (sbyte[])(object)encoded)
{
string s = new string(p, 0, encoded.Length, TestHelper.GB18030Encoding);
src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:142
- This logic can emit path segments that exceed
MaxPathSegmentName: the byte-count check happens before appending the next text element, socurrentcan grow beyond the limit and only be split on the following iteration (after it's already too large). This can make the filesystem tests fail when creating a segment longer than the intended cap.
foreach (string element in GetTextElements(data))
{
if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
{
result.Add(current);
src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexGeneratorHelper.netcoreapp.cs:50
- Using the null-forgiving operator here will turn a missing directory name into a later
ArgumentNullException/NullReferenceExceptionwithout context. Since this is test infrastructure, it's better to throw an explicit exception if the directory can't be determined.
MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
MetadataReference.CreateFromFile(Path.Combine(Path.GetDirectoryName(corelibPath)!, "System.Runtime.dll")),
MetadataReference.CreateFromFile(typeof(Unsafe).Assembly.Location),
src/libraries/System.Runtime/tests/System.Globalization.Tests/System/Globalization/CharUnicodeInfoTestData.cs:17
- The null-forgiving operator will cause a later null deref if the embedded resource name changes or isn't embedded correctly. Throwing a descriptive exception here makes failures much easier to diagnose.
string fileName = "UnicodeData.txt";
Stream stream = typeof(CharUnicodeInfoTestData).GetTypeInfo().Assembly.GetManifestResourceStream(fileName)!;
using (StreamReader reader = new StreamReader(stream))
src/libraries/Common/tests/ComplianceTests/GB18030/Tests/RegexTests.cs:89
- Minor formatting: missing space between the array type and parameter name (
string[]characters) reduces readability and doesn't match typical style in nearby code.
[MemberData(nameof(NamedBlock_MemberData))]
public async Task NamedBlock_InclusionAsync(string[] characters, RegexNamedBlock namedBlock, RegexEngine engine, CultureInfo culture)
{
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
src/libraries/Common/tests/ComplianceTests/GB18030/TestHelper.cs:148
- NonExceedingPathNameMaxDecodedTestData splitting is incorrect: it checks the byte count of the current segment after it already exceeded the limit, so it can emit path segments whose encoded byte length is > MaxPathSegmentName. That defeats the purpose of the helper and can still produce overlong path components on some platforms.
List<string> result = new();
string current = string.Empty;
foreach (string element in GetTextElements(data))
{
if (fileSystemEncoding.GetByteCount(current) > MaxPathSegmentName)
{
result.Add(current);
current = string.Empty;
}
current += element;
}
result.Add(current);
return result;
| File.AppendAllText(tempFile, s_expectedText, TestHelper.GB18030Encoding); | ||
|
|
||
| byte[] expected = TestHelper.GB18030Encoding.GetBytes(initialContent + s_expectedText); |
| #pragma warning disable xUnit2009 // Do not use boolean check to check for substrings | ||
| #pragma warning disable xUnit2010 // Do not use boolean check to check for string equality |
No description provided.