bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

shihai1991 · Sep 12, 2020

encodings.normalize_encoding() now ignores non-ASCII characters.

https://bugs.python.org/issue39337

update test

Lib/test/test_source_encoding.py

This reverts commit 0fcafb8.

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst

Lib/test/test_source_encoding.py

* origin/master: (147 commits) Fix the attribute names in the docstring of GenericAlias (pythonGH-22594) bpo-39337: Add a test case for normalizing of codec names (pythonGH-19069) bpo-41557: Update Windows installer to use SQLite 3.33.0 (pythonGH-21960) bpo-41976: Fix the fallback to gcc of ctypes.util.find_library when using gcc>9 (pythonGH-22598) bpo-41306: Allow scale value to not be rounded (pythonGH-21715) bpo-41970: Avoid test failure in test_lib2to3 if the module is already imported (pythonGH-22595) bpo-41376: Fix the documentation of `site.getusersitepackages()` (pythonGH-21602) Revert "bpo-26680: Incorporate is_integer in all built-in and standard library numeric types (pythonGH-6121)" (pythonGH-22584) bpo-41923: PEP 613: Add TypeAlias to typing module (python#22532) Fix comment about PyObject_IsTrue. (pythonGH-22343) bpo-38605: Make 'from __future__ import annotations' the default (pythonGH-20434) bpo-41905: Add abc.update_abstractmethods() (pythonGH-22485) bpo-41944: No longer call eval() on content received via HTTP in the UnicodeNames tests (pythonGH-22575) bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (pythonGH-22566) Post 3.10.0a1 Python 3.10.0a1 bpo-41584: clarify when the reflected method of a binary arithemtic operator is called (python#22505) bpo-41939: Fix test_site.test_license_exists_at_url() (python#22559) bpo-41774: Tweak new programming FAQ entry (pythonGH-22562) bpo-41936. Remove macros Py_ALLOW_RECURSION/Py_END_ALLOW_RECURSION (pythonGH-22552) ...

Lib/test/test_codecs.py

vstinner · Oct 11, 2020

Lib/test/test_codecs.py

+        self.assertEqual(out, 'utf_8')
+        out = encodings.normalize_encoding('utf   8')
+        self.assertEqual(out, 'utf_8')
+        out = encodings.normalize_encoding('UTF 8')


Would you mind to add a comment to explain that the function does not convert upper case letters to lower case letters, just to make the purpose of this test even more explicit?

To be honest, I don't know how to exact explain it~
I found a case in https://github.com/python/cpython/blob/master/Lib/locale.py#L358.
Looks like It's fine to update encodings.normalize_encoding() to conver to lower-case.

just describe the fact, Lol~

Maybe we can enhance this encodings.normalize_encoding()? I am not sure~

Lib/test/test_codecs.py

vstinner · Oct 11, 2020

Lib/encodings/__init__.py

@@ -61,7 +61,8 @@ def normalize_encoding(encoding):
        if c.isalnum() or c == '.':
            if punct and chars:
                chars.append('_')
-            chars.append(c)
+            if c.isascii():
+                chars.append(c)


I wanted to ask you to add a ".. versionchanged:: 3.10" entry in the documentation, but then I noticed that the encodings module was never documented! Oh!

If end user will use this function or module, I can try to create the doc, but I need some time to do it :)

It can and must be addressed in a separated PR anymore. The lack of documentation should not hold this change.

ok, copy that.

Lib/test/test_codecs.py

vstinner

LGTM.

@taleinat @serhiy-storchaka or @malemburg: Do you want to double check this PR?

Lib/test/test_codecs.py

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst

Doc/whatsnew/3.10.rst

vstinner · Oct 14, 2020

Merged, thanks.

shihai1991 · Oct 14, 2020

thanks victor for your tons of comments ;)

…cters (pythonGH-22219)

encodings.normalize_encoding() should ignore non-ASCII letters

8ae6408

the-knights-who-say-ni added the CLA signed label Sep 12, 2020

bedevere-bot added the awaiting review label Sep 12, 2020

shihai1991 changed the title ~~encodings.normalize_encoding() should ignore non-ASCII letters~~ WIP: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020

update test

0fcafb8

update test

shihai1991 commented Sep 12, 2020

View reviewed changes

Lib/test/test_source_encoding.py Outdated Show resolved Hide resolved

shihai1991 changed the title ~~WIP: encodings.normalize_encoding() should ignore non-ASCII letters~~ [WIP]bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020

shihai1991 changed the title ~~[WIP]bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters~~ bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020

Revert "update test"

dea24d8

This reverts commit 0fcafb8.

shihai1991 mentioned this pull request Sep 13, 2020

bpo-37751: Document the change in What's New in Python 3.9 #17997

Closed

shihai1991 mentioned this pull request Sep 29, 2020

bpo-39337: Add a test case for normalizing of codec names #19069

Merged

taleinat requested a review from vstinner September 30, 2020 06:44

vstinner reviewed Oct 1, 2020

View reviewed changes

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst Outdated Show resolved Hide resolved

Lib/test/test_source_encoding.py Outdated Show resolved Hide resolved

apply victor's comment

dbb0062

shihai1991 changed the title ~~bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters~~ bpo-39337: encodings.normalize_encoding() now ignores non-ASCII letters Oct 3, 2020

vstinner reviewed Oct 8, 2020

View reviewed changes

Lib/test/test_source_encoding.py Outdated Show resolved Hide resolved

shihai1991 added 2 commits October 9, 2020 12:34

apply victor's comment

3fa221f

vstinner reviewed Oct 9, 2020

View reviewed changes

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

apply victor's comments

2e73d13

vstinner reviewed Oct 11, 2020

View reviewed changes

apply victor's comment

95c1d98

vstinner reviewed Oct 12, 2020

View reviewed changes

Lib/test/test_codecs.py Outdated Show resolved Hide resolved

apply victor's comment

03bfd9b

vstinner approved these changes Oct 12, 2020

View reviewed changes

Lib/test/test_codecs.py Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting review labels Oct 12, 2020

apply victor's comment

38f28bd

vstinner reviewed Oct 13, 2020

View reviewed changes

Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst Outdated Show resolved Hide resolved

apply victor's comment

5982784

vstinner reviewed Oct 13, 2020

View reviewed changes

Doc/whatsnew/3.10.rst Outdated Show resolved Hide resolved

apply victor's comment

4ecb8a1

shihai1991 changed the title ~~bpo-39337: encodings.normalize_encoding() now ignores non-ASCII letters~~ bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters Oct 13, 2020

vstinner merged commit c5b049b into python:master Oct 14, 2020

bedevere-bot removed the awaiting merge label Oct 14, 2020

xzy3 pushed a commit to xzy3/cpython that referenced this pull request Oct 18, 2020

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII chara…

bb71c6d

…cters (pythonGH-22219)

adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 13, 2021

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII chara…

5959a19

…cters (pythonGH-22219)

Search code, repositories, users, issues, pull requests...

Uh oh!

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

Uh oh!

Conversation

shihai1991 commented Sep 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner Oct 11, 2020

Choose a reason for hiding this comment

Uh oh!

shihai1991 Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner Oct 11, 2020

Choose a reason for hiding this comment

Uh oh!

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

vstinner Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

shihai1991 Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vstinner commented Oct 14, 2020

Uh oh!

shihai1991 commented Oct 14, 2020

Uh oh!

Uh oh!

shihai1991 commented Sep 12, 2020 •

edited

Loading

shihai1991 Oct 12, 2020 •

edited

Loading