Support non-ASCII identifiers following UAX #31 - #6968
#6968Open
Otzie2023 wants to merge 4 commits into
openscad:masteropenscad/openscad:masterfrom
Otzie2023:feat/unicode-identifiersOtzie2023/openscad:feat/unicode-identifiersCopy head branch name to clipboard
Open
Support non-ASCII identifiers following UAX #31#6968Otzie2023 wants to merge 4 commits intoopenscad:masteropenscad/openscad:masterfrom Otzie2023:feat/unicode-identifiersOtzie2023/openscad:feat/unicode-identifiersCopy head branch name to clipboard
Otzie2023 wants to merge 4 commits into
openscad:masteropenscad/openscad:masterfrom
Otzie2023:feat/unicode-identifiersOtzie2023/openscad:feat/unicode-identifiersCopy head branch name to clipboard
Conversation
Identifiers were limited to [A-Za-z_$][A-Za-z0-9_]*, and anything else was
rejected by the lexer without a message. This adds an opt-in identifier
syntax behind --enable=unicode-identifiers:
ID_Start := (XID_Start + { U+0024, U+005F }) & Identifier_Status=Allowed
ID_Continue := XID_Continue & Identifier_Status=Allowed
XID_Start/XID_Continue is the UAX openscad#31 default that C++23 and Rust use. The
intersection with the UTS openscad#39 general security profile removes code points
that are invisible, obsolete, or canonical duplicates; plain XID does not
cover this, since variation selectors are Mn and therefore XID_Continue.
Identifiers are normalised to NFC before they are validated, so that
canonically equivalent spellings name the same variable. Validating the
normalised form also accepts the canonical singletons U+2126, U+212A and
U+212B, which the UTS openscad#39 profile excludes precisely because NFC folds them.
Encoding the code point ranges as UTF-8 byte patterns in the flex rules grows
the scanner tables by more than an order of magnitude, so the rules match any
non-ASCII sequence and the classification happens in the action, as a binary
search over a generated table. The tables are generated from the UCD by
scripts/generate-unicode-identifier-tables.py and checked in, so there is no
build-time dependency on the Unicode data files.
The editor's lexertl rules are widened to match, otherwise a non-ASCII byte
splits an identifier into differently styled tokens.
Fixes openscad#3736.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support non-ASCII identifiers following UAX #31
Fixes #3736. Closes #737 as a side effect (see below).
Identifiers are currently limited to
[A-Za-z_$][A-Za-z0-9_]*; anything else isrejected by the lexer without a message. This adds an opt-in identifier syntax
based on Unicode Standard Annex #31, behind
--enable=unicode-identifiers.The character set
XID_Start/XID_Continueis the UAX #31 default, and the same grammar C++23(P1949R7) and Rust 1.53 use.
U+0024 DOLLAR SIGNhas to be added explicitlybecause it is
Sc, notXID, and OpenSCAD needs it for$fn,$faand therest;
U+005F LOW LINEis added for compatibility with the current syntax.The intersection with
Identifier_Status=Allowed, the general security profileof UTS #39, is the part that goes beyond C++ and Rust's grammar. It addresses
the objection raised in this issue directly: plain XID does not keep invisible
characters out, because variation selectors U+FE00–FE0F are
Mnand thereforein
XID_Continue. The UTS #39 profile removes them, along with obsoletescripts, technical symbols, and canonical duplicates.
Emoji, ZWJ/ZWNJ, and mathematical operators (∑ ∏ ∫ ∂ ∇ √ ∞ °) are outside the
set. The Greek letters people usually reach for — π, Δ, Σ, Ω, θ, λ, φ — are in
XID_Startalready, so most of the "special math symbols" wish from the 2021comment is covered without a bespoke allowlist.
Normalisation
Identifiers are normalised to NFC in the lexer, and validated after
normalisation. P1949 instead makes non-NFC source ill-formed; normalising seemed
the better fit here, because
äas U+00E4 and as U+0061 U+0308 areindistinguishable in the editor and in a diff, and most OpenSCAD files are
edited in the bundled editor rather than in a toolchain that can enforce NFC on
save. Rust made the same call.
Validating the normalised form also accepts the canonical singletons: U+2126
OHM SIGN, U+212A KELVIN SIGN and U+212B ANGSTROM SIGN are excluded from the UTS
#39 profile precisely because NFC folds them into U+03A9, U+004B and U+00C5, so
after normalisation they are ordinary identifier characters and
Ωtyped eitherway is one variable. Roughly 1,000 code points fall into that category,
including the composition exclusions in Devanagari, Bengali and Gurmukhi that
several input methods emit by default.
The generator verifies that the profile is closed under NFC before writing the
tables, so normalising an accepted identifier can never turn it into a rejected
one.
g_utf8_normalize()comes from glib, which is already a required dependency.The lexer tables
The 2021 discussion stalled on the scanner tables blowing up. That happens when
the code point ranges are encoded as UTF-8 byte patterns in the rules — Unicode
16 has 767
XID_Startand 1400XID_Continueranges, which expand to hundredsof byte alternatives each. Measured against this tree with flex 2.6.4:
The last row does not compile with flex's defaults at all (
Definition value for {XIDS} too long), and after splitting the pattern across 61 definitions it uses12,502 of the 13,000 default NFA slots.
This PR keeps Unicode out of the DFA. The rules match any non-ASCII sequence
using a variant of the
{UNICODE}macro that was already inlexer.l, and theclassification happens in the action as a binary search over a generated table:
The exclusions carved out of
{UNICODE}are U+00A0 and U+FEFF, whichlexer.ltreats as whitespace; without that,
a<NBSP>= 1would lex as a single token andnbsp-utf8-test.scadwould break.The ASCII rule
{IDSTART}{IDREST}*is kept unchanged and placed before the newone. Both match a pure-ASCII identifier equally long, so flex resolves the tie
in favour of the earlier rule and ASCII input never reaches the new code path —
no runtime check needed.
0x1F,1aand the deprecated digit-leading form areunaffected.
Because
{UNICODEID}is a coarse approximation, it also accepts overlong forms,surrogates and values above U+10FFFF; the action validates the encoding with
g_utf8_validate()before normalising. Malformed input now reports why it wasrejected instead of failing silently, which is an improvement even for people
who never write a non-ASCII identifier.
The blanket
{UNICODE}reject rule inINITIALbecomes unreachable and isremoved — flex confirms this with
rule cannot be matchedif it is left in.Tables
src/core/UnicodeIdentifierTables.his generated and checked in, the way GCCdoes with
ucnid.h, so there is no build-time dependency on the UCD.scripts/generate-unicode-identifier-tables.pyregenerates it fromDerivedCoreProperties.txtandIdentifierStatus.txt; it refuses to run if thetwo files are from different Unicode versions. Current tables: Unicode 16.0.0,
305 start ranges and 380 continue ranges, 5,480 bytes. ASCII is deliberately not
tabulated.
Editor highlighting
src/gui/ScadLexer.ccneeds a matching change. The editor uses a separatelexertl-based lexer whose identifier rules are ASCII-only, so a non-ASCII byte
splits an identifier into several tokens with different styles:
The second line is the visible one: the tail of
$wandstärkefalls back fromthe special-variable colour to the ordinary variable colour. The rules now
accept any byte outside ASCII:
That is deliberately coarser than the grammar in
lexer.l— the editor lexeronly drives highlighting, and the parser reports the code points that are not
actually allowed. One cosmetic consequence:
a<NBSP>= 1now highlights as asingle identifier token, though it still parses as two.
Other code paths
I traced the other places that produce identifier names:
-D/--Dis appended to the source text (openscad.cc) and goes through thenormal lexer, so it is covered.
assignment->getName(), i.e. from theAST, so they are covered too.
customizer/comment_lexer.lonly lexes annotation bodies; the parameter nameis matched by source location, so it needs no change.
ParameterSet::readFile()uses parameter names as JSON keys and could see anon-NFC key from an older file or an external tool. Not addressed here, since
it cannot happen until non-ASCII identifiers are actually in use — happy to
add it to this PR if you'd prefer.
Identifiers are plain
std::stringthroughLookup,AssignmentandContext::lookup_variable, so normalising at lex time means nothing downstreamchanges.
Issue #737
µm = 0.001 * mm;from 2014 now producesParser error: Identifier cannot start with U+00B5instead of failing silently.U+00B5 MICRO SIGN is
Restrictedin UTS #39 because it is confusable withU+03BC GREEK SMALL LETTER MU and NFC does not fold the two.
μmwith the Greekletter works. This is also the worked example in Rust's
uncommon_codepointslint.
Not included
as a warning, along the lines of Rust's
confusable_identsandmixed_script_confusables, but it is separable and should not gate thegrammar change.
Feature flag
Everything is behind
Feature::ExperimentalUnicodeIdentifiers. With the flagoff, the only change in behaviour is that a non-ASCII identifier now says
Non-ASCII identifiers are experimental, enable them with --enable=unicode-identifiersinstead of failing without a message. Dropping thegate later is a one-line change.
Tests
src/core/UnicodeIdentifier_test.cc— 33 Catch2 assertions covering theaccepted set, NFC folding in both directions, the canonical singletons, the
rejected classes, and malformed UTF-8.
tests/data/scad/misc/unicode-identifiers.scad— echo test with--enable=unicode-identifiers, including a variable defined precomposed andread decomposed.
tests/data/scad/misc/unicode-identifiers-fail.scad— the#737case, as anexpected parse failure.
Built and run against this tree on Linux with GCC 15.2, Qt 6.10.2,

QScintilla 2.14.1, CGAL 6.1.1, Boost 1.90