Skip to content

Navigation Menu

Sign in
Appearance settings
Sign up
Appearance settings

Support non-ASCII identifiers following UAX #31 - #6968

#6968
Open
Otzie2023 wants to merge 4 commits into
openscad:masteropenscad/openscad:masterfrom
Otzie2023:feat/unicode-identifiersOtzie2023/openscad:feat/unicode-identifiersCopy head branch name to clipboard
Open

Support non-ASCII identifiers following UAX #31#6968
Otzie2023 wants to merge 4 commits into
openscad:masteropenscad/openscad:masterfrom
Otzie2023:feat/unicode-identifiersOtzie2023/openscad:feat/unicode-identifiersCopy head branch name to clipboard

Conversation

@Otzie2023

Copy link
Copy Markdown

Support non-ASCII identifiers following UAX #31

Fixes #3736. Closes #737 as a side effect (see below).

Identifiers are currently limited to [A-Za-z_$][A-Za-z0-9_]*; anything else is
rejected by the lexer without a message. This adds an opt-in identifier syntax
based on Unicode Standard Annex #31, behind --enable=unicode-identifiers.

The character set

ID_Start    := (XID_Start + { U+0024, U+005F }) & Identifier_Status=Allowed
ID_Continue := XID_Continue & Identifier_Status=Allowed

XID_Start/XID_Continue is the UAX #31 default, and the same grammar C++23
(P1949R7) and Rust 1.53 use. U+0024 DOLLAR SIGN has to be added explicitly
because it is Sc, not XID, and OpenSCAD needs it for $fn, $fa and the
rest; U+005F LOW LINE is added for compatibility with the current syntax.

The intersection with Identifier_Status=Allowed, the general security profile
of UTS #39, is the part that goes beyond C++ and Rust's grammar. It addresses
the objection raised in this issue directly: plain XID does not keep invisible
characters out, because variation selectors U+FE00–FE0F are Mn and therefore
in XID_Continue. The UTS #39 profile removes them, along with obsolete
scripts, technical symbols, and canonical duplicates.

Emoji, ZWJ/ZWNJ, and mathematical operators (∑ ∏ ∫ ∂ ∇ √ ∞ °) are outside the
set. The Greek letters people usually reach for — π, Δ, Σ, Ω, θ, λ, φ — are in
XID_Start already, so most of the "special math symbols" wish from the 2021
comment is covered without a bespoke allowlist.

Normalisation

Identifiers are normalised to NFC in the lexer, and validated after
normalisation. P1949 instead makes non-NFC source ill-formed; normalising seemed
the better fit here, because ä as U+00E4 and as U+0061 U+0308 are
indistinguishable in the editor and in a diff, and most OpenSCAD files are
edited in the bundled editor rather than in a toolchain that can enforce NFC on
save. Rust made the same call.

Validating the normalised form also accepts the canonical singletons: U+2126
OHM SIGN, U+212A KELVIN SIGN and U+212B ANGSTROM SIGN are excluded from the UTS
#39 profile precisely because NFC folds them into U+03A9, U+004B and U+00C5, so
after normalisation they are ordinary identifier characters and Ω typed either
way is one variable. Roughly 1,000 code points fall into that category,
including the composition exclusions in Devanagari, Bengali and Gurmukhi that
several input methods emit by default.

The generator verifies that the profile is closed under NFC before writing the
tables, so normalising an accepted identifier can never turn it into a rejected
one.

g_utf8_normalize() comes from glib, which is already a required dependency.

The lexer tables

The 2021 discussion stalled on the scanner tables blowing up. That happens when
the code point ranges are encoded as UTF-8 byte patterns in the rules — Unicode
16 has 767 XID_Start and 1400 XID_Continue ranges, which expand to hundreds
of byte alternatives each. Measured against this tree with flex 2.6.4:

NFA states DFA states Table entries
current lexer 437 198 1,755
this PR 479 212 2,073
XID encoded in the rules 12,502 1,510 62,843

The last row does not compile with flex's defaults at all (Definition value for {XIDS} too long), and after splitting the pattern across 61 definitions it uses
12,502 of the 13,000 default NFA slots.

This PR keeps Unicode out of the DFA. The rules match any non-ASCII sequence
using a variant of the {UNICODE} macro that was already in lexer.l, and the
classification happens in the action as a binary search over a generated table:

U2ID    \xc2[\x80-\x9f\xa1-\xbf]|[\xc3-\xdf]{U}
U3ID    [\xe0-\xee]{U}{U}|\xef[\x80-\xba\xbc-\xbf]{U}|\xef\xbb[\x80-\xbe]
UNICODEID {U2ID}|{U3ID}|{U4}{U}{U}{U}

The exclusions carved out of {UNICODE} are U+00A0 and U+FEFF, which lexer.l
treats as whitespace; without that, a<NBSP>= 1 would lex as a single token and
nbsp-utf8-test.scad would break.

The ASCII rule {IDSTART}{IDREST}* is kept unchanged and placed before the new
one. Both match a pure-ASCII identifier equally long, so flex resolves the tie
in favour of the earlier rule and ASCII input never reaches the new code path —
no runtime check needed. 0x1F, 1a and the deprecated digit-leading form are
unaffected.

Because {UNICODEID} is a coarse approximation, it also accepts overlong forms,
surrogates and values above U+10FFFF; the action validates the encoding with
g_utf8_validate() before normalising. Malformed input now reports why it was
rejected instead of failing silently, which is an improvement even for people
who never write a non-ASCII identifier.

The blanket {UNICODE} reject rule in INITIAL becomes unreachable and is
removed — flex confirms this with rule cannot be matched if it is left in.

Tables

src/core/UnicodeIdentifierTables.h is generated and checked in, the way GCC
does with ucnid.h, so there is no build-time dependency on the UCD.
scripts/generate-unicode-identifier-tables.py regenerates it from
DerivedCoreProperties.txt and IdentifierStatus.txt; it refuses to run if the
two files are from different Unicode versions. Current tables: Unicode 16.0.0,
305 start ranges and 380 continue ranges, 5,480 bytes. ASCII is deliberately not
tabulated.

Editor highlighting

src/gui/ScadLexer.cc needs a matching change. The editor uses a separate
lexertl-based lexer whose identifier rules are ASCII-only, so a non-ASCII byte
splits an identifier into several tokens with different styles:

VARIABLE   <geh>   TEXT <\xc3> TEXT <\xa4>  VARIABLE <use>
SPECIALVAR <$wandst> TEXT <\xc3> TEXT <\xa4> VARIABLE <rke>

The second line is the visible one: the tail of $wandstärke falls back from
the special-variable colour to the ordinary variable colour. The rules now
accept any byte outside ASCII:

rules_.push("[a-zA-Z0-9_\\x80-\\xff]+", evariable);
rules_.push("[$][a-zA-Z0-9_\\x80-\\xff]+", especialVariable);

That is deliberately coarser than the grammar in lexer.l — the editor lexer
only drives highlighting, and the parser reports the code points that are not
actually allowed. One cosmetic consequence: a<NBSP>= 1 now highlights as a
single identifier token, though it still parses as two.

Other code paths

I traced the other places that produce identifier names:

  • -D/--D is appended to the source text (openscad.cc) and goes through the
    normal lexer, so it is covered.
  • Customizer parameter names come from assignment->getName(), i.e. from the
    AST, so they are covered too.
  • customizer/comment_lexer.l only lexes annotation bodies; the parameter name
    is matched by source location, so it needs no change.
  • ParameterSet::readFile() uses parameter names as JSON keys and could see a
    non-NFC key from an older file or an external tool. Not addressed here, since
    it cannot happen until non-ASCII identifiers are actually in use — happy to
    add it to this PR if you'd prefer.

Identifiers are plain std::string through Lookup, Assignment and
Context::lookup_variable, so normalising at lex time means nothing downstream
changes.

Issue #737

µm = 0.001 * mm; from 2014 now produces
Parser error: Identifier cannot start with U+00B5 instead of failing silently.
U+00B5 MICRO SIGN is Restricted in UTS #39 because it is confusable with
U+03BC GREEK SMALL LETTER MU and NFC does not fold the two. μm with the Greek
letter works. This is also the worked example in Rust's uncommon_codepoints
lint.

Not included

  • Confusable and mixed-script detection (UTS Infobox #39 §4/§5). Worth having eventually
    as a warning, along the lines of Rust's confusable_idents and
    mixed_script_confusables, but it is separable and should not gate the
    grammar change.
  • The deprecated digit-leading identifier form stays ASCII-only.

Feature flag

Everything is behind Feature::ExperimentalUnicodeIdentifiers. With the flag
off, the only change in behaviour is that a non-ASCII identifier now says
Non-ASCII identifiers are experimental, enable them with --enable=unicode-identifiers instead of failing without a message. Dropping the
gate later is a one-line change.

Tests

  • src/core/UnicodeIdentifier_test.cc — 33 Catch2 assertions covering the
    accepted set, NFC folding in both directions, the canonical singletons, the
    rejected classes, and malformed UTF-8.
  • tests/data/scad/misc/unicode-identifiers.scad — echo test with
    --enable=unicode-identifiers, including a variable defined precomposed and
    read decomposed.
  • tests/data/scad/misc/unicode-identifiers-fail.scad — the #737 case, as an
    expected parse failure.

Built and run against this tree on Linux with GCC 15.2, Qt 6.10.2,
QScintilla 2.14.1, CGAL 6.1.1, Boost 1.90
Bildschirmfoto vom 2026-08-22 15-54-59

Otzie2023 and others added 2 commits August 22, 2026 15:42
Identifiers were limited to [A-Za-z_$][A-Za-z0-9_]*, and anything else was
rejected by the lexer without a message. This adds an opt-in identifier
syntax behind --enable=unicode-identifiers:

  ID_Start    := (XID_Start + { U+0024, U+005F }) & Identifier_Status=Allowed
  ID_Continue := XID_Continue & Identifier_Status=Allowed

XID_Start/XID_Continue is the UAX openscad#31 default that C++23 and Rust use. The
intersection with the UTS openscad#39 general security profile removes code points
that are invisible, obsolete, or canonical duplicates; plain XID does not
cover this, since variation selectors are Mn and therefore XID_Continue.

Identifiers are normalised to NFC before they are validated, so that
canonically equivalent spellings name the same variable. Validating the
normalised form also accepts the canonical singletons U+2126, U+212A and
U+212B, which the UTS openscad#39 profile excludes precisely because NFC folds them.

Encoding the code point ranges as UTF-8 byte patterns in the flex rules grows
the scanner tables by more than an order of magnitude, so the rules match any
non-ASCII sequence and the classification happens in the action, as a binary
search over a generated table. The tables are generated from the UCD by
scripts/generate-unicode-identifier-tables.py and checked in, so there is no
build-time dependency on the Unicode data files.

The editor's lexertl rules are widened to match, otherwise a non-ASCII byte
splits an identifier into differently styled tokens.

Fixes openscad#3736.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant

Morty Proxy This is a proxified and sanitized view of the page, visit original site.