fix(text): preserve surrogate pairs at truncation boundaries#16352
fix(text): preserve surrogate pairs at truncation boundaries#163520xfullex merged 5 commits intoCherryHQ:mainCherryHQ/cherry-studio:mainfrom he-yufeng:fix/channels-split-surrogate-pairshe-yufeng/cherry-studio:fix/channels-split-surrogate-pairsCopy head branch name to clipboard
Conversation
DeJeune
left a comment
There was a problem hiding this comment.
LGTM. Correct, surgical fix for lone-surrogate emission when truncating outgoing messages. clampSurrogateBoundary is sound (a pair is exactly 2 code units, so one step-back always suffices), both raw-slice truncation sites (Discord/Slack editMessage) are covered, and the shared splitMessage hard-cut fallback protects the other four adapters. Tests match the implementation and cover the meaningful paths.
5c4c3f9 to
ca7ce10
Compare
eeee0717
left a comment
There was a problem hiding this comment.
LGTM. Correct and surgical fix for lone-surrogate emission on truncation boundaries.
Verified:
clampSurrogateBoundarylogic is sound — a surrogate pair is exactly 2 code units, so a single step-back always suffices; string-edge guards are correct.- All outgoing-message truncation sites are covered: the shared
splitMessagehard-cut fallback (protects all 6 adapters) plus the two streamingeditMessageraw-slice sites (Discord/Slack). The remaining.slice(0,...)hits in the channels dir are UUIDs / diagnostic error text, not chat payloads — correctly left untouched. - Truncated lengths stay within platform limits (Discord ≤2000, Slack ≤4000).
- Unit tests are meaningful and match the implementation: surrogate-pair fallback, unchanged whitespace path, and the helper's edge cases. The CJK case (no spaces → forced hard cut) hits exactly the fixed path.
- CI is green.
Minimal and well-scoped. Nice work.
e1971fc to
242a36a
Compare
…st-message title truncateText() derives a topic title with a hard slice(0, 50) when the first user message has no whitespace in its first 50 chars (common for CJK text). A code point above U+FFFF is two UTF-16 units, so a cut between them keeps the high surrogate and drops the low one, persisting a lone surrogate that renders as the replacement glyph in the sidebar until the user renames the topic by hand. Step the cut back one unit when the boundary falls between a surrogate pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for channel message truncation. Add a regression test (49 CJK chars + an emoji on the 50-char boundary) that fails on main and passes with this change. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
|
Thanks — Since the sibling #16404 (same author, same root cause: naive UTF-16 truncation on the CJK hard-cut fallback) needs the exact same logic for topic-title truncation and currently inlines a byte-identical copy of this boolean, the main thing I'd raise on this PR is where the shared fix should live:
Not blocking. |
…st-message title truncateText() derives a topic title with a hard slice(0, 50) when the first user message has no whitespace in its first 50 chars (common for CJK text). A code point above U+FFFF is two UTF-16 units, so a cut between them keeps the high surrogate and drops the low one, persisting a lone surrogate that renders as the replacement glyph in the sidebar until the user renames the topic by hand. Step the cut back one unit when the boundary falls between a surrogate pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for channel message truncation. Add a regression test (49 CJK chars + an emoji on the 50-char boundary) that fails on main and passes with this change. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
242a36a to
5888a45
Compare
…st-message title truncateText() derives a topic title with a hard slice(0, 50) when the first user message has no whitespace in its first 50 chars (common for CJK text). A code point above U+FFFF is two UTF-16 units, so a cut between them keeps the high surrogate and drops the low one, persisting a lone surrogate that renders as the replacement glyph in the sidebar until the user renames the topic by hand. Step the cut back one unit when the boundary falls between a surrogate pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for channel message truncation. Add a regression test (49 CJK chars + an emoji on the 50-char boundary) that fails on main and passes with this change. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
b82315c to
acb2540
Compare
…st-message title truncateText() derives a topic title with a hard slice(0, 50) when the first user message has no whitespace in its first 50 chars (common for CJK text). A code point above U+FFFF is two UTF-16 units, so a cut between them keeps the high surrogate and drops the low one, persisting a lone surrogate that renders as the replacement glyph in the sidebar until the user renames the topic by hand. Step the cut back one unit when the boundary falls between a surrogate pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for channel message truncation. Add a regression test (49 CJK chars + an emoji on the 50-char boundary) that fails on main and passes with this change. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
acb2540 to
6c3904f
Compare
|
Done — promoted the helper to I kept the function as That leaves the sibling #16404 and the |
…st-message title truncateText() derives a topic title with a hard slice(0, 50) when the first user message has no whitespace in its first 50 chars (common for CJK text). A code point above U+FFFF is two UTF-16 units, so a cut between them keeps the high surrogate and drops the low one, persisting a lone surrogate that renders as the replacement glyph in the sidebar until the user renames the topic by hand. Step the cut back one unit when the boundary falls between a surrogate pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for channel message truncation. Add a regression test (49 CJK chars + an emoji on the 50-char boundary) that fails on main and passes with this change. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
…ssages splitMessage's hard-cut fallback and the Discord/Slack streaming editMessage truncations sliced on UTF-16 code units. Text without a whitespace boundary within the limit (CJK has no spaces) takes the fallback, and when an emoji or other astral character straddles the cut the slice leaves a lone surrogate, which chat APIs reject or render as a replacement character. Clamp the cut to a code-point boundary so a straddling pair is kept whole. The boundary helper is pure, stateless string logic with no channel-specific dependency, so it lives in src/shared/utils/text.ts and the channel code imports it from there. That gives the sibling topic-naming fix (CherryHQ#16404) and the renderer naming fallback a single home to share instead of re-implementing the surrogate arithmetic. The unit tests for the helper move alongside it under shared/utils.
6c3904f to
ec2d5dc
Compare
|
Maintainer decision on consolidation: let's fold the sibling #16404 into this PR and fix both surfaces in one shot, rather than landing them sequentially. Why consolidate here:
Concretely, please pull #16404's
Then close #16404. I verified the helper promotion itself is correct: three real consumers, no duplicate entry point, pure stateless One non-blocking note for the merged result: Out of scope for this PR, worth a separate follow-up: |
…essage titles Folds CherryHQ#16404 into this PR per maintainer request: truncateText now clamps the cut through the shared clampSurrogateBoundary helper instead of a raw slice(0, 50), and the regression test moves over with it. Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
|
Done, folded #16404 into this PR as requested. |
Upstream refactored TopicNamingService and moved the first-message title truncation into the new shared conversationTitle helper. Reapplied the surrogate-pair guard on truncateFirstUserMessageTitleSource there instead of the old truncateText path, and kept the regression test alongside it. The channels-side fix is unchanged.
|
Rebased on latest The Coverage is at both layers:
Verified locally: the |
…Q#16352) Co-authored-by: he-yufeng <2003heyufeng@gmail.com> Co-authored-by: fullex <0xfullex@gmail.com> Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
What this PR does
Before this PR:
splitMessage(used by every channel adapter) and the Discord/Slack streamingeditMessagetruncations sliced outgoing text on raw UTF-16 code units.splitMessageonly falls back to a hard cut when there's no\n\n,\n, or space within the platform limit, which is always the case for CJK text since it has no spaces. When an emoji or other astral character straddles the cut, the slice leaves a lone surrogate behind:A lone surrogate is rejected by some chat APIs and renders as a replacement character (�) on others.
After this PR:
A small
clampSurrogateBoundaryhelper steps the cut back by one when it would land between the two halves of a surrogate pair, so the pair is carried whole into the next chunk (or dropped whole on a hard truncate). Whitespace-boundary splitting is unchanged.Why we need it and why it was done in this way
The hard cut is unavoidable for long unbroken text, so the fix is the minimal one: clamp only the fallback boundary off a surrogate pair rather than reworking the whole splitting strategy. The same helper is reused for the Discord/Slack
editMessagetruncations, which had the identical raw-sliceissue. CJK users are the most affected since their messages never hit the whitespace boundaries.Special notes for your reviewer
Added unit tests in
src/main/ai/channels/__tests__/utils.test.tscovering the surrogate-pair fallback, the unchanged whitespace path, and theclampSurrogateBoundaryedges. I verified the before/after behavior locally with a standalone Node script (the old slice leaves\uD83D, the new one doesn't); the vitest suite runs in CI.Release note