Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

fix(text): preserve surrogate pairs at truncation boundaries#16352

Merged
0xfullex merged 5 commits into
CherryHQ:mainCherryHQ/cherry-studio:mainfrom
he-yufeng:fix/channels-split-surrogate-pairshe-yufeng/cherry-studio:fix/channels-split-surrogate-pairsCopy head branch name to clipboard
Jul 12, 2026
Merged

fix(text): preserve surrogate pairs at truncation boundaries#16352
0xfullex merged 5 commits into
CherryHQ:mainCherryHQ/cherry-studio:mainfrom
he-yufeng:fix/channels-split-surrogate-pairshe-yufeng/cherry-studio:fix/channels-split-surrogate-pairsCopy head branch name to clipboard

Conversation

@he-yufeng

@he-yufeng he-yufeng commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Before this PR:

splitMessage (used by every channel adapter) and the Discord/Slack streaming editMessage truncations sliced outgoing text on raw UTF-16 code units. splitMessage only falls back to a hard cut when there's no \n\n, \n, or space within the platform limit, which is always the case for CJK text since it has no spaces. When an emoji or other astral character straddles the cut, the slice leaves a lone surrogate behind:

const text = '字'.repeat(9) + '😀' + '文'.repeat(20)
splitMessage(text, 10)[0] // ends with a lone high surrogate \uD83D

A lone surrogate is rejected by some chat APIs and renders as a replacement character (�) on others.

After this PR:

A small clampSurrogateBoundary helper steps the cut back by one when it would land between the two halves of a surrogate pair, so the pair is carried whole into the next chunk (or dropped whole on a hard truncate). Whitespace-boundary splitting is unchanged.

Why we need it and why it was done in this way

The hard cut is unavoidable for long unbroken text, so the fix is the minimal one: clamp only the fallback boundary off a surrogate pair rather than reworking the whole splitting strategy. The same helper is reused for the Discord/Slack editMessage truncations, which had the identical raw-slice issue. CJK users are the most affected since their messages never hit the whitespace boundaries.

Special notes for your reviewer

Added unit tests in src/main/ai/channels/__tests__/utils.test.ts covering the surrogate-pair fallback, the unchanged whitespace path, and the clampSurrogateBoundary edges. I verified the before/after behavior locally with a standalone Node script (the old slice leaves \uD83D, the new one doesn't); the vitest suite runs in CI.

Release note

Fix Discord/Slack and chunked channel replies corrupting emoji or CJK characters that landed on a message-length truncation boundary.

@he-yufeng
he-yufeng requested a review from a team June 24, 2026 14:21
@he-yufeng
he-yufeng requested a review from DeJeune as a code owner June 24, 2026 14:21

@DeJeune DeJeune left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Correct, surgical fix for lone-surrogate emission when truncating outgoing messages. clampSurrogateBoundary is sound (a pair is exactly 2 code units, so one step-back always suffices), both raw-slice truncation sites (Discord/Slack editMessage) are covered, and the shared splitMessage hard-cut fallback protects the other four adapters. Tests match the implementation and cover the meaningful paths.

@DeJeune
DeJeune requested a review from eeee0717 June 24, 2026 15:17
@he-yufeng
he-yufeng force-pushed the fix/channels-split-surrogate-pairs branch 3 times, most recently from 5c4c3f9 to ca7ce10 Compare June 25, 2026 08:00

@eeee0717 eeee0717 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Correct and surgical fix for lone-surrogate emission on truncation boundaries.

Verified:

  • clampSurrogateBoundary logic is sound — a surrogate pair is exactly 2 code units, so a single step-back always suffices; string-edge guards are correct.
  • All outgoing-message truncation sites are covered: the shared splitMessage hard-cut fallback (protects all 6 adapters) plus the two streaming editMessage raw-slice sites (Discord/Slack). The remaining .slice(0,...) hits in the channels dir are UUIDs / diagnostic error text, not chat payloads — correctly left untouched.
  • Truncated lengths stay within platform limits (Discord ≤2000, Slack ≤4000).
  • Unit tests are meaningful and match the implementation: surrogate-pair fallback, unchanged whitespace path, and the helper's edge cases. The CJK case (no spaces → forced hard cut) hits exactly the fixed path.
  • CI is green.

Minimal and well-scoped. Nice work.

@DeJeune DeJeune added the ready-to-merge This PR has sufficient reviewer approvals but is awaiting code owner approval. label Jun 25, 2026
@he-yufeng
he-yufeng force-pushed the fix/channels-split-surrogate-pairs branch 2 times, most recently from e1971fc to 242a36a Compare June 25, 2026 11:38
he-yufeng added a commit to he-yufeng/cherry-studio that referenced this pull request Jun 25, 2026
…st-message title

truncateText() derives a topic title with a hard slice(0, 50) when the
first user message has no whitespace in its first 50 chars (common for
CJK text). A code point above U+FFFF is two UTF-16 units, so a cut
between them keeps the high surrogate and drops the low one, persisting
a lone surrogate that renders as the replacement glyph in the sidebar
until the user renames the topic by hand.

Step the cut back one unit when the boundary falls between a surrogate
pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for
channel message truncation.

Add a regression test (49 CJK chars + an emoji on the 50-char boundary)
that fails on main and passes with this change.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@0xfullex

Copy link
Copy Markdown
Member

Thanks — clampSurrogateBoundary is correct and the edge tests are thorough.

Since the sibling #16404 (same author, same root cause: naive UTF-16 truncation on the CJK hard-cut fallback) needs the exact same logic for topic-title truncation and currently inlines a byte-identical copy of this boolean, the main thing I'd raise on this PR is where the shared fix should live:

  1. Promote the helper to src/shared/utils/. It's pure, stateless, cross-process-safe string logic, and the need isn't channel-specific — topic naming hits it, and src/renderer/utils/naming.ts has the same latent hard cut in its fallback (text.substring(0, maxLength)). One shared home lets fix(topic-naming): keep surrogate pairs whole when truncating the first-message title #16404 and the renderer import it instead of re-implementing. It also removes the layering reason fix(topic-naming): keep surrogate pairs whole when truncating the first-message title #16404 gave for inlining (a services/ai/channels/ import would invert layering).

  2. For the one-shot truncation callers, a code-point view reads cleaner than the manual high/low-surrogate check and is just as correct: [...text].slice(0, maxLength).join('') — the spread iterates code points, so surrogate pairs are atomic, no 0xd800/0xdc00 math. Note that's for the truncate-once shape (topic naming). For splitMessage here, keep the index-clamp you have: it clamps an arbitrary boundary inside a loop, and maxLength is a hard platform limit, so the budget must stay in code units — [...text].slice would shift it to code points. So a reasonable shared util exposes both: clampSurrogateBoundary(text, index) for channels and a thin truncate(text, max) wrapper for one-shot callers.

Intl.Segmenter ('grapheme') would be the most thorough — it also keeps ZWJ emoji, flags, skin-tone and combining sequences whole — but it isn't worth it here: the only visible corruption is the from a split surrogate, which the code-point boundary already removes; a split grapheme still renders as a valid (smaller) glyph. It's also heavier (construct once at module scope) and counts graphemes, which can overshoot a hard length budget. Worth revisiting only for a general UI-truncation util later.

Not blocking.

@0xfullex 0xfullex removed the ready-to-merge This PR has sufficient reviewer approvals but is awaiting code owner approval. label Jun 25, 2026
@0xfullex 0xfullex self-assigned this Jun 25, 2026
he-yufeng added a commit to he-yufeng/cherry-studio that referenced this pull request Jun 25, 2026
…st-message title

truncateText() derives a topic title with a hard slice(0, 50) when the
first user message has no whitespace in its first 50 chars (common for
CJK text). A code point above U+FFFF is two UTF-16 units, so a cut
between them keeps the high surrogate and drops the low one, persisting
a lone surrogate that renders as the replacement glyph in the sidebar
until the user renames the topic by hand.

Step the cut back one unit when the boundary falls between a surrogate
pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for
channel message truncation.

Add a regression test (49 CJK chars + an emoji on the 50-char boundary)
that fails on main and passes with this change.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@he-yufeng
he-yufeng force-pushed the fix/channels-split-surrogate-pairs branch from 242a36a to 5888a45 Compare June 25, 2026 14:21
he-yufeng added a commit to he-yufeng/cherry-studio that referenced this pull request Jun 25, 2026
…st-message title

truncateText() derives a topic title with a hard slice(0, 50) when the
first user message has no whitespace in its first 50 chars (common for
CJK text). A code point above U+FFFF is two UTF-16 units, so a cut
between them keeps the high surrogate and drops the low one, persisting
a lone surrogate that renders as the replacement glyph in the sidebar
until the user renames the topic by hand.

Step the cut back one unit when the boundary falls between a surrogate
pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for
channel message truncation.

Add a regression test (49 CJK chars + an emoji on the 50-char boundary)
that fails on main and passes with this change.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@he-yufeng
he-yufeng force-pushed the fix/channels-split-surrogate-pairs branch 2 times, most recently from b82315c to acb2540 Compare June 26, 2026 04:54
he-yufeng added a commit to he-yufeng/cherry-studio that referenced this pull request Jun 26, 2026
…st-message title

truncateText() derives a topic title with a hard slice(0, 50) when the
first user message has no whitespace in its first 50 chars (common for
CJK text). A code point above U+FFFF is two UTF-16 units, so a cut
between them keeps the high surrogate and drops the low one, persisting
a lone surrogate that renders as the replacement glyph in the sidebar
until the user renames the topic by hand.

Step the cut back one unit when the boundary falls between a surrogate
pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for
channel message truncation.

Add a regression test (49 CJK chars + an emoji on the 50-char boundary)
that fails on main and passes with this change.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@he-yufeng
he-yufeng force-pushed the fix/channels-split-surrogate-pairs branch from acb2540 to 6c3904f Compare June 26, 2026 08:03
@he-yufeng

Copy link
Copy Markdown
Contributor Author

Done — promoted the helper to src/shared/utils/text.ts and pointed the channel code (splitMessage plus the Discord/Slack truncations) at it, so there's one home for the surrogate arithmetic now. The unit tests moved alongside it under shared/utils/__tests__/.

I kept the function as clampSurrogateBoundary(text, index) rather than swapping splitMessage to a code-point view, for the reason you noted: the platform caps here are hard limits in UTF-16 code units, so the budget has to stay in code units. I wrote that trade-off into the JSDoc so the soft, one-shot callers know to reach for the code-point view instead.

That leaves the sibling #16404 and the src/renderer/utils/naming.ts fallback free to add a thin code-point truncate wrapper next to it. I'll fold #16404 onto this shared home once this lands, as agreed. Rebased onto main.

he-yufeng added a commit to he-yufeng/cherry-studio that referenced this pull request Jun 26, 2026
…st-message title

truncateText() derives a topic title with a hard slice(0, 50) when the
first user message has no whitespace in its first 50 chars (common for
CJK text). A code point above U+FFFF is two UTF-16 units, so a cut
between them keeps the high surrogate and drops the low one, persisting
a lone surrogate that renders as the replacement glyph in the sidebar
until the user renames the topic by hand.

Step the cut back one unit when the boundary falls between a surrogate
pair, matching the clampSurrogateBoundary check added in CherryHQ#16352 for
channel message truncation.

Add a regression test (49 CJK chars + an emoji on the 50-char boundary)
that fails on main and passes with this change.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
…ssages

splitMessage's hard-cut fallback and the Discord/Slack streaming editMessage
truncations sliced on UTF-16 code units. Text without a whitespace boundary
within the limit (CJK has no spaces) takes the fallback, and when an emoji or
other astral character straddles the cut the slice leaves a lone surrogate,
which chat APIs reject or render as a replacement character.

Clamp the cut to a code-point boundary so a straddling pair is kept whole.

The boundary helper is pure, stateless string logic with no channel-specific
dependency, so it lives in src/shared/utils/text.ts and the channel code imports
it from there. That gives the sibling topic-naming fix (CherryHQ#16404) and the renderer
naming fallback a single home to share instead of re-implementing the surrogate
arithmetic. The unit tests for the helper move alongside it under shared/utils.
@he-yufeng
he-yufeng force-pushed the fix/channels-split-surrogate-pairs branch from 6c3904f to ec2d5dc Compare June 26, 2026 10:54
@0xfullex

Copy link
Copy Markdown
Member

Maintainer decision on consolidation: let's fold the sibling #16404 into this PR and fix both surfaces in one shot, rather than landing them sequentially.

Why consolidate here:

  1. Both surfaces are main-process and now share the same arithmetic. Keeping fix(topic-naming): keep surrogate pairs whole when truncating the first-message title #16404's inline copy would re-introduce exactly the duplication this promotion eliminated. A single PR that wires both consumers to src/shared/utils/text.ts reaches the consolidated end-state atomically, with no transitional inline copy to clean up later.
  2. It's easier to review as one change — and this PR needs a fresh review anyway: the two existing approvals are against acb2540ceb (the pre-promotion inline version), so they don't cover the current shared-util structure.

Concretely, please pull #16404's TopicNamingService.truncateText fix and its regression test in here, so this PR ends up with:

  • clampSurrogateBoundary(text, index) consumed by the channels sites (splitMessage + Discord/Slack editMessage) — the index-clamp form, correct for the hard code-unit caps.
  • the topic-naming call site wired to the shared helper — either a thin code-point truncate(text, max) wrapper next to it (cleanest for the soft one-shot cap) or a direct slice(0, clampSurrogateBoundary(normalized, maxLength)). Your call; both remove the duplicate boolean.

Then close #16404.

I verified the helper promotion itself is correct: three real consumers, no duplicate entry point, pure stateless @shared util, placement and @shared alias resolve cleanly, and the channels integration test keeps the surrogate path covered through splitMessage's public API. No moved bug at any call site — all operate with 0 < index < length, and truncated lengths stay within the platform limits (Discord ≤2000, Slack ≤4000).

One non-blocking note for the merged result: splitMessage's safe precondition narrows to maxLength >= 2 (at maxLength === 1 with a leading surrogate pair the clamp returns 0 → empty chunk → no progress). Unreachable with every real platform limit, so just noting it.

Out of scope for this PR, worth a separate follow-up: src/renderer/utils/naming.ts has the same latent naive-truncation pattern (text.substring(0, maxLength) in its fallback). It's renderer-side and pre-existing, so it shouldn't ride along here — a small follow-up PR can adopt the same shared helper (the truncate wrapper fits its soft-cap shape).

…essage titles

Folds CherryHQ#16404 into this PR per maintainer request: truncateText now clamps
the cut through the shared clampSurrogateBoundary helper instead of a raw
slice(0, 50), and the regression test moves over with it.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@he-yufeng
he-yufeng requested a review from 0xfullex as a code owner July 11, 2026 15:27
@he-yufeng

Copy link
Copy Markdown
Contributor Author

Done, folded #16404 into this PR as requested. TopicNamingService.truncateText now clamps its cut through the same clampSurrogateBoundary helper from src/shared/utils/text.ts, and the regression test (emoji straddling the 50-char first-message cut) came over with it. All 6 tests in TopicNamingService.test.ts pass. Closing #16404.

Upstream refactored TopicNamingService and moved the first-message title
truncation into the new shared conversationTitle helper. Reapplied the
surrogate-pair guard on truncateFirstUserMessageTitleSource there instead
of the old truncateText path, and kept the regression test alongside it.
The channels-side fix is unchanged.
@he-yufeng

he-yufeng commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on latest main to clear the conflict. Heads-up on how the fold-in from #16404 shifted, since main moved under it:

The TopicNamingService.truncateText path I pointed at on 07-11 was refactored away upstream — first-user-message title truncation now goes through the new shared helper truncateFirstUserMessageTitleSource in src/shared/utils/conversationTitle.ts, which does a bare slice(0, 50). That's the same mid-surrogate cut, just relocated, so I moved the guard there (it now slices at clampSurrogateBoundary(...)). Net effect: TopicNamingService.ts itself no longer needs a change and drops out of the diff.

Coverage is at both layers:

Verified locally: the shared and main vitest projects are green (11 and 32 tests). Channels side is untouched.

@0xfullex 0xfullex changed the title fix(channels): keep surrogate pairs whole when truncating outgoing messages fix(text): preserve surrogate pairs at truncation boundaries Jul 12, 2026
@0xfullex
0xfullex merged commit 087f24d into CherryHQ:main Jul 12, 2026
10 checks passed
EasongChung pushed a commit to EasongChung/cherry-studio that referenced this pull request Jul 16, 2026
…Q#16352)

Co-authored-by: he-yufeng <2003heyufeng@gmail.com>
Co-authored-by: fullex <0xfullex@gmail.com>
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.