Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

fix(core): read BMP height as signed int32 for top-down bitmaps#5227

Merged
wenshao merged 1 commit into
QwenLM:mainQwenLM/qwen-code:mainfrom
he-yufeng:fix/bmp-top-down-heighthe-yufeng/qwen-code:fix/bmp-top-down-heightCopy head branch name to clipboard
Jun 18, 2026
Merged

fix(core): read BMP height as signed int32 for top-down bitmaps#5227
wenshao merged 1 commit into
QwenLM:mainQwenLM/qwen-code:mainfrom
he-yufeng:fix/bmp-top-down-heighthe-yufeng/qwen-code:fix/bmp-top-down-heightCopy head branch name to clipboard

Conversation

@he-yufeng

Copy link
Copy Markdown
Contributor

What this PR does

Reads the BMP height field as a signed int32 instead of uint32 so top-down bitmaps report their real height. extractBmpDimensions already intends to handle the negative-height case (Math.abs(height) with a "Height can be negative for top-down BMPs" comment), but it reads the field with readUInt32LE, so a negative height like -50 becomes 4294967246 before Math.abs runs — and Math.abs of an already-huge positive number is still huge. Switching to readInt32LE lets Math.abs recover the true height.

Why it's needed

A BMP stores a negative height to signal top-down row order — a valid, spec-defined layout. With the unsigned read, any top-down BMP parses to a ~4-billion-pixel height, and after calculateTokensWithScaling clamps to maxPixels the token estimate is pinned to the per-image maximum regardless of the image's real size — a large, silent over-count for that whole class of BMPs. (Same failure mode as the TIFF SHORT fix in #5209, just on the BMP path.)

Reviewer Test Plan

How to verify

Run the image tokenizer unit tests. Two new cases build a minimal BMP header: a bottom-up (positive height) BMP and a top-down (negative height) one. The top-down case asserts the height comes back as its absolute value (50) rather than the pre-fix 4294967246.

npx vitest run packages/core/src/utils/request-tokenizer/imageTokenizer.test.ts

Expected: 13 passed. The top-down case fails on main (expected 4294967246 to be 50) and passes with the fix.

Evidence (Before & After)

Non-UI change (token-accounting math), captured by the added unit test. Before: top-down BMP height 4294967246; after: 50.

Tested on

OS Status
🍏 macOS ⚠️ not tested
🪟 Windows ✅ tested
🐧 Linux ⚠️ not tested

Environment (optional)

Unit tests only (vitest).

Risk & Scope

  • Main risk or tradeoff: low — only the height read changes; bottom-up BMPs (the common case) are unaffected since a positive value reads the same signed or unsigned. Width stays unsigned (BMP width is always positive).
  • Not validated / out of scope: real-world BMP files beyond the synthetic header fixtures.
  • Breaking changes / migration notes: none.

@wenshao

wenshao commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

✅ Local verification — fix(core): read BMP height as signed int32 for top-down bitmaps

Verdict: LGTM — safe to merge. The fix is correct, minimal, and properly guarded by the two new tests. One non-blocking note about the PR description's symptom wording (§4 below).

Verified as maintainer on 🐧 Linux (the OS the PR marked not tested), Node v22.22.2, against PR head ade8f2d. Real testing in an isolated tmux socket: unit tests both directions → harness over the real built packages/core/dist with real on-disk BMP files → confirmation the fix shipped into the user-facing dist/cli.js bundle → a live end-to-end CLI turn.

1. Unit tests & static checks

  • PR branch: imageTokenizer.test.ts13/13 pass; full request-tokenizer/ suite → 55/55 pass; tsc --noEmit on packages/coreclean.
  • Regression proof — reverting only the one-line change (readInt32LEreadUInt32LE) fails exactly the top-down case while the bottom-up case keeps passing, so the test genuinely guards the bug:
× ImageTokenizer > BMP dimension extraction > reads a top-down (negative height) BMP as its absolute height
  → expected 4294967246 to be 50
 Tests  1 failed | 12 passed (13)

2. Real built code + real BMP files

Generated two genuine BMPs with PIL (15054 bytes each, real pixel data): a bottom-up (height +50) and a top-down (height -50; header bytes at offset 22 = ce ff ff ff). Fed both through the built ImageTokenizer / RequestTokenEstimator (not a re-implementation):

=== top-down (height -50) | topdown.bmp (15054 bytes) ===
  raw header @22  readUInt32LE=4294967246  readInt32LE=-50
  extractImageMetadata -> width=100 height=50            ✅ recovered
  calculateTokens (fixed dim 100x50)         = 10
  calculateTokens (buggy dim 100x4294967246) = 2         (garbage)
  RequestTokenEstimator(request): totalTokens=10 imageTokens=10
=== bottom-up (height +50) ===
  10 tokens both ways → common case untouched             ✅

3. Live tmux end-to-end

(a) The exact computation OpenAIContentGenerator.countTokens() performs (new RequestTokenEstimator().calculateTokens(request)), run live on the real top-down BMP:

LIVE countTokens() on real top-down BMP (100x-50):
  raw @22 uint32 : 4294967246 (pre-fix)   int32 : -50 (post-fix)
  => totalTokens : 10   breakdown {"imageTokens":10,...}
  RESULT: PASS (sane 10-token count, not 2/131074 garbage)

(b) A full real CLI turn — qwen -p '… @topdown.bmp …' — ingested the real top-down file end-to-end; the vision model read it back as 100x50, exit 0, no crash on the would-be 4-billion dimension.

4. 📌 Non-blocking note on the PR description

The motivation text says the bug yields "a large, silent over-count … pinned to the per-image maximum." Running the real calculateTokens across many top-down dimensions shows the opposite for typical images: after scaling, the (real-sized) width floors to 0, so the estimate collapses to just the 2 special tokens — a silent under-count. The over-count direction only shows up for pathological near-1px-wide images:

width height  preFix(uint32)  postFix(abs)
100   50      2               10       <- under-count
1920  1080    2               2693     <- under-count
4000  3000    2               15303    <- under-count
1     1       131074          6        <- over-count (degenerate only)

Either way the pre-fix estimate is garbage by orders of magnitude and the fix is unambiguously correct — only the description's wording could be tweaked. Does not affect the merge decision.

Out of scope (FYI — pre-existing, not introduced here)

  • width stays unsigned (readUInt32LE) — fine; BMP width is positive by spec.
  • The TIFF type === 3 ? value : value no-op ternaries (≈ lines 429/432) are unrelated to this PR.

Recommendation: merge. ✅

🇨🇳 中文版(点击展开)

✅ 本地验证 — fix(core): read BMP height as signed int32 for top-down bitmaps

结论:LGTM,可以合并。 修复正确、改动最小,且被两个新增测试有效守护。仅有一条不阻塞合并的说明,关于 PR 描述里对“症状”的措辞(见 §4)。

以维护者身份在 🐧 Linux(PR 标注为「未测试」的系统)、Node v22.22.2、PR head ade8f2d 上验证。在隔离的 tmux socket 中进行真实测试:双向单测 → 用真实构建产物 packages/core/dist + 真实磁盘 BMP 文件跑 harness → 确认修复已进入面向用户的 dist/cli.js 打包产物 → 一次真实的端到端 CLI 调用。

1. 单元测试与静态检查

  • PR 分支:imageTokenizer.test.ts13/13 通过;整个 request-tokenizer/ 套件 → 55/55 通过packages/coretsc --noEmit无错误
  • 回归证明 —— 只把这一行从 readInt32LE 改回 readUInt32LE,就恰好让 top-down 用例失败、而 bottom-up 用例依然通过,说明该测试确实守护住了这个 bug:
× ... reads a top-down (negative height) BMP as its absolute height
  → expected 4294967246 to be 50
 Tests  1 failed | 12 passed (13)

2. 真实构建产物 + 真实 BMP 文件

用 PIL 生成两个真实 BMP(各 15054 字节,含真实像素数据):bottom-up(height +50)与 top-down(height -50;偏移 22 处头字节为 ce ff ff ff)。两者都喂给构建后的 ImageTokenizer / RequestTokenEstimator(非重新实现):

=== top-down (height -50) | topdown.bmp (15054 字节) ===
  原始头 @22  readUInt32LE=4294967246  readInt32LE=-50
  extractImageMetadata -> width=100 height=50            ✅ 已还原
  calculateTokens (修复后维度 100x50)        = 10
  calculateTokens (错误维度 100x4294967246)  = 2          (垃圾值)
  RequestTokenEstimator(request): totalTokens=10 imageTokens=10
=== bottom-up (height +50) ===
  两种读法都是 10 tokens → 常见路径不受影响               ✅

3. tmux 真实端到端

(a) 直接复现 OpenAIContentGenerator.countTokens() 实际执行的计算(new RequestTokenEstimator().calculateTokens(request)),在真实 top-down BMP 上现场运行:

真实 top-down BMP (100x-50) 上的 countTokens():
  raw @22 uint32 : 4294967246 (修复前)   int32 : -50 (修复后)
  => totalTokens : 10   breakdown {"imageTokens":10,...}
  结果:PASS(合理的 10 tokens,而非 2 / 131074 垃圾值)

(b) 一次完整的真实 CLI 调用 —— qwen -p '… @topdown.bmp …' —— 端到端地读入了真实 top-down 文件;视觉模型把它识别为 100x50,退出码 0,没有因为那个本会是 40 亿的维度而崩溃。

4. 📌 关于 PR 描述的不阻塞说明

描述中称该 bug 会导致*“一个很大的、静默的超额计数……被钉在每图最大值上”*。但用真实calculateTokens 跑了多种 top-down 维度后,典型图片的表现恰好相反:缩放后(真实大小的)宽度向下取整为 0,于是估算坍缩到只剩 2 个特殊 token —— 是一种静默的少算。只有在病态的近 1 像素宽图片上才会出现超额计数:

width height  修复前(uint32)  修复后(abs)
100   50      2               10       <- 少算
1920  1080    2               2693     <- 少算
4000  3000    2               15303    <- 少算
1     1       131074          6        <- 超额(仅退化情形)

无论哪个方向,修复前的估算都偏差了好几个数量级,修复本身毫无疑问是正确的 —— 只是描述措辞可以微调一下,不影响合并决定

范围之外(仅供参考 —— 既有问题,非本 PR 引入)

  • width 仍按无符号读取(readUInt32LE)—— 没问题;按规范 BMP 宽度恒为正。
  • TIFF 中 type === 3 ? value : value 这种两边相同的空三元(约 429/432 行)与本 PR 无关。

建议:合并。✅

Verification artifacts: real BMP fixtures (md5 topdown.bmp = 79c3d11f…), dist harness, and live tmux captures retained locally.

@wenshao

wenshao commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓

On direction: this fixes a real, concrete bug — top-down BMPs (negative height) produce a ~4-billion-pixel dimension that corrupts token estimation. The code already had the Math.abs() intent but the unsigned read defeated it. Claude Code's CHANGELOG has a related BMP fix ("Fixed image pasting not working on WSL2 systems where Windows copies images as BMP format"), confirming the BMP path matters in practice. Clearly aligned with the project.

On approach: this is about as minimal as a fix gets — one line changed (readUInt32LEreadInt32LE), one comment added, two focused tests. Positive heights read identically signed or unsigned, so the common bottom-up case is untouched. Width stays unsigned, which is correct. Nothing to cut.

Moving on to code review and testing. 🔍

中文说明

感谢贡献!

模板完整 ✓

方向:修复了一个真实存在的 bug — top-down BMP(负高度)会产生约 40 亿像素的尺寸,破坏 token 估算。代码中已有 Math.abs() 的意图,但无符号读取使其失效。Claude Code 的 CHANGELOG 也有相关 BMP 修复("修复了 WSL2 系统上 Windows 以 BMP 格式复制图片导致无法正常粘贴的问题"),确认 BMP 路径在实践中确实重要。方向明确对齐。

方案:改动极简 — 只改了一行(readUInt32LEreadInt32LE),加了一行注释,新增两个聚焦的测试。正高度值在无符号和有符号读取下结果一致,因此常见的 bottom-up 情况不受影响。宽度保持无符号读取,这是正确的。没有可精简的部分。

进入代码审查和测试 🔍

Qwen Code · qwen3.7-max

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

Code Review

My independent proposal: change readUInt32LE(22)readInt32LE(22) in extractBmpDimensions. The existing Math.abs(height) on the return line already handles the negative case — the bug is purely that the unsigned read converts -50 to 4294967246 before Math.abs ever sees it. Add two tests (positive and negative height).

The PR's approach matches exactly. One-line production change, two focused tests, width correctly left as unsigned. Nothing missed, nothing extra. No blockers.

Testing

Unit tests (PR branch): 13/13 passed, including both new BMP cases.

$ cd packages/core && npx vitest run src/utils/request-tokenizer/imageTokenizer.test.ts

 RUN  v3.2.4 packages/core
      Coverage enabled with v8

 ✓ src/utils/request-tokenizer/imageTokenizer.test.ts (13 tests) 9ms

 Test Files  1 passed (1)
      Tests  13 passed (13)
   Duration  3.92s

TypeScript: tsc --noEmit on packages/core — clean, no errors.

Regression proof — reverting only readInt32LEreadUInt32LE fails exactly the top-down case while bottom-up still passes:

$ sed -i 's/buffer.readInt32LE(22)/buffer.readUInt32LE(22)/' .../imageTokenizer.ts
$ npx vitest run src/utils/request-tokenizer/imageTokenizer.test.ts

 ❯ src/utils/request-tokenizer/imageTokenizer.test.ts (13 tests | 1 failed) 16ms
   ✓ ... reads a bottom-up (positive height) BMP 0ms
   × ... reads a top-down (negative height) BMP as its absolute height 8ms
     → expected 4294967246 to be 50 // Object.is equality

- Expected
+ Received

- 50
+ 4294967246

 Test Files  1 failed (1)
      Tests  1 failed | 12 passed (13)

The test genuinely guards the bug — restoring readInt32LE makes it pass again.

中文说明

代码审查

我的独立方案:把 extractBmpDimensions 中的 readUInt32LE(22) 改为 readInt32LE(22)。返回行已有的 Math.abs(height) 已能处理负值——bug 纯粹是因为无符号读取把 -50 变成了 4294967246Math.abs 还没来得及执行就已经是巨大了。增加两个测试(正高度和负高度)。

PR 的方案完全一致。生产代码改了一行,两个聚焦的测试,宽度正确地保持无符号读取。没有遗漏,没有多余。无阻塞项。

测试

单元测试(PR 分支): 13/13 通过,包括两个新增 BMP 用例。

TypeScript: packages/coretsc --noEmit 无错误。

回归证明: 仅将 readInt32LE 改回 readUInt32LE,恰好只有 top-down 用例失败,bottom-up 用例依然通过。测试确实守护住了这个 bug——恢复 readInt32LE 后测试重新通过。

Qwen Code · qwen3.7-max

@qwen-code-ci-bot

Copy link
Copy Markdown
Collaborator

This is a textbook bugfix — the kind of PR you wish every contribution looked like.

The bug is real and well-diagnosed: BMP height is a signed int32, the code read it unsigned, Math.abs on a huge positive number does nothing. The fix is one line (readUInt32LEreadInt32LE), the existing Math.abs does the rest, and the comment explains why. Width stays unsigned because BMP width is always positive — correct asymmetry.

The two tests are exactly the right shape: one bottom-up (positive height, common case, should keep working), one top-down (negative height, the bug). The regression proof — reverting the one-line change causes exactly the top-down test to fail while bottom-up passes — confirms the tests genuinely guard the fix rather than just happening to pass.

The existing maintainer verification comment on this PR already covers real BMP files through the built ImageTokenizer and a live CLI turn. My testing confirms the unit tests and regression guard. Nothing left to verify.

Verdict: Approve.

中文说明

这是一个教科书级的 bug 修复——理想中每个 PR 都该有的样子。

Bug 真实且诊断清楚:BMP 高度是有符号 int32,代码用无符号读取,Math.abs 对一个巨大正数不起作用。修复只需一行(readUInt32LEreadInt32LE),已有的 Math.abs 完成剩下的工作,注释解释了原因。宽度保持无符号读取因为 BMP 宽度始终为正——正确的不对称处理。

两个测试恰好是合适的形态:一个 bottom-up(正高度,常见情况,应继续正常工作),一个 top-down(负高度,即 bug 所在)。回归证明——把这一行改回去恰好只有 top-down 测试失败而 bottom-up 继续通过——确认测试确实守护住了修复,而非凑巧通过。

该 PR 上已有的维护者验证评论已覆盖了通过构建后的 ImageTokenizer 处理真实 BMP 文件和真实 CLI 调用的场景。我的测试确认了单元测试和回归守护。没有剩余需要验证的。

结论:批准。

Qwen Code · qwen3.7-max

@qwen-code-ci-bot qwen-code-ci-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship. ✅

@wenshao
wenshao merged commit c419814 into QwenLM:main Jun 18, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Morty Proxy This is a proxified and sanitized view of the page, visit original site.