feat(perf): move token counting to Web Worker architecture #2848
Conversation
|
|
The changes in this pull request are focused on improving token counting performance by moving it to a Web Worker architecture. The modifications across different files are interconnected and support this architectural shift. Therefore, it is not necessary to split this pull request into smaller ones, as the changes are part of a cohesive feature enhancement. |
| retryDelay: config.retryDelay ?? 1000, | ||
| } | ||
|
|
||
| const absolutePath = path.resolve(__dirname, workerPath) |
There was a problem hiding this comment.
The resolution of the worker file path using __dirname may not correctly locate the worker in a production build. If WorkerManager is compiled into a subdirectory (like dist/services/workers), appending a relative path like workers/token-counter.worker.js might resolve incorrectly. Consider revising the path computation or using an absolute path.
| return worker | ||
| } | ||
|
|
||
| private async handleWorkerError(workerId: string, config: WorkerConfig): Promise<void> { |
There was a problem hiding this comment.
The retry logic in handleWorkerError only terminates and deletes the failing worker without automatically reinitializing it. Consider either restarting the worker after the delay or document that subsequent calls to initializeWorker will create a new instance.
This comment was generated because it violated a code review rule: mrule_aQsEnH8jWdOfHq2Z.
|
Ah, interesting idea - will play around with it! |
Improve performance by offloading token counting to a dedicated Web Worker. This prevents blocking the main thread during token calculations. Key changes: Add token-counter worker with optimized Tiktoken usage Implement robust WorkerManager with error handling and retries Update base provider to use worker for token counting Configure esbuild to properly bundle worker code
3942456 to
5d414dd
Compare
Move token counting logic to a dedicated worker thread for improved performance. Adds support for both text and image content blocks using cl100k_base encoding. The worker is lazily initialized and reused across requests. Key changes: Create token-counter.worker.ts for threaded token counting Update BaseProvider to use worker for countTokens method Add proper cleanup and error handling Support image token estimation
9d1c2ef to
f4ac42c
Compare
|
I patched this into a local Roo build, with hopes that it would help with the frequent and lengthy pauses I've been encountering while using the extension. It worked like a charm. I've been coding for about 16 hours with the patch, and not so much as a hiccup. |
|
PR #2848 moves token counting to Web Workers to prevent UI blocking. Tested by @shyndman with 16 hours of usage, reporting no issues. Uses • base-provider.ts | token counting implementation location |
Thanks for your review @shyndman 💪 |
Improve performance by offloading token counting to a dedicated Web Worker. This prevents blocking the main thread during token calculations.
Key changes:
Add token-counter worker with optimized Tiktoken usage Implement robust WorkerManager with error handling and retries Update base provider to use worker for token counting Configure esbuild to properly bundle worker code
Context
because of Current token counter that make Roo un-responsive when we have large input & output token
ref: https://discord.com/channels/1332146336664915968/1364053939045007391
Implementation
(nodejs is still single thread using worker to improve performance)
Screenshots
How to Test
Get in Touch
Important
Offloads token counting to a Web Worker architecture, improving performance by preventing main thread blocking.
token-counter.worker.tsto prevent main thread blocking.BaseProviderto useWorkerManagerfor token counting.WorkerManagerinWorkerManager.tsfor managing worker lifecycle, error handling, and retries.esbuild.jsto bundletoken-counter.worker.tswithworkerConfig.BaseProviderandWorkerManager.This description was created by
for 3942456. You can customize this summary. It will automatically update as commits are pushed.