You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For a repo scan, this opens and reads every file in text mode, which on a 1 MB max-size file with errors='ignore' runs UTF-8 decode over the entire buffer just to throw it away. On a large monorepo (e.g. tens of thousands of TS files in a yarn workspace), this is the dominant cost in RepoScanner.scan — and the result (line_count) is only used for display.
(1) alone gives roughly a 3-4× speedup vs the current implementation in my quick mental check. (2) caps memory usage if the limit is ever raised above 1 MB.
Not a blocker, but RepoScanner.scan is in the hot path for context building (see app.code.context.context_pack.py), so any free wins here pay back at every request.
backend/app/code/context/repo_scanner.py:232-238:For a repo scan, this opens and reads every file in text mode, which on a 1 MB max-size file with
errors='ignore'runs UTF-8 decode over the entire buffer just to throw it away. On a large monorepo (e.g. tens of thousands of TS files in a yarn workspace), this is the dominant cost inRepoScanner.scan— and the result (line_count) is only used for display.Two cheap improvements:
Read in bytes, then count
b'\n':Same answer for any file with a trailing newline, slightly off (by 1) for files without — but
line_countis informational, not load-bearing.Stream in chunks for files near the 1 MB cap:
(1) alone gives roughly a 3-4× speedup vs the current implementation in my quick mental check. (2) caps memory usage if the limit is ever raised above 1 MB.
Not a blocker, but
RepoScanner.scanis in the hot path for context building (seeapp.code.context.context_pack.py), so any free wins here pay back at every request.