fix: Fix calculation of CPU utilization from SystemInfo events #447
Conversation
f388d1f to
b63a243
Compare
There was a problem hiding this comment.
Just something to think about: Is this scaling based on multiple cpus load working well in python context?
I think that there we are mostly running on one core all the time, not sure if there are some load relevant sub-processes. So scaling based on other cores being not utilized does not seem very compatible with current somewhat single process crawlee architecture.
Well, about the same as in Javascript context really 🤷
Playwright is the most relevant one, I suppose. For HTTP-based crawlers, I agree that we probably won't be able to utilize more than one core. But also, most people probably won't try to run those on thicker units than your standard 4GB one. |
Apify reports CPU utilization as the sum of utilization of all CPUs, but Crawlee expects a number between 0 and 1. Because of this, it is impossible for AutoscaledPool to scale beyond one CPU.