Replies: 3 comments · 3 replies
-
|
Hello! Your best course of action here would be to implement a custom RequestManager that would use Redis as the storage backend. Then, you can pass an instance of the new class to your crawler constructor, e.g., |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @janbuchar . That sounds like a good solution. Since I'm using |
Beta Was this translation helpful? Give feedback.
-
|
Glad to help. |
Beta Was this translation helpful? Give feedback.
-
|
Connecting to remote browsers is tracked in this feature request for JS crawlee. Once we manage to implement that, a Python version is sure to follow. It may come sooner, too - we'll see 🙂 |
Beta Was this translation helpful? Give feedback.
This comment was marked as spam.
This comment was marked as spam.
-
|
Hi, I would like to add that in the future, implementing a custom storage client (e.g., for Redis) should become much easier, as we have a storage client redesign on our roadmap (#783, #92, #307). |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Currently, as per source code, RequestQueue uses MemoryStorageClient by default. However, MemoryStorageClient uses file-based storage rather than in-memory storage. I am planning to use Redis as a storage backend for higher throughput.
Can anyone help me with an example of how to configure RequestQueue or MemoryStorageClient to use Redis as storage backend?
Beta Was this translation helpful? Give feedback.
All reactions