An actor.json like this:
{
"actorSpecification": 1,
"name": "example-actor",
"version": "0.0",
"buildTag": "latest",
"environmentVariables": {},
"input": {
"properties": {
"startUrls": {
"title": "Start URLs",
"type": "array",
"description": "URLs to start with",
"prefill": [
{
"url": "https://apify.com"
}
],
"editor": "requestListSources"
},
"proxyConfiguration": {
"title": "Proxy configuration",
"type": "object",
"description": "Select proxies to be used by your crawler.",
"prefill": {
"useApifyProxy": true
},
"editor": "proxy"
}
}
}
}
will have an input editor like this:
and calling Actor.get_input() in the actor will result into something like this:
{
"proxyConfiguration": {
"useApifyProxy": True,
"apifyProxyGroups": ["BUYPROXIES63811", "SHADER"],
},
"startUrls": [
{"url": "https://google.com", "method": "GET"},
{"url": "https://salas.hu", "method": "GET"},
{"url": "https://trivago.hu", "method": "GET"},
{"requestsFromUrl": "https://lorem.se/ipsum.txt", "method": "GET"},
],
}It would be awesome if we could implement some helpers to simplify converting these structures into crawlee.storage.RequestList or apify.ProxyConfiguration.
Since it is pretty convenient to have the input structure described with a Pydantic model (so you can do Input.model_validate(await Actor.get_input()), I would love to explore the possibility of adding some Pydantic models that you could use in your Input models and then pass them to ProxyConfiguration or RequestList constructors (or some factory methods).
An
actor.jsonlike this:{ "actorSpecification": 1, "name": "example-actor", "version": "0.0", "buildTag": "latest", "environmentVariables": {}, "input": { "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "URLs to start with", "prefill": [ { "url": "https://apify.com" } ], "editor": "requestListSources" }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Select proxies to be used by your crawler.", "prefill": { "useApifyProxy": true }, "editor": "proxy" } } } }will have an input editor like this:
and calling
Actor.get_input()in the actor will result into something like this:{ "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["BUYPROXIES63811", "SHADER"], }, "startUrls": [ {"url": "https://google.com", "method": "GET"}, {"url": "https://salas.hu", "method": "GET"}, {"url": "https://trivago.hu", "method": "GET"}, {"requestsFromUrl": "https://lorem.se/ipsum.txt", "method": "GET"}, ], }It would be awesome if we could implement some helpers to simplify converting these structures into
crawlee.storage.RequestListorapify.ProxyConfiguration.Since it is pretty convenient to have the input structure described with a Pydantic model (so you can do
Input.model_validate(await Actor.get_input()), I would love to explore the possibility of adding some Pydantic models that you could use in yourInputmodels and then pass them toProxyConfigurationorRequestListconstructors (or some factory methods).