You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I find myself often wanting to run code on a worker, rather than on my local client. This happens in a few settings:
My workers have access to a data store that I don't, so I need to call something like dd.read_parquet remotely (cc @martindurant@jcrist )
My workers are far away from my client, so client-heavy operations like joblib or Hyperband incur a serious bottleneck from client-scheduler communication (cc @stsievert )
My workers have hardware or libraries like GPUs/RAPIDS that I don't have locally (cc @quasiben@kkraus14)
Today I can do this by writing a function and submitting that function as a task
I know that @eriknw has done magic like this in the past. We could enlist this help. However, we may not want to do this due to the magical and novel behavior.
I find myself often wanting to run code on a worker, rather than on my local client. This happens in a few settings:
dd.read_parquetremotely (cc @martindurant @jcrist )Today I can do this by writing a function and submitting that function as a task
It might make sense to provide syntax around this to make it more magical (or it might not). We might do something like the following:
I know that @eriknw has done magic like this in the past. We could enlist this help. However, we may not want to do this due to the magical and novel behavior.