Thanks for the detailed and thoughtful response! This really helps clarify the RNG reproducibility landscape. (@rstub is the bees knees for RNGs.)

Your explanation of the per-worker RNG state initialization in parallel approaches particularly helped me understand why we're seeing different results with different numbers of workers. The example output showing how the results change with 1-4 workers really drove that point home.

A few follow-up questions about the Future package's approach:

1. You mentioned that Future takes a conservative approach making results invariant to worker count. Could you elaborate on what design tradeoffs this introduces?

2. When you implement the planned per-worker RNG state option as an alternative to per-iteration, will this be opt-in or will the default be updated?

3. For my specific use case, we sometimes need to switch between sequential and parallel execution based on input size. Would you recommend:

   • Always using Future's approach for consistency even when running sequentially
   • Using native sequential/parallel approaches for better performance and accepting the numerical differences
   • Some other strategy?

Regarding additional thoughts on this approach, maybe we should ping @pierrelecuyer? I'm not sure how active he is on community sites; but, about ~8 years back he provided guidance on MT, c.f. https://cs.stackexchange.com/a/82490.

Thanks again for taking the time to explain this complex topic. The background really helps and I'll try to write up a blog post summarizing the above plus putting a request for comments out.

From my understanding, the Future package is generating specific seeds for every iteration, which is not exactly cheap.¹ This might be the reason for the better performance of the native methods compared with Future's approach that you alluded to. The upside is that this makes it independent of the number of workers.

I don't think that one approach is inherently better than the other: Per worker seeds is fast but gives only statistical reproducibility, while per iteration seeds is more costly but gives also numerical reproducibility. I think it is good to have choice between the two approaches, since it requires subject-matter expertise to tell which of the two is needed.

1 I plan to benchmark dqrng in that regard: daqana/dqrng#95

BTW, more clarifications of our RNG options (ignore the RNG kind per se):

I think there are actually two type of per-worker RNG sets. If workers are persistent (e.g. multisession; parallel PSOCK workers), then we can initiate the RNG state once when the parallel worker is first launched. If workers are transient (e.g. multicore, future.callr::callr, ...), then a new worker is launched for each future, meaning the RNG state has to be initiated once per future.

In contrast, the once-per-iteration approach (the current approach), initiates the RNG state ones per iteration. Each future may process many iterations.

For instance, consider future_lapply(X, FUN = myfcn, future.seed = TRUE) where length(X) is 100. By default, future_lapply() will partition X into 4 (=number of workers) uniformly size chunks (Xchunk[[1]], Xchunk[[2]], ...) and then run lapply(Xchunk[[kk]], FUN = myfcn) on each chunk kk = 1, 2, 3, 4, and finally collected and rearrange the results. The per-iteration RNG approach, pre-generated initial RNG states for each of the length(X) elements (aka "iterations"). These pregenerated RNG states are then applied one by one just as each new myfcn(Xchunk[[kk]][[jj]]) call is made. The expensive part with this per-iteration approach is that we have to generate length(X) RNG states ("streams") each time future_lapply() is called.

Next, consider, we have 4 persistent workers. If we take the per-worker approach, we could just initiate the RNG state once, when we launch the workers. Think, parallel::clusterSetRNGStream(workers, ...). This approach is much cheaper that the per-iteration approach, because we only have to generate nbrOfWorkers() RNG streams once per R session. The difference in setup time becomes significant when n * length(X) >> nbrOfWorkers(), where n is the number of calls to future_lapply().

Then, consider we have 4 transient workers. If we take the per-worker approach, we can only initiate their RNG states when these workers are launched - a worker is only launched when the future is launched. So in our future_lapply() approach, that would mean we have to pregenerate one RNG state per chunk. The default is nbrOfWorkers() chunks, so that means nbrOfWorkers() RNG streams per future_apply() call. So, compares to the per-worker approach with persistent workers, this requires generation of n * nbrOfWorkers() RNG streams if we call future_lapply() n times. So, with transient workers we will have to produce n times more RNG streams than with persistent workers.

All three of these strategies will produce different random number sets.

PS. The per-worker approach with one RNG stream per chunk is easier to implement, because it can be used everywhere without knowing if the future backend runs on persistent or transient workers. With that argument, that might be what will be added first.