You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, a new pool is allocated whenever map() is called. For use in more realtime scenarios and general optimizing, it would be nice to be able to do this once and use the pool for several passes of calculations. Special care has to be taken that shared memory is allocated before the pool.
The text was updated successfully, but these errors were encountered:
If you're using a process pool, you also need to set up any inputs before allocating the pool, I think - they have to be there when you fork. That could easily be confusing to people who haven't studied the internals.
When I added an option for multithreaded assembly to EXtra-geom (European-XFEL/EXtra-geom#17), I designed it so you could reuse a thread pool rather than creating a new one each time. But my experiments with it suggested that starting & stopping a concurrent.futures.ThreadPoolExecutor is actually pretty cheap, so there's not much pressure to reuse one. (Unfortunately I didn't write down the absolute numbers I saw, just that it was quick compared to assembling images)
So I might suggest leaving this on the back burner until there's a clearer need. I think it's actually a strength for fork-based parallelism that defining your pool (creating a ProcessContext) isn't tied to forking, and forking is delayed until you call map().
Right now, a new pool is allocated whenever
map()
is called. For use in more realtime scenarios and general optimizing, it would be nice to be able to do this once and use the pool for several passes of calculations. Special care has to be taken that shared memory is allocated before the pool.The text was updated successfully, but these errors were encountered: