Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ahead-of-time pool allocation #3

Open
philsmt opened this issue Feb 18, 2021 · 1 comment
Open

Allow ahead-of-time pool allocation #3

philsmt opened this issue Feb 18, 2021 · 1 comment

Comments

@philsmt
Copy link
Collaborator

philsmt commented Feb 18, 2021

Right now, a new pool is allocated whenever map() is called. For use in more realtime scenarios and general optimizing, it would be nice to be able to do this once and use the pool for several passes of calculations. Special care has to be taken that shared memory is allocated before the pool.

@takluyver
Copy link
Member

If you're using a process pool, you also need to set up any inputs before allocating the pool, I think - they have to be there when you fork. That could easily be confusing to people who haven't studied the internals.

When I added an option for multithreaded assembly to EXtra-geom (European-XFEL/EXtra-geom#17), I designed it so you could reuse a thread pool rather than creating a new one each time. But my experiments with it suggested that starting & stopping a concurrent.futures.ThreadPoolExecutor is actually pretty cheap, so there's not much pressure to reuse one. (Unfortunately I didn't write down the absolute numbers I saw, just that it was quick compared to assembling images)

So I might suggest leaving this on the back burner until there's a clearer need. I think it's actually a strength for fork-based parallelism that defining your pool (creating a ProcessContext) isn't tied to forking, and forking is delayed until you call map().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants