pooled-io

The motivation for this package was to run computations on multiple cores that need to write intermediate results to disk. The functions restrict the number of simultaneously running jobs to a user given number or to the number of capabilities the Haskell program was started with, i.e. the number after the RTS option -N.

There are some flavors of this functionality:

Control.Concurrent.PooledIO.Independent: run independent actions without results in parallel Control.Concurrent.PooledIO.Final: run independent actions with a final result in parallel Control.Concurrent.PooledIO.InOrder: run jobs in parallel with data dependencies like make -j n

Additionally there is the module Control.Concurrent.PooledIO.Sequence that helps to serialize I/O actions from multiple threads. It is certainly most useful in connection with Control.Concurrent.PooledIO.Independent.

Related packages:

lazyio: interleave IO actions in a single thread async: start threads and wait for their results, forward exceptions, but do not throttle concurrency with respect to number of available cores parallel-tasks: parallel-io: threadPool: threads-pool: cio: Control-Engine: taskpool, async-pool: