Concurrency¶
Daemonic processes¶
A child process can be daemonic or non-daemonic. When a process exits, it attempts to terminate all of its daemonic child processes (SIGTERM) while it will wait for non-daemonic child processes to finish. An interrupt (SIGINT) is propagated to both daemonic and non-daemonic child processes.
A daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits.
Create daemonic child processes
multiprocessing.Pool (cannot be used in a daemonic process)
billiard.Pool (can be used in a daemonic process)
Create non-daemonic child processes
concurrent.futures.ProcessPoolExecutor
Create non-daemonic child processes by default (can be daemonic if requested)
multiprocessing.Process
billiard.Process
This can be verified with
python -m pypushflow.tests.concurrent.check_daemonic
Pool types¶
Pypushflow supports these pools for concurrent execution of workflow tasks
gevent: pool of greenlets from gevent
thread: pool of threads from the concurrent.futures module
process: pool of non-daemonic processes from the concurrent.futures module
ndprocess: pool of non-daemonic processes from the concurrent.futures module
multiprocessing: pool of daemonic processes from the multiprocessing module
ndmultiprocessing: pool of non-daemonic processes from the multiprocessing module
billiard: pool of daemonic processes from the billiard library
The pool type process and ndprocess are the same, but ndprocess uses explicit non-daemonic processes like ndmultiprocessing.
In a gevent-patched environment the default pool type is gevent else it is process.
Workflow tasks¶
simple: no subprocess
subprocess: the subprocess module
cfpool: process pool from the concurrent.futures module
mppool: process pool from the multiprocessing module
mpprocess: process from the multiprocessing module
bpool: process pool from the billiard library
bprocess: process from the billiard library
Pool-task compatibility¶
Workflow tasks that can be used in each pool type (no gevent monkey patching)
gevent: -
thread: simple, subprocess, cfpool, mppool, mpprocess, bpool, bprocess
process: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
ndprocess: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
multiprocessing: simple, subprocess, bpool (hangs sometimes), bprocess
ndmultiprocessing: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
billiard: simple, subprocess, bpool (hangs sometimes), bprocess
Workflow tasks that can be used in each pool type (with gevent monkey patching)
gevent: simple, subprocess, cfpool, mpprocess, bprocess
thread: simple, subprocess, cfpool, mpprocess, bprocess
process: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)
ndprocess: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)
multiprocessing: -
ndmultiprocessing: -
billiard: -