Concurrency

Daemonic processes

A child process can be daemonic or non-daemonic. When a process exits, it attempts to terminate all of its daemonic child processes (SIGTERM) while it will wait for non-daemonic child processes to finish. An interrupt (SIGINT) is propagated to both daemonic and non-daemonic child processes.

A daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits.

Create daemonic child processes

  • multiprocessing.Pool (cannot be used in a daemonic process)

  • billiard.Pool (can be used in a daemonic process)

Create non-daemonic child processes

  • concurrent.futures.ProcessPoolExecutor

Create non-daemonic child processes by default (can be daemonic if requested)

  • multiprocessing.Process

  • billiard.Process

This can be verified with

python -m pypushflow.tests.concurrent.check_daemonic

Pool types

Pypushflow supports these pools for concurrent execution of workflow tasks

  • gevent: pool of greenlets from gevent

  • thread: pool of threads from the concurrent.futures module

  • process: pool of non-daemonic processes from the concurrent.futures module

  • ndprocess: pool of non-daemonic processes from the concurrent.futures module

  • multiprocessing: pool of daemonic processes from the multiprocessing module

  • ndmultiprocessing: pool of non-daemonic processes from the multiprocessing module

  • billiard: pool of daemonic processes from the billiard library

The pool type process and ndprocess are the same, but ndprocess uses explicit non-daemonic processes like ndmultiprocessing.

In a gevent-patched environment the default pool type is gevent else it is process.

Workflow tasks

  • simple: no subprocess

  • subprocess: the subprocess module

  • cfpool: process pool from the concurrent.futures module

  • mppool: process pool from the multiprocessing module

  • mpprocess: process from the multiprocessing module

  • bpool: process pool from the billiard library

  • bprocess: process from the billiard library

Pool-task compatibility

Workflow tasks that can be used in each pool type (no gevent monkey patching)

  • gevent: -

  • thread: simple, subprocess, cfpool, mppool, mpprocess, bpool, bprocess

  • process: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess

  • ndprocess: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess

  • multiprocessing: simple, subprocess, bpool (hangs sometimes), bprocess

  • ndmultiprocessing: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess

  • billiard: simple, subprocess, bpool (hangs sometimes), bprocess

Workflow tasks that can be used in each pool type (with gevent monkey patching)

  • gevent: simple, subprocess, cfpool, mpprocess, bprocess

  • thread: simple, subprocess, cfpool, mpprocess, bprocess

  • process: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)

  • ndprocess: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)

  • multiprocessing: -

  • ndmultiprocessing: -

  • billiard: -