Concurrency
===========

Daemonic processes
------------------

A child process can be *daemonic* or *non-daemonic*. When a process exits, it attempts to terminate
all of its daemonic child processes (SIGTERM) while it will wait for non-daemonic child processes to finish.
An interrupt (SIGINT) is propagated to both daemonic and non-daemonic child processes.

A daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children
orphaned if it gets terminated when its parent process exits.

Create daemonic child processes

* `multiprocessing.Pool` (cannot be used in a daemonic process)
* `billiard.Pool` (can be used in a daemonic process)

Create non-daemonic child processes

* `concurrent.futures.ProcessPoolExecutor`

Create non-daemonic child processes by default (can be daemonic if requested)

* `multiprocessing.Process`
* `billiard.Process`

This can be verified with

.. code:: python

    python -m pypushflow.tests.concurrent.check_daemonic

Pool types
----------

Pypushflow supports these pools for concurrent execution of workflow tasks

* gevent: pool of greenlets from `gevent`
* thread: pool of threads from the `concurrent.futures` module
* process: pool of non-daemonic processes from the `concurrent.futures` module
* ndprocess: pool of non-daemonic processes from the `concurrent.futures` module
* multiprocessing: pool of daemonic processes from the `multiprocessing` module
* ndmultiprocessing: pool of non-daemonic processes from the `multiprocessing` module
* billiard: pool of daemonic processes from the `billiard` library

The pool type `process` and `ndprocess` are the same, but `ndprocess` uses explicit
non-daemonic processes like `ndmultiprocessing`.

In a gevent-patched environment the default pool type is *gevent* else it is *process*.

Workflow tasks
--------------

* simple: no subprocess
* subprocess: the `subprocess` module
* cfpool: process pool from the `concurrent.futures` module
* mppool: process pool from the `multiprocessing` module
* mpprocess: process from the `multiprocessing` module
* bpool: process pool from the `billiard` library
* bprocess: process from the `billiard` library

Pool-task compatibility
-----------------------

Workflow tasks that can be used in each pool type (no gevent monkey patching)

* gevent: -
* thread: simple, subprocess, cfpool, mppool, mpprocess, bpool, bprocess
* process: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
* ndprocess: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
* multiprocessing: simple, subprocess, bpool (hangs sometimes), bprocess
* ndmultiprocessing: simple, subprocess, cfpool, mppool, mpprocess, bpool (hangs sometimes), bprocess
* billiard: simple, subprocess, bpool (hangs sometimes), bprocess

Workflow tasks that can be used in each pool type (with gevent monkey patching)

* gevent: simple, subprocess, cfpool, mpprocess, bprocess
* thread: simple, subprocess, cfpool, mpprocess, bprocess
* process: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)
* ndprocess: simple, subprocess, cfpool (not spawn), mpprocess (not spawn), bprocess (not spawn)
* multiprocessing: -
* ndmultiprocessing: -
* billiard: -