multiprocessing with ugly pickle

Q

• Is multiprocessing the ultimate solution for concurrency?
• What are the patterns of concurrent programs?

GIL issues

On multi-core machines, threads will race each other to get the GIL, which make the performance even worse.

Serialization

• What can be pickled and unpickled?
• Parallelism and Serialization : how poor pickling breaks multiprocessing
• Python Pickling and multiprocessing
• What can multiprocessing and dill do together?
• Multiprocessing: using Pool.map on a function defined in a class
• Multiprocessing: using Pool.map on a function defined in a class
• Python multiprocessing with pathos

communication

• Multiprocessing, shared memory vs. pickled copies
• Python multiprocess share memory vs using arguments
• Python multiprocessing: shared memory and pickle issue
• What are the different use cases of joblib versus pickle?
• Python’s Hardest Problem, Revisited
• Shared-memory objects in python multiprocessing
• Python multiprocessing shared memory

Tutorials

• Parallel programming with Python’s multiprocessing library
• An introduction to parallel programming: using Python’s multiprocessing module
• Python - parallelizing CPU-bound tasks with multiprocessing
• A Brief Introduction To Multiprocessing
• Python Multithreading Tutorial: Concurrency and Parallelism
• PARALLELISING PYTHON WITH THREADING AND MULTIPROCESSING
• Python Multiprocessing Tutorial - Asynchronous Programming
• Multiprocessing in Python: a guided tour with examples
• Learning Python’s Multiprocessing Module
• Threading in Python [closed]
• Secrets of the Multiprocessing Module

Other reference

• Common use-cases for pickle in Python

GIL

• Understanding the Python GIL
• Beating the (sh** out of the) GIL - Multithreading vs. Multiprocessing
• Why does Python provide locking mechanisms if it’s subject to a GIL?
• Are locks unnecessary in multi-threaded Python code because of the GIL?