Offensive Security Researcher & Quantum Cryptography Analyst

Running Two Simple Processes in Python: Complete Multiprocessing and Parallel Execution Tutorial

Abdul Wahab Junaid
Running Two Simple Processes in python

Running Two Simple Processes in python

The moment I hit a genuinely CPU-bound bottleneck in Python — a script crunching numbers across a large dataset that threading couldn’t speed up no matter how I structured it — I finally sat down and learned multiprocessing properly. Where threading is limited by the Global Interpreter Lock for CPU-bound work, spawning separate processes sidesteps that limitation entirely, because each process gets its own Python interpreter and its own GIL. This guide walks through running two (and more) processes in Python, from the basics up through the internal mechanics that explain why processes behave so differently from threads.

Why Processes Instead of Threads

I covered this in more depth in my multithreading guide, but the short version bears repeating here since it’s the entire reason multiprocessing exists: CPython’s Global Interpreter Lock prevents true parallel execution of Python bytecode across threads within a single process. multiprocessing solves this by launching entirely separate OS processes, each with its own independent Python interpreter, its own memory space, and critically, its own GIL. This means CPU-bound work genuinely runs in parallel across multiple CPU cores when using processes.

The trade-off is that separate memory spaces mean processes can’t share data directly the way threads can — anything passed between processes has to be serialized (pickled) and sent through inter-process communication mechanisms, which adds overhead that threading doesn’t have.

Running Your First Two Processes

import multiprocessing
import time

def worker(name):
    print(f"Process {name} starting")
    time.sleep(2)
    print(f"Process {name} finished")

if __name__ == '__main__':
    p1 = multiprocessing.Process(target=worker, args=("A",))
    p2 = multiprocessing.Process(target=worker, args=("B",))

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    print("Both processes have completed")

Output:

Process A starting
Process B starting
Process A finished
Process B finished
Both processes have completed

Both processes start almost simultaneously and run their two-second sleep concurrently, so the whole thing finishes in roughly 2 seconds rather than 4.

Why the if name == ‘main‘: Guard Is Mandatory

This is the single most common mistake I see (and made myself) when starting with multiprocessing, and it deserves a proper explanation rather than just “always do this.” On Windows (and on macOS since Python 3.8, where the default start method changed to spawn), creating a new process works by launching a fresh Python interpreter and having it re-import your main script as a module to find the target function.

If your process-creation code isn’t guarded by if __name__ == '__main__':, that re-import would trigger the process-creation code again inside the new child process — which would then try to spawn its own children, which would import the script again, and so on, causing a runaway recursive explosion of processes. The guard prevents this by ensuring the process-spawning code only runs when the script is executed directly, not when it’s imported as a module by a child process.

# WITHOUT the guard, this can cause infinite process spawning on Windows/macOS
import multiprocessing

def worker():
    print("Working")

p = multiprocessing.Process(target=worker)  # DANGEROUS at module level
p.start()

I now write this guard reflexively in every multiprocessing script, regardless of platform, since it’s required for correctness on Windows/macOS and doesn’t hurt anything on Linux.

Start Methods: fork, spawn, and forkserver

Understanding how a new process is created explains a lot of multiprocessing’s quirks. Python supports three start methods:

import multiprocessing

if __name__ == '__main__':
    print("Available start methods:", multiprocessing.get_all_start_methods())
    print("Current default:", multiprocessing.get_start_method())

I explicitly set the start method when I need consistent behavior across platforms:

import multiprocessing

if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')
    # ... rest of the program

Passing Data Between Processes: Queue

Since processes don’t share memory, I need explicit communication channels. multiprocessing.Queue is the one I reach for most often — it’s process-safe and handles the pickling/unpickling of data automatically.

import multiprocessing

def producer(q):
    for i in range(5):
        q.put(i)
    q.put(None)  # sentinel to signal completion

def consumer(q):
    while True:
        item = q.get()
        if item is None:
            break
        print(f"Received: {item}")

if __name__ == '__main__':
    q = multiprocessing.Queue()
    p1 = multiprocessing.Process(target=producer, args=(q,))
    p2 = multiprocessing.Process(target=consumer, args=(q,))

    p1.start()
    p2.start()
    p1.join()
    p2.join()

Sharing State: Value and Array

For simple shared numeric data, multiprocessing.Value and multiprocessing.Array provide shared memory that multiple processes can read and write directly, backed by actual OS-level shared memory rather than message passing.

import multiprocessing

def increment(shared_counter, lock):
    for _ in range(100000):
        with lock:
            shared_counter.value += 1

if __name__ == '__main__':
    counter = multiprocessing.Value('i', 0)  # 'i' = signed integer
    lock = multiprocessing.Lock()

    p1 = multiprocessing.Process(target=increment, args=(counter, lock))
    p2 = multiprocessing.Process(target=increment, args=(counter, lock))

    p1.start()
    p2.start()
    p1.join()
    p2.join()

    print(f"Final counter value: {counter.value}")

Just like with threads, shared mutable state across processes needs a lock to avoid race conditions — the fact that it’s a separate process rather than a thread doesn’t eliminate the need for synchronization when multiple execution contexts modify the same shared value.

Returning Results from Processes

Unlike threads, where I can just store a result as an instance attribute and read it after join(), a Process object doesn’t give me direct access to its target function’s return value, because the function actually ran in a completely separate memory space. I need an explicit channel to get data back.

import multiprocessing

def compute_square(number, result_queue):
    result_queue.put(number ** 2)

if __name__ == '__main__':
    result_queue = multiprocessing.Queue()
    p1 = multiprocessing.Process(target=compute_square, args=(5, result_queue))
    p2 = multiprocessing.Process(target=compute_square, args=(7, result_queue))

    p1.start()
    p2.start()
    p1.join()
    p2.join()

    results = [result_queue.get() for _ in range(2)]
    print(f"Results: {results}")

The Simpler Way for Common Cases: Pool

For running the same function across multiple inputs — the most common pattern I actually need in practice — multiprocessing.Pool is far less boilerplate than manually managing individual Process objects.

import multiprocessing

def square(n):
    return n * n

if __name__ == '__main__':
    with multiprocessing.Pool(processes=2) as pool:
        results = pool.map(square, [1, 2, 3, 4, 5])
    print(results)

Output:

[1, 4, 9, 16, 25]

Pool manages a fixed number of worker processes internally, distributing work items across them and handling all the queue-based communication for me. I use this constantly for embarrassingly parallel tasks — applying the same transformation across a large list of independent items.

Demonstrating Real Parallel Speedup

Here’s the comparison that convinced me multiprocessing was worth the added complexity for CPU-bound work:

import multiprocessing
import time

def cpu_heavy(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

if __name__ == '__main__':
    numbers = [10_000_000] * 4

    start = time.perf_counter()
    results_sequential = [cpu_heavy(n) for n in numbers]
    print(f"Sequential: {time.perf_counter() - start:.2f}s")

    start = time.perf_counter()
    with multiprocessing.Pool(processes=4) as pool:
        results_parallel = pool.map(cpu_heavy, numbers)
    print(f"Parallel (4 processes): {time.perf_counter() - start:.2f}s")

On a 4-core machine, the parallel version typically runs close to 3-4 times faster than sequential — a speedup threading simply cannot deliver for CPU-bound work in CPython, precisely because processes escape the GIL’s single-interpreter limitation.

Performance Considerations: When Processes Aren’t Worth It

Spawning a process has real overhead — starting a new interpreter (especially with spawn), and pickling/unpickling data to move it between processes, both take measurable time. For small, fast tasks, this overhead can exceed any parallelism benefit.

import multiprocessing
import time

def tiny_task(n):
    return n + 1

if __name__ == '__main__':
    start = time.perf_counter()
    results = [tiny_task(n) for n in range(1000)]
    print(f"Sequential: {time.perf_counter() - start:.6f}s")

    start = time.perf_counter()
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(tiny_task, range(1000))
    print(f"Parallel: {time.perf_counter() - start:.6f}s")

For work this small, sequential execution is almost always faster — the process pool overhead dwarfs the actual computation. I reserve multiprocessing for tasks substantial enough that the parallelism gain clearly outweighs the process-management overhead.

Common Mistakes I’ve Made

Real-World Use Cases

  1. Parallel data processing — applying the same CPU-intensive transformation across a large dataset split into chunks.
  2. Image or video processing pipelines where each frame or file can be processed independently.
  3. Running multiple independent simulations simultaneously to make full use of available CPU cores.
  4. Batch scientific computation where NumPy/SciPy calculations need to scale beyond single-core performance.

FAQs

What’s the real difference between multiprocessing and threading? Threads share memory and are limited by the GIL for CPU-bound work; processes have separate memory (requiring explicit communication) but achieve genuine parallelism across CPU cores.

Why do I need if name == ‘main’: for multiprocessing but not threading? Because process creation on Windows/macOS re-imports the main script in the child process, and without the guard, that re-import would re-trigger process-spawning code, causing a runaway spawn loop.

Can processes share a Python list directly? Not by default. Regular Python objects aren’t shared across process boundaries; use multiprocessing.Manager() for shared, proxy-based objects like lists and dicts, or Queue/Pipe for explicit message passing.

When should I use Pool instead of individual Process objects? When applying the same function across many independent inputs — Pool handles worker management and result collection with far less boilerplate than manual Process objects.

Is multiprocessing always faster than a single process? No — for small or fast tasks, the overhead of process creation and inter-process communication can make multiprocessing slower than simply running the code sequentially.

Summary

Running processes in Python instead of threads is the right move specifically when CPU-bound work needs genuine parallelism across multiple cores, something threading can’t deliver in CPython due to the GIL. The trade-off is added complexity — separate memory spaces mean explicit communication via Queue, Value, Array, or Pool, and platform-specific quirks like the mandatory __name__ == '__main__' guard. Once I understood why that guard exists and how fork versus spawn actually creates a new process, multiprocessing stopped feeling like unpredictable magic and became a tool I could reason about confidently.

References

Exit mobile version