Multithreading was one of those topics I avoided for longer than I should have, mostly because the Global Interpreter Lock (GIL) discourse online made it sound like threading in Python was pointless. Once I actually started using it for the right kinds of problems — I/O-bound work specifically — it became one of the most practical tools in my everyday scripting. This guide walks through everything from creating your first thread to the synchronization primitives that keep shared data safe, along with an honest look at when threading actually helps and when it doesn’t.
What a Thread Actually Is
A thread is a separate sequence of execution within the same process, sharing that process’s memory space with all other threads in it. This shared memory is both threading’s biggest advantage (no need to serialize data between threads, unlike separate processes) and its biggest danger (multiple threads can corrupt shared data if they modify it simultaneously without coordination).
Python’s threading module, part of the standard library, is my entry point for all of this.
import threading
Creating and Starting a Thread
The simplest way I create a thread is passing a target function to threading.Thread.
import threading
import time
def greet(name):
time.sleep(1)
print(f"Hello, {name}!")
thread = threading.Thread(target=greet, args=("Alice",))
thread.start()
print("Main thread continues immediately, doesn't wait for greet()")
thread.join() # now the main thread waits here until greet() finishes
print("Main thread confirms greet() has completed")
Output:
Main thread continues immediately, doesn't wait for greet()
Hello, Alice!
Main thread confirms greet() has completed
start() launches the thread and returns immediately — it does not wait for the thread’s function to finish. join() is what actually blocks the calling thread until the target thread completes. I’ve seen people forget join() and wonder why their program exits before background work finishes; without it, there’s nothing forcing the main thread to wait.
Creating Threads via Subclassing
For threads that need more internal state or multiple methods, I subclass Thread directly and override run().
import threading
import time
class DownloadWorker(threading.Thread):
def __init__(self, url):
super().__init__()
self.url = url
self.result = None
def run(self):
time.sleep(1) # simulating a network call
self.result = f"Downloaded content from {self.url}"
worker = DownloadWorker("https://example.com")
worker.start()
worker.join()
print(worker.result)
I call super().__init__() here without fail — skipping it is a mistake I made once, and it leaves the Thread object in a broken, unusable state since none of its internal bookkeeping gets initialized.
Running Multiple Threads Concurrently
import threading
import time
def worker(worker_id):
print(f"Worker {worker_id} starting")
time.sleep(2)
print(f"Worker {worker_id} finished")
threads = []
for i in range(5):
t = threading.Thread(target=worker, args=(i,))
threads.append(t)
t.start()
for t in threads:
t.join()
print("All workers have completed")
Because I start all five threads before joining any of them, they run concurrently — the whole thing finishes in roughly 2 seconds total rather than 10, since the time.sleep(2) calls overlap across threads rather than executing one after another.
The Global Interpreter Lock: What It Actually Means for You
This is the concept that confused me most starting out, so I’ll explain it as plainly as I can. CPython (the standard Python implementation) has a Global Interpreter Lock that ensures only one thread executes Python bytecode at any given instant, even on a multi-core machine. This means Python threads do not achieve true parallelism for CPU-bound work — running four CPU-heavy computations across four threads won’t run meaningfully faster than running them one after another, because the GIL forces them to take turns.
import threading
import time
def cpu_bound_work():
total = 0
for i in range(30_000_000):
total += i
return total
start = time.perf_counter()
cpu_bound_work()
cpu_bound_work()
print(f"Sequential: {time.perf_counter() - start:.2f}s")
start = time.perf_counter()
t1 = threading.Thread(target=cpu_bound_work)
t2 = threading.Thread(target=cpu_bound_work)
t1.start(); t2.start()
t1.join(); t2.join()
print(f"Threaded: {time.perf_counter() - start:.2f}s")
On my machine, these two timings come out roughly the same, sometimes with threading even slightly slower due to thread-switching overhead — a clear demonstration that threading doesn’t speed up CPU-bound work in CPython.
But here’s the part that makes threading genuinely valuable: the GIL is released during I/O operations — network requests, file reads, database queries, time.sleep(). While one thread is waiting on a slow network response, the GIL is free, and another thread can run Python code during that wait. This is why threading shines specifically for I/O-bound work.
import threading
import time
def io_bound_work():
time.sleep(2) # simulates waiting on network/disk, GIL is released during this
start = time.perf_counter()
threads = [threading.Thread(target=io_bound_work) for _ in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"5 concurrent I/O waits took: {time.perf_counter() - start:.2f}s")
This finishes in about 2 seconds, not 10, because all five sleep() calls overlap — the GIL isn’t a bottleneck here since none of these threads need to execute Python bytecode simultaneously; they’re all just waiting.
Race Conditions: Why Shared Data Needs Protection
Even with the GIL, race conditions are very real in Python threading, because many operations that look like a single step are actually multiple bytecode instructions under the hood.
import threading
counter = 0
def increment():
global counter
for _ in range(1_000_000):
counter += 1
threads = [threading.Thread(target=increment) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"Expected 2000000, got {counter}")
Running this repeatedly, I almost never get exactly 2,000,000. counter += 1 isn’t atomic — it involves reading the current value, adding one, and writing it back, and the GIL can switch between threads in the middle of that sequence, causing lost updates when two threads interleave badly.
Synchronization with Lock
The fix is a threading.Lock, which ensures only one thread can execute a critical section of code at a time.
import threading
counter = 0
lock = threading.Lock()
def safe_increment():
global counter
for _ in range(1_000_000):
with lock:
counter += 1
threads = [threading.Thread(target=safe_increment) for _ in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()
print(f"Expected 2000000, got {counter}")
With the lock in place, this now reliably prints exactly 2,000,000 every time. The with lock: block guarantees that the read-modify-write sequence for counter += 1 completes fully in one thread before another thread can begin its own increment.
Other Synchronization Primitives
Beyond Lock, the threading module provides several other tools I reach for depending on the situation:
import threading
# RLock - a lock that the same thread can acquire multiple times (reentrant)
rlock = threading.RLock()
# Semaphore - limits how many threads can access a resource simultaneously
semaphore = threading.Semaphore(3) # max 3 threads at once
# Condition - lets threads wait for a specific condition to become true
condition = threading.Condition()
# Barrier - makes a group of threads wait until all of them reach a point
barrier = threading.Barrier(3)
I use Semaphore most often after Lock — it’s perfect for rate-limiting, like capping how many simultaneous connections a script makes to an external API.
import threading
import time
semaphore = threading.Semaphore(2) # only 2 threads can run this block at once
def limited_task(task_id):
with semaphore:
print(f"Task {task_id} started")
time.sleep(1)
print(f"Task {task_id} finished")
threads = [threading.Thread(target=limited_task, args=(i,)) for i in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
Thread-Safe Data Structures: Queue
Rather than manually managing locks around shared lists or dicts, I reach for queue.Queue whenever threads need to hand off data to each other — it’s built to be thread-safe internally, with all locking already handled for me.
import threading
import queue
import time
def producer(q):
for i in range(5):
q.put(i)
print(f"Produced {i}")
time.sleep(0.5)
def consumer(q):
while True:
item = q.get()
if item is None:
break
print(f"Consumed {item}")
q.task_done()
q = queue.Queue()
prod_thread = threading.Thread(target=producer, args=(q,))
cons_thread = threading.Thread(target=consumer, args=(q,))
prod_thread.start()
cons_thread.start()
prod_thread.join()
q.put(None) # sentinel value to signal the consumer to stop
cons_thread.join()
Common Mistakes I’ve Made
- Expecting threading to speed up CPU-bound code and being confused when it didn’t — that’s the GIL’s doing, and
multiprocessingis the right tool there instead. - Forgetting to lock shared mutable state, producing subtle, hard-to-reproduce race condition bugs that only show up occasionally.
- Holding a lock for too long, serializing work that didn’t need to be serialized and eliminating the concurrency benefit entirely.
- Deadlocking by acquiring multiple locks in inconsistent order across different threads.
- Forgetting
.join(), letting the main program exit while background threads were still mid-task.
Real-World Use Cases
- Concurrent network requests — fetching data from multiple APIs or URLs simultaneously rather than sequentially.
- Responsive GUI applications — running long tasks on a background thread so the UI thread doesn’t freeze.
- Producer-consumer pipelines — one thread reading data while another processes it.
- Rate-limited concurrent downloads, using a semaphore to cap simultaneous connections.
FAQs
Does Python threading give true parallelism? Not for CPU-bound work, due to the GIL. For I/O-bound work, threads genuinely overlap their waiting time, which provides real concurrency benefits even without CPU parallelism.
When should I use multiprocessing instead of threading? For CPU-bound tasks that need to use multiple cores simultaneously — multiprocessing sidesteps the GIL entirely by using separate processes, each with its own interpreter and memory space.
What’s the difference between Lock and RLock? A Lock can only be acquired once before it must be released; trying to acquire it again from the same thread deadlocks. An RLock (reentrant lock) can be acquired multiple times by the same thread without deadlocking, as long as it’s released the same number of times.
Why did my counter come out wrong even with the GIL? The GIL guarantees only one thread executes bytecode at a time, but counter += 1 compiles to multiple bytecode instructions, and the GIL can switch threads between those instructions, causing lost updates without explicit locking.
Is Queue faster than manually locking a list? queue.Queue handles its own internal locking efficiently and is the recommended, safer choice for producer-consumer patterns rather than manually coordinating access to a shared list.
Summary
Multithreading in Python is genuinely useful, but only once you understand what the GIL does and doesn’t allow — real concurrency for I/O-bound waiting, no real parallelism for CPU-bound computation. Creating and joining threads is the easy part; the skill that actually matters is recognizing when shared state needs a Lock, Semaphore, or thread-safe structure like Queue to avoid race conditions. Once that clicked for me, threading went from a source of confusing, intermittent bugs to a reliable tool I reach for constantly in I/O-heavy scripts.
References
- Python Official Documentation: threading — Thread-based parallelism
- Python Official Documentation: queue — A synchronized queue class
- Python Official Documentation: Global Interpreter Lock (glossary)
