I didn’t touch Python’s set type at all for the first year I was learning the language. I just used lists for everything, including cases where I only cared about unique values and membership checks. Once I actually needed to deduplicate a huge list of records efficiently, and realized my list-based approach was painfully slow, I finally learned why sets exist as their own type — and it came down almost entirely to the same hash table foundation that makes dictionaries fast.
What Is a Set?
A set is an unordered collection of unique, hashable elements. No duplicates are allowed, and there’s no guaranteed order to how elements are stored or iterated.
fruits = {"apple", "banana", "cherry"}
print(fruits)
print(type(fruits))
Output (order may vary):
{'banana', 'apple', 'cherry'}
(class 'set')
Creating Sets
literal_set = {1, 2, 3}
from_list = set([1, 2, 2, 3, 3, 3])
empty_set = set()
comprehension_set = {x**2 for x in range(5)}
print(literal_set)
print(from_list)
print(empty_set)
print(comprehension_set)
Output:
{1, 2, 3}
{1, 2, 3}
set()
{0, 1, 4, 9, 16}
Notice that duplicates in from_list were automatically collapsed down to unique values. Also notice you can’t create an empty set with {} — that syntax creates an empty dictionary instead, so you must use set() explicitly.
not_a_set = {}
print(type(not_a_set))
Output:
(class 'dict')
Adding and Removing Elements
colors = {"red", "green"}
colors.add("blue")
print(colors)
colors.remove("red")
print(colors)
colors.discard("purple") # no error even though "purple" isn't present
print(colors)
Output:
{'red', 'green', 'blue'}
{'green', 'blue'}
{'green', 'blue'}
The difference between .remove() and .discard() matters: .remove() raises a KeyError if the element doesn’t exist, while .discard() silently does nothing in that case.
colors = {"red", "green"}
colors.remove("purple")
KeyError: 'purple'
How Sets Work Internally — Same Hash Table as Dictionaries
Sets in CPython are implemented using essentially the same hash table structure that powers dictionaries, just without storing an associated value for each key — think of a set as a dictionary that only cares about keys. When you add an element, Python computes hash(element) and uses that hash to determine its position in the internal table, exactly the same way a dictionary key is placed.
This is precisely why set membership testing (x in my_set) runs in average O(1) constant time, versus the O(n) linear scan required to check membership in a list.
import timeit
my_list = list(range(100000))
my_set = set(range(100000))
list_time = timeit.timeit(lambda: 99999 in my_list, number=1000)
set_time = timeit.timeit(lambda: 99999 in my_set, number=1000)
print(f"List membership check: {list_time:.5f}s")
print(f"Set membership check: {set_time:.5f}s")
Typical output:
List membership check: 0.85000s
Set membership check: 0.00015s
This is the exact performance gap that made me switch to sets for any deduplication or repeated membership-checking task. And, exactly like dictionary keys, set elements must be hashable — meaning immutable types like strings, numbers, and tuples work fine, but lists and dictionaries cannot be added to a set.
bad_set = {[1, 2], [3, 4]}
TypeError: unhashable type: 'list'
Mathematical Set Operations
This is where sets genuinely shine — Python implements real mathematical set theory operations directly as built-in methods and operators.
a = {1, 2, 3, 4}
b = {3, 4, 5, 6}
print(a | b) # union
print(a & b) # intersection
print(a - b) # difference
print(a ^ b) # symmetric difference
Output:
{1, 2, 3, 4, 5, 6}
{3, 4}
{1, 2}
{1, 2, 5, 6}
Each operator has an equivalent named method: .union(), .intersection(), .difference(), and .symmetric_difference(). I tend to use the operators for readability in simple expressions, and the method form when I need to pass in keyword-friendly chained operations across multiple sets.
a = {1, 2, 3}
b = {2, 3, 4}
c = {3, 4, 5}
print(a.union(b, c))
print(a.intersection(b, c))
Output:
{1, 2, 3, 4, 5}
{3}
Subset, Superset, and Disjoint Checks
a = {1, 2, 3}
b = {1, 2, 3, 4, 5}
print(a.issubset(b))
print(b.issuperset(a))
print(a.isdisjoint({7, 8, 9}))
Output:
True
True
True
These methods map directly to formal set theory concepts, and I’ve used them extensively when working on permission systems — checking whether a user’s granted roles are a subset of the roles required for a given action, for example.
Frozensets: The Immutable Set
Just as tuples are the immutable counterpart to lists, frozenset is the immutable counterpart to set. Because it’s immutable and therefore hashable, a frozenset can itself be used as a dictionary key or as an element inside another set — something a regular mutable set cannot do.
regular_set = {1, 2, 3}
frozen = frozenset([1, 2, 3])
print(frozen)
print(type(frozen))
# This works because frozenset is hashable
nested = {frozenset([1, 2]), frozenset([3, 4])}
print(nested)
Output:
frozenset({1, 2, 3})
(class 'frozenset')
{frozenset({1, 2}), frozenset({3, 4})}
Attempting the same thing with a regular set fails, exactly as expected given the hashability rules discussed earlier:
nested = {set([1, 2]), set([3, 4])}
TypeError: unhashable type: 'set'
Iterating Over Sets
tags = {"python", "programming", "coding"}
for tag in tags:
print(tag)
Output (order not guaranteed, and may differ between runs):
python
coding
programming
Unlike dictionaries, which have guaranteed insertion order since Python 3.7, sets make no ordering guarantee at all. If order matters to your logic, a set is the wrong tool — reach for a list or an ordered structure instead.
Real-World and Automation Use Cases
- Deduplicating data: removing repeated entries from a large dataset quickly and cleanly.
- Membership testing at scale: checking whether a value exists in a large collection, especially inside loops where this check happens repeatedly.
- Finding differences between two datasets: comparing an old list of user IDs against a new one to detect additions and removals.
- Tag and permission systems: representing a user’s assigned tags or permissions as a set, and using intersection/subset checks for access control logic.
old_ids = {101, 102, 103, 104}
new_ids = {102, 103, 105, 106}
added = new_ids - old_ids
removed = old_ids - new_ids
print(f"Added: {added}")
print(f"Removed: {removed}")
Output:
Added: {105, 106}
Removed: {101, 104}
Here’s a practical deduplication example, the kind I run constantly on log files or CSV exports:
emails = ["a@x.com", "b@x.com", "a@x.com", "c@x.com", "b@x.com"]
unique_emails = list(set(emails))
print(unique_emails)
print(f"Removed {len(emails) - len(unique_emails)} duplicates")
Output (order may vary):
['c@x.com', 'a@x.com', 'b@x.com']
Removed 2 duplicates
Note that converting to a set and back to a list loses the original order. If preserving order while deduplicating matters, I use a slightly different approach with dict.fromkeys(), which relies on the dictionary’s guaranteed insertion order:
emails = ["a@x.com", "b@x.com", "a@x.com", "c@x.com", "b@x.com"]
ordered_unique = list(dict.fromkeys(emails))
print(ordered_unique)
Output:
['a@x.com', 'b@x.com', 'c@x.com']
Best Practices
- Use sets when you need fast membership testing or automatic deduplication, and don’t care about order.
- Use
frozensetwhen you need an immutable, hashable set — for example, as a dictionary key or as an element within another set. - Use
.discard()instead of.remove()when you’re not sure an element exists and don’t want to handle aKeyError. - Convert to a list only at the very end, if output order matters for display or serialization.
Common Mistakes
Assuming {} creates an empty set is a very common trap, as shown earlier — it actually creates an empty dictionary. Another common mistake is assuming sets preserve insertion order the same way dictionaries do since Python 3.7 — they don’t, and code that silently relies on set ordering can behave unpredictably across different runs or Python versions.
FAQs
How is a set different from a list? A set stores only unique, hashable elements with no guaranteed order and offers O(1) average membership testing. A list preserves order, allows duplicates, and has O(n) membership testing.
How do I create an empty set? Use set(). Using {} creates an empty dictionary instead.
Can a set contain a list? No. Set elements must be hashable, and lists are mutable and therefore unhashable. Use a tuple instead if you need a sequence-like element inside a set.
What’s the difference between set and frozenset? A set is mutable — you can add and remove elements. A frozenset is immutable and hashable, making it usable as a dictionary key or as an element within another set.
How do I remove duplicates from a list while preserving order? Use list(dict.fromkeys(my_list)), which relies on the dictionary’s guaranteed insertion-order behavior rather than a plain set conversion.
Summary
Sets bring real mathematical set theory — union, intersection, difference, subset checks — directly into everyday Python code, backed by the same hash table implementation that makes dictionaries fast. That shared foundation is exactly why membership testing and deduplication with a set vastly outperform the equivalent operations on a list at scale. Once I understood that a set is essentially “a dictionary that only stores keys,” both its performance characteristics and its hashability requirements made complete sense, and sets quickly became one of my go-to tools for cleaning and comparing real-world data.
References
- Python official documentation on set types: https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset
- Python tutorial on sets: https://docs.python.org/3/tutorial/datastructures.html#sets
- Python Time Complexity wiki (official): https://wiki.python.org/moin/TimeComplexity
