I reach for Selenium specifically when a website refuses to give up its data any other way — when the content I need only appears after JavaScript runs, after a button click, or after scrolling triggers a lazy-loaded section. Plain HTTP requests and HTML parsing can’t do any of that, because they never actually execute JavaScript or render a page the way a real browser does. Selenium solves this by driving an actual browser programmatically. This guide covers everything I’ve learned setting up, using, and troubleshooting Selenium for web scraping, along with the practices that keep scripts reliable rather than randomly breaking.
Why Selenium Instead of requests + BeautifulSoup
I want to be upfront about this trade-off, because reaching for Selenium when it isn’t needed makes scripts slower and more fragile than necessary. Libraries like requests fetch raw HTML exactly as the server sends it — before any JavaScript executes. For traditional server-rendered pages, that’s everything you need, and parsing it with BeautifulSoup is faster and simpler than running a whole browser.
Selenium becomes necessary specifically when the content is generated or modified by JavaScript after the initial page load — single-page applications built with React, Vue, or Angular, infinite-scroll feeds, content behind a login form, or data that only appears after interacting with the page (clicking a “load more” button, filling a search box, waiting for an AJAX call to finish).
Setting Up Selenium
pip install selenium --break-system-packages
Since Selenium 4, I no longer need to manually download and manage browser driver binaries — Selenium Manager handles that automatically.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://example.com")
print(driver.title)
driver.quit()
Locating Elements: The Core Skill
Everything in Selenium scraping comes down to finding elements on the page and extracting data from them. The By class defines the different strategies for locating elements.
from selenium.webdriver.common.by import By
# By ID - fastest and most reliable when available
element = driver.find_element(By.ID, "main-title")
# By CSS selector - my most-used strategy, flexible and familiar from web dev
element = driver.find_element(By.CSS_SELECTOR, "div.product-card > h2")
# By XPath - powerful for complex or text-based matching
element = driver.find_element(By.XPATH, "//button[contains(text(), 'Submit')]")
# By class name
elements = driver.find_elements(By.CLASS_NAME, "product-item") # note: find_elements (plural)
# By tag name
links = driver.find_elements(By.TAG_NAME, "a")
I default to CSS selectors for most cases since I already know that syntax well from front-end work, and reach for XPath specifically when I need to match based on text content or navigate relationships (like “parent of” or “sibling of”) that CSS selectors can’t express.
A Complete Scraping Example
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
try:
driver.get("https://example.com/products")
# wait explicitly for product elements to actually appear before scraping
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".product-card")))
products = driver.find_elements(By.CSS_SELECTOR, ".product-card")
results = []
for product in products:
name = product.find_element(By.CSS_SELECTOR, "h2.product-name").text
price = product.find_element(By.CSS_SELECTOR, "span.price").text
results.append({"name": name, "price": price})
for item in results:
print(item)
finally:
driver.quit()
I wrap the entire scraping logic in a try/finally block so driver.quit() always runs, even if an exception occurs mid-scrape — leaving orphaned browser processes running in the background is a mistake I made constantly before adopting this habit, and on a long-running scraping script, those leaked processes add up fast.
Explicit Waits: The Single Most Important Selenium Concept
This is the concept that separates reliable Selenium scripts from flaky ones. Because JavaScript-rendered content appears asynchronously, the page might not have finished loading the data I need at the exact moment my script tries to find it. WebDriverWait combined with expected_conditions polls the page repeatedly until a condition becomes true, rather than assuming content is immediately available.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
wait = WebDriverWait(driver, 15)
# Wait until a specific element is present in the DOM
element = wait.until(EC.presence_of_element_located((By.ID, "results")))
# Wait until an element is not just present, but actually visible
element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "loaded-content")))
# Wait until an element is clickable (present, visible, and enabled)
button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.load-more")))
button.click()
I never use time.sleep(5) as a substitute for explicit waits anymore — it’s both unreliable (the page might need longer on a slow connection, or might be ready faster) and unnecessarily slow (waiting a fixed 5 seconds even when the content loaded in 1). Explicit waits check repeatedly and proceed the moment the condition is actually met, which is both faster on average and far more robust against variable page-load timing.
Implicit Waits: A Simpler, Blunter Alternative
Selenium also offers implicit waits, set once for the entire driver session, which apply a default timeout to every find_element call.
driver.implicitly_wait(10) # applies to all subsequent find_element calls
I generally avoid mixing implicit and explicit waits in the same script, since Selenium’s documentation specifically warns this can cause unpredictable, inconsistent wait times. I pick one strategy — almost always explicit waits, since they’re more precise about exactly what condition I’m waiting for — and stick with it throughout a script.
Interacting with Pages: Clicks, Forms, and Scrolling
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
search_box = driver.find_element(By.NAME, "q")
search_box.send_keys("python web scraping")
search_box.send_keys(Keys.RETURN)
login_button = driver.find_element(By.ID, "login-btn")
login_button.click()
# Scroll to the bottom of the page to trigger lazy-loaded content
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
execute_script() lets me run arbitrary JavaScript directly in the page context, which I use for scrolling, and occasionally for extracting data that’s easier to grab via a JS expression than through Selenium’s own element API.
Handling Pagination and Infinite Scroll
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
all_items = []
previous_height = driver.execute_script("return document.body.scrollHeight")
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# give the page a moment to load new content after scrolling
WebDriverWait(driver, 5).until(
lambda d: d.execute_script("return document.body.scrollHeight") != previous_height
or True
)
time.sleep(1.5) # brief pause; a hybrid approach since scroll-triggered loads vary
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == previous_height:
break # no new content loaded, we've reached the bottom
previous_height = new_height
items = driver.find_elements(By.CSS_SELECTOR, ".item")
print(f"Collected {len(items)} items total")
Infinite scroll is genuinely one of the trickier scraping scenarios, since there’s no fixed number of “pages” — I keep scrolling and checking whether the page’s height actually grew, stopping once it stops changing.
Running Headless (Without a Visible Browser Window)
For scripts running on a server or in automation pipelines, I don’t want an actual browser window popping up.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1920,1080")
driver = webdriver.Chrome(options=options)
driver.get("https://example.com")
I always set an explicit --window-size when running headless, since some sites serve different (often stripped-down) layouts for very small or unset viewport sizes, which can change where elements appear or whether they render at all.
Internal Working: How Selenium Actually Controls the Browser
Selenium communicates with the browser through the WebDriver protocol, a standardized, W3C-specified HTTP-based API. When I call driver.find_element(...), Selenium’s Python bindings send an HTTP request to a local WebDriver server (like chromedriver), which translates that request into low-level browser automation commands, executes them against the actual running browser instance, and sends the result back as an HTTP response. This is fundamentally different from requests, which just fetches raw bytes over HTTP with no browser or JavaScript engine involved at all — Selenium is automating a genuine, full browser rendering engine, which is exactly why it can see JavaScript-generated content that requests never will.
This architecture also explains why Selenium scripts are inherently slower than pure HTTP scraping — every interaction involves a real browser actually rendering, executing JavaScript, and responding, rather than a lightweight text-based HTTP exchange.
Error Handling for Common Failures
from selenium.common.exceptions import (
NoSuchElementException,
TimeoutException,
StaleElementReferenceException
)
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
try:
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.ID, "content")))
print(element.text)
except TimeoutException:
print("Element did not appear within the timeout period.")
except NoSuchElementException:
print("Element not found on the page.")
except StaleElementReferenceException:
print("Element reference is stale - the page likely changed since it was located.")
StaleElementReferenceException is one that confused me the first several times I hit it — it happens when I locate an element, the page then re-renders that portion of the DOM (common in JavaScript-heavy apps), and I try to interact with my now-outdated reference to an element that technically no longer exists in the current DOM. The fix is almost always to re-locate the element immediately before interacting with it, rather than holding onto a reference across other actions that might trigger a re-render.
Being a Responsible Scraper
I always check a site’s robots.txt and terms of service before scraping, add reasonable delays between actions rather than hammering a server as fast as possible, and avoid scraping data behind authentication unless I have explicit permission to access and use it that way. Selenium is a powerful tool, and with that power comes the responsibility to not degrade a service for other users or violate a site’s stated usage policies.
Common Mistakes I’ve Made
- Using
time.sleep()everywhere instead of explicit waits, producing scripts that are both slow and unreliable. - Not calling
driver.quit(), leaving zombie browser processes accumulating on the machine. - Holding stale element references across actions that re-render the page.
- Scraping without checking
robots.txtor terms of service first. - Not running headless in production/CI environments, causing scripts to fail outright on servers without a display.
FAQs
Do I need to install ChromeDriver separately? Not since Selenium 4.6+ — Selenium Manager automatically detects your installed browser version and downloads a matching driver.
Why does my script fail with “element not interactable”? Usually the element exists in the DOM but isn’t visible or enabled yet — use EC.element_to_be_clickable instead of just checking presence, and confirm nothing (like a cookie banner) is covering it.
Is Selenium slower than requests-based scraping? Yes, significantly — it’s running and rendering a full browser rather than just fetching raw bytes, so I only reach for it when JavaScript rendering is genuinely necessary.
How do I scrape a site that requires login? Automate the login form submission with Selenium itself (locate the username/password fields, fill them, submit), then proceed to scrape authenticated pages, respecting the site’s terms of service.
What’s the difference between find_element and find_elements? find_element (singular) returns the first match or raises NoSuchElementException if none exists. find_elements (plural) returns a list of all matches, or an empty list if none are found.
Summary
Selenium WebDriver earns its complexity specifically for the scraping scenarios that plain HTTP requests can’t handle — JavaScript-rendered content, interactive forms, infinite scroll, and authenticated sessions. The core skill is locating elements reliably and waiting for content properly using explicit waits rather than arbitrary sleeps, and understanding that Selenium is genuinely driving a full browser through the WebDriver protocol explains both its power and its inherent slowness compared to lighter scraping tools. Used responsibly and with proper error handling around the DOM’s asynchronous, ever-changing nature, it remains one of the most capable tools I have for extracting data the web doesn’t hand over easily.
References
- Selenium Official Documentation: https://www.selenium.dev/documentation/
- Selenium Python Bindings Documentation: https://selenium-python.readthedocs.io/
- W3C WebDriver Specification: https://www.w3.org/TR/webdriver/
