Of all the branches of machine learning I’ve studied, reinforcement learning is the one that feels the most like watching something actually “learn” in a way that resembles how humans and animals do — through trial, error, and reward. In this article, I want to explain what reinforcement learning is at its core, how it fundamentally differs from supervised and unsupervised learning, and how neural networks have become central to modern reinforcement learning through an approach called deep reinforcement learning.
What Is Reinforcement Learning?
Reinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by interacting with an environment, receiving rewards or penalties based on the actions it takes, and adjusting its behavior over time to maximize cumulative reward.
Unlike supervised learning, where a model is trained on a fixed dataset of labeled input-output pairs, reinforcement learning involves an ongoing, interactive process: the agent takes an action, the environment responds with a new state and a reward signal, and the agent uses that feedback to improve its future decision-making.
The Core Components of Reinforcement Learning
To understand RL properly, I need to define its key building blocks, which are formalized using a mathematical framework called a Markov Decision Process (MDP).
| Component | Description |
|---|---|
| Agent | The learner or decision-maker |
| Environment | The world the agent interacts with |
| State ($s$) | A representation of the current situation |
| Action ($a$) | A choice the agent can make |
| Reward ($r$) | Feedback signal indicating how good or bad an action was |
| Policy ($\pi$) | The agent’s strategy for choosing actions given a state |
| Value Function ($V$) | Expected cumulative future reward from a given state |
| Q-Function ($Q$) | Expected cumulative future reward from a given state-action pair |
The Reinforcement Learning Loop
graph LR
A[Agent] -->|Action a_t| B[Environment]
B -->|New State s_t+1| A
B -->|Reward r_t| A
At each time step $t$, the agent observes the current state $s_t$, selects an action $a_t$ according to its policy $\pi$, receives a reward $r_t$, and transitions to a new state $s_{t+1}$. This loop repeats, and the agent’s goal is to learn a policy that maximizes its expected cumulative reward over time.
The Mathematical Objective
The agent’s goal is to maximize the expected cumulative discounted reward, often called the return:
$$ G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1} $$
Where $\gamma \in [0, 1]$ is the discount factor, controlling how much the agent values future rewards relative to immediate ones. A $\gamma$ close to 0 makes the agent short-sighted (focused on immediate reward), while a $\gamma$ close to 1 makes the agent value long-term reward almost as much as immediate reward.
The value function under a policy $\pi$ is defined as the expected return starting from state $s$:
$$ V^{\pi}(s) = \mathbb{E}_{\pi}\left[ G_t \mid s_t = s \right] $$
And the Q-function (action-value function) extends this to state-action pairs:
$$ Q^{\pi}(s, a) = \mathbb{E}_{\pi}\left[ G_t \mid s_t = s, a_t = a \right] $$
Where Neural Networks Come In: Deep Reinforcement Learning
Traditional reinforcement learning methods worked well for problems with small, discrete state and action spaces, where value functions could be represented using simple lookup tables (like Q-tables). But real-world problems — like playing video games from raw pixels, controlling robots, or managing complex logistics — often have enormous, continuous, or high-dimensional state spaces, where tabular methods become completely impractical.
This is where neural networks enter the picture. In deep reinforcement learning, a neural network is used as a function approximator to represent the policy, the value function, or the Q-function, allowing RL to scale to problems with massive or continuous state spaces.
Deep Q-Networks (DQN)
Instead of a lookup table, a neural network takes the state $s$ as input and outputs an estimated Q-value for each possible action:
$$ Q(s, a; \theta) \approx Q^*(s, a) $$
Where $\theta$ represents the neural network’s trainable parameters. This was famously demonstrated by DeepMind’s DQN algorithm, which learned to play Atari games directly from raw pixel input, achieving human-level or superhuman performance on many games.
Policy Gradient Methods
Rather than estimating value functions, policy gradient methods use a neural network to directly represent the policy $\pi_\theta(a|s)$ — a probability distribution over actions given a state — and adjust the network’s parameters to increase the probability of actions that lead to higher rewards.
$$ \nabla_\theta J(\theta) = \mathbb{E}{\pi\theta}\left[ \nabla_\theta \log \pi_\theta(a|s) \cdot G_t \right] $$
This is the foundation of algorithms like REINFORCE, and more advanced variants like Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO), which are widely used in modern applications, including reinforcement learning from human feedback (RLHF) used to fine-tune large language models.
Actor-Critic Methods
Actor-critic methods combine both approaches: an “actor” network learns the policy, while a “critic” network learns the value function, providing a more stable and efficient learning signal than pure policy gradient methods alone.
Comparison: Reinforcement Learning vs. Supervised Learning
| Aspect | Supervised Learning | Reinforcement Learning |
|---|---|---|
| Training Data | Fixed, labeled dataset | Sequential experience from interacting with an environment |
| Feedback | Direct label for every input | Reward signal, often sparse and delayed |
| Objective | Minimize prediction error | Maximize cumulative reward |
| Data Distribution | Fixed, i.i.d. assumption | Changes based on agent’s own behavior (non-stationary) |
| Exploration | Not required | Central challenge — agent must explore to discover good strategies |
The Exploration-Exploitation Tradeoff
One of the defining challenges of reinforcement learning, which doesn’t really appear in supervised learning, is the exploration-exploitation tradeoff. The agent must balance:
- Exploitation: Choosing actions known to yield high reward based on past experience
- Exploration: Trying new, potentially suboptimal actions to discover whether they might lead to even better outcomes
Common strategies to manage this tradeoff include epsilon-greedy exploration (choosing a random action with small probability $\epsilon$) and more sophisticated approaches like upper confidence bound (UCB) methods or entropy regularization in policy gradient methods.
A Simple Q-Learning Implementation (Without Deep Learning)
Before jumping into deep RL, it helps to see the tabular version, since it makes the underlying logic much clearer:
import numpy as np
# Simple grid-world Q-learning example
num_states = 6
num_actions = 2 # 0 = left, 1 = right
Q = np.zeros((num_states, num_actions))
learning_rate = 0.1
discount_factor = 0.9
epsilon = 0.1
episodes = 500
def get_reward(state):
return 1 if state == 5 else 0
for episode in range(episodes):
state = 0
while state != 5:
if np.random.rand() < epsilon:
action = np.random.choice(num_actions) # Explore
else:
action = np.argmax(Q[state]) # Exploit
next_state = min(state + 1, 5) if action == 1 else max(state - 1, 0)
reward = get_reward(next_state)
# Q-learning update rule
Q[state, action] += learning_rate * (
reward + discount_factor * np.max(Q[next_state]) - Q[state, action]
)
state = next_state
print("Learned Q-table:")
print(Q)
A Deep Q-Network Sketch in PyTorch
import torch
import torch.nn as nn
class DQN(nn.Module):
def __init__(self, state_dim, action_dim):
super(DQN, self).__init__()
self.network = nn.Sequential(
nn.Linear(state_dim, 128),
nn.ReLU(),
nn.Linear(128, 128),
nn.ReLU(),
nn.Linear(128, action_dim) # Outputs a Q-value per action
)
def forward(self, state):
return self.network(state)
# Example usage
state_dim = 4 # e.g., CartPole environment
action_dim = 2 # e.g., push left or right
model = DQN(state_dim, action_dim)
This network takes a state as input and outputs an estimated Q-value for each possible action — the agent then selects the action with the highest predicted Q-value (or explores randomly, depending on the exploration strategy).
Advantages of Reinforcement Learning
- Capable of learning complex, sequential decision-making strategies without needing labeled data for every decision
- Well-suited to problems involving long-term planning and delayed rewards
- Has achieved remarkable results in domains like game playing (AlphaGo, Atari), robotics, and fine-tuning large language models via RLHF
- Learns directly from interaction, which can uncover novel strategies humans might not have considered
Disadvantages and Limitations
- Often requires an enormous number of interactions (samples) to learn effectively, which can be costly or impractical in real-world settings (e.g., physical robots)
- Reward function design is difficult — poorly designed rewards can lead to unintended or exploitative behaviors (“reward hacking”)
- Training can be unstable, especially when combined with deep neural networks (non-stationary targets, correlated data)
- Exploration in large or continuous action spaces remains a significant challenge
Real-World Use Cases
- Game playing: AlphaGo and AlphaZero (board games), OpenAI Five (Dota 2), DQN (Atari games)
- Robotics: Learning locomotion and manipulation policies for physical robots
- Recommendation systems: Optimizing long-term user engagement rather than immediate clicks
- Large language model fine-tuning: Reinforcement Learning from Human Feedback (RLHF), used to align models like ChatGPT with human preferences
- Resource management: Data center cooling optimization, traffic signal control
Best Practices for Deep Reinforcement Learning
- Design reward functions carefully to avoid unintended behaviors — reward shaping is often more art than science.
- Use experience replay (storing and randomly sampling past experiences) to break correlations between consecutive training samples in DQN-style algorithms.
- Use target networks in Q-learning approaches to stabilize training by preventing the “moving target” problem.
- Normalize state inputs and rewards to help neural network training stability.
- Start with simpler algorithms (like DQN or REINFORCE) before moving to more sophisticated methods like PPO or SAC.
- Monitor exploration behavior to ensure the agent doesn’t converge prematurely to a suboptimal policy.
Summary
Reinforcement learning is a paradigm of machine learning in which an agent learns to make sequential decisions by interacting with an environment and receiving reward signals, with the goal of maximizing cumulative long-term reward. It differs fundamentally from supervised and unsupervised learning in that it involves active interaction, delayed feedback, and an inherent exploration-exploitation tradeoff. Neural networks have become essential to modern reinforcement learning through deep reinforcement learning, where they serve as powerful function approximators for policies, value functions, and Q-functions — enabling RL to scale to complex, high-dimensional problems like game playing, robotics, and the fine-tuning of large language models.
References
- Sutton, R. S., & Barto, A. G. Reinforcement Learning: An Introduction. MIT Press. http://incompleteideas.net/book/the-book-2nd.html
- Mnih, V., et al. (2015). Human-level Control through Deep Reinforcement Learning. Nature. https://www.nature.com/articles/nature14236
- Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. https://arxiv.org/abs/1707.06347
- Silver, D., et al. (2016). Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature.
- OpenAI Spinning Up in Deep RL: https://spinningup.openai.com/
