Offensive Security Researcher & Quantum Cryptography Analyst

How Is Q-Learning Used in Reinforcement Learning?

Abdul Wahab Junaid
How is Q-learning used in reinforcement learning

If you’ve spent any time reading about reinforcement learning, you’ve probably run into Q-learning within the first ten minutes. It’s one of the oldest, most influential, and most widely taught algorithms in the field — and for good reason. It’s simple enough to implement in an afternoon, yet powerful enough to be the conceptual ancestor of Deep Q-Networks (DQN), the algorithm that famously learned to play Atari games at superhuman level.

1. The Core Idea Behind Q-Learning

Q-learning is a model-free, off-policy reinforcement learning algorithm. Let’s unpack those two terms because they matter:

At its heart, Q-learning tries to learn a function called the action-value function, or Q-function, denoted $Q(s, a)$. This function estimates the expected total future reward of taking action $a$ in state $s$, and then acting optimally afterward.

2. The Q-Function, Formally

$$ Q^\pi(s, a) = \mathbb{E}_\pi \left[ G_t \mid S_t = s, A_t = a \right] $$

where $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$ is the discounted return, and $\gamma$ is the discount factor.

The optimal Q-function is defined as:

$$ Q^*(s, a) = \max_\pi Q^\pi(s, a) $$

Once we know $Q^*(s,a)$ for every state-action pair, the optimal policy is trivial to extract:

$$ \pi^(s) = \arg\max_a Q^(s, a) $$

In other words, in any given state, just pick whichever action has the highest Q-value.

3. The Bellman Optimality Equation for Q

Q-learning is built directly on the Bellman Optimality Equation:

$$ Q^(s, a) = \mathbb{E}\left[ R_{t+1} + \gamma \max_{a’} Q^(S_{t+1}, a’) \mid S_t = s, A_t = a \right] $$

This says: the optimal value of taking action $a$ in state $s$ equals the expected immediate reward, plus the discounted value of acting optimally from the next state onward.

4. The Q-Learning Update Rule

Since we usually don’t know the environment’s dynamics, Q-learning estimates $Q^*(s,a)$ iteratively using observed transitions $(s, a, r, s’)$:

$$ Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \Big[ R_{t+1} + \gamma \max_{a’} Q(S_{t+1}, a’) – Q(S_t, A_t) \Big] $$

Breaking this down:

This is the entire algorithm. Every time the agent takes an action and observes a reward and next state, it nudges its Q-value estimate a little closer to the new target.

5. Why Is Q-Learning “Off-Policy”?

Notice the $\max_{a’}$ term in the update rule. This means that regardless of which action the agent actually takes next (which might be an exploratory, random action), the update always assumes the agent will act greedily afterward. This decoupling between “the policy used to collect data” (called the behavior policy) and “the policy being learned” (called the target policy) is what makes Q-learning off-policy.

This is different from SARSA, an on-policy alternative, whose update rule uses the actual next action taken:

$$ Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \Big[ R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) – Q(S_t, A_t) \Big] $$

FeatureQ-LearningSARSA
Policy typeOff-policyOn-policy
Update target$\max_{a’} Q(s’, a’)$$Q(s’, a’)$ using actual next action
BehaviorLearns optimal policy regardless of explorationLearns value of the policy actually being followed
Risk sensitivityCan learn riskier optimal pathsTends to learn safer paths (accounts for exploration mistakes)
ConvergenceConverges to $Q^*$Converges to $Q^\pi$ for the policy being followed

6. Exploration vs. Exploitation in Q-Learning

Q-learning needs to explore the environment to discover good actions, but it also needs to exploit what it has already learned to accumulate reward. The most common strategy for balancing this is epsilon-greedy action selection:

$$ a_t = \begin{cases} \text{random action} & \text{with probability } \epsilon \ \arg\max_a Q(s_t, a) & \text{with probability } 1 – \epsilon \end{cases} $$

Typically, $\epsilon$ starts high (encouraging exploration early on) and decays over time as the agent gains confidence in its Q-value estimates.

7. The Q-Learning Algorithm Step-by-Step

  1. Initialize $Q(s,a)$ arbitrarily for all states and actions (often all zeros).
  2. For each episode:
    • Initialize the starting state $s$.
    • For each step in the episode:
      1. Choose action $a$ using an exploration strategy (e.g., epsilon-greedy) based on $Q$.
      2. Take action $a$, observe reward $r$ and next state $s’$.
      3. Update: $Q(s,a) \leftarrow Q(s,a) + \alpha [r + \gamma \max_{a’} Q(s’,a’) – Q(s,a)]$
      4. Set $s \leftarrow s’$.
    • Repeat until $s$ is a terminal state.
  3. Repeat over many episodes until $Q$ converges.
flowchart TD
    A[Initialize Q table] --> B[Observe current state s]
    B --> C{Explore or exploit?}
    C -->|Explore epsilon| D[Choose random action]
    C -->|Exploit 1-epsilon| E[Choose argmax Q_s_a]
    D --> F[Take action, observe reward r and next state s']
    E --> F
    F --> G[Update Q_s_a using TD target]
    G --> H{Terminal state?}
    H -->|No| B
    H -->|Yes| I[End episode]
    I --> A

8. Implementing Q-Learning in Python

Let’s implement classic tabular Q-learning on a simple grid environment using OpenAI Gym’s FrozenLake.

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=False)

n_states = env.observation_space.n
n_actions = env.action_space.n

Q = np.zeros((n_states, n_actions))

alpha = 0.1        # learning rate
gamma = 0.99       # discount factor
epsilon = 1.0      # exploration rate
epsilon_decay = 0.995
epsilon_min = 0.01
episodes = 5000

for episode in range(episodes):
    state, _ = env.reset()
    done = False

    while not done:
        # Epsilon-greedy action selection
        if np.random.rand() < epsilon:
            action = env.action_space.sample()
        else:
            action = np.argmax(Q[state])

        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated

        # Q-learning update
        best_next_action = np.argmax(Q[next_state])
        td_target = reward + gamma * Q[next_state, best_next_action]
        Q[state, action] += alpha * (td_target - Q[state, action])

        state = next_state

    epsilon = max(epsilon_min, epsilon * epsilon_decay)

print("Learned Q-table:")
print(Q)

After training, the agent’s Q-table encodes the estimated value of every action in every state, and you can derive the optimal policy simply by taking np.argmax(Q[state]) in any state.

9. From Tabular Q-Learning to Deep Q-Networks (DQN)

Tabular Q-learning works fine when the number of states and actions is small. But what happens when the state space is enormous — like raw pixels from an Atari game, with millions of possible configurations? Storing a table becomes impossible.

This is where Deep Q-Networks (DQN), introduced by DeepMind in 2015, come in. Instead of a table, a neural network $Q_\theta(s,a)$ approximates the Q-function:

$$ Q_\theta(s, a) \approx Q^*(s, a) $$

The network is trained by minimizing the mean squared error between the predicted Q-value and the TD target:

$$ L(\theta) = \mathbb{E}{(s,a,r,s’) \sim D} \Big[ \big( r + \gamma \max{a’} Q_{\theta^-}(s’, a’) – Q_\theta(s,a) \big)^2 \Big] $$

Two key innovations made DQN stable:

  1. Experience Replay: Instead of training on transitions in the order they occur, DQN stores them in a replay buffer $D$ and samples random mini-batches. This breaks correlations between consecutive samples and improves data efficiency.
  2. Target Network: A separate, slowly-updated copy of the network, $Q_{\theta^-}$, is used to compute the TD target, which prevents the “moving target” problem where the network is chasing its own rapidly changing predictions.

A minimal DQN network in PyTorch:

import torch
import torch.nn as nn

class DQN(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )

    def forward(self, state):
        return self.net(state)   # outputs Q-values for each action

10. Advantages of Q-Learning

11. Disadvantages and Limitations

12. Real-World Applications of Q-Learning

13. Best Practices for Using Q-Learning

  1. Decay epsilon gradually. Starting with high exploration and slowly reducing it typically leads to better final policies than fixed exploration rates.
  2. Normalize and scale rewards, especially when using function approximation — large reward magnitudes can destabilize neural network training.
  3. Use experience replay for any Q-learning variant involving function approximation, since it dramatically improves sample efficiency and stability.
  4. Use target networks (or Polyak averaging) to prevent the moving-target problem in deep Q-learning.
  5. Consider Double Q-learning to mitigate overestimation bias, especially in noisy or stochastic environments.
  6. Track Q-value statistics during training (mean, max) to catch overestimation or divergence early.
  7. Tune the discount factor $\gamma$ based on the task horizon — tasks with long-term dependencies need $\gamma$ closer to 1.

14. Summary

Q-learning is a foundational reinforcement learning algorithm that learns the value of taking specific actions in specific states, without needing a model of the environment. Its off-policy nature — learning about the greedy policy while behaving according to an exploratory one — makes it flexible and sample-reusable, which is why it became the backbone of Deep Q-Networks and the modern deep RL revolution.

We covered:

If you understand Q-learning deeply, you’ll find it much easier to understand nearly every value-based deep RL algorithm that came after it — because most of them are, in one way or another, answering the question Q-learning first asked: “What is the value of taking this action, right now, in this state?”

References

Exit mobile version