Offensive Security Researcher & Quantum Cryptography Analyst

What Is the Concept of a State-Value Function in Reinforcement Learning?

Abdul Wahab Junaid
What is the concept of a state-value function in reinforcement learning?

When I first started learning reinforcement learning (RL), one idea kept showing up everywhere before I even understood what it meant: the state-value function. Every textbook, every lecture, every research paper leaned on it as if it were the most obvious thing in the world. It took me a while to realize that this single concept is actually the backbone of almost everything else in RL — from Q-learning to policy gradients to deep RL algorithms like PPO and A3C.

In this article, I want to walk through the state-value function from the ground up. I’ll start with the intuition, move into the math, show you how it’s computed, and then connect it to real code and real-world applications. By the end, you should be comfortable enough with the idea that when you see $V(s)$ in a paper, you won’t flinch.

1. The Big Picture: What Problem Are We Solving?

Reinforcement learning is about an agent that interacts with an environment. At each time step, the agent is in some state $s$, takes an action $a$, receives a reward $r$, and transitions to a new state $s’$. The agent’s goal is to learn a policy — a strategy for choosing actions — that maximizes the total reward it collects over time.

But here’s the catch: rewards aren’t just about the immediate payoff. An action might give a small reward now but set the agent up for a much bigger reward later (or vice versa — a big reward now might lead to disaster later). So the agent needs some way to judge “how good” it is to be in a particular state, considering everything that might happen afterward.

That’s exactly what the state-value function answers: if I’m in this state and I follow my current strategy, how much total reward can I expect to collect from here onward?

2. Defining the State-Value Function

The state-value function, usually written $V^\pi(s)$, is defined as the expected return (cumulative future reward) starting from state $s$ and following policy $\pi$ thereafter.

Formally:

$$ V^\pi(s) = \mathbb{E}_\pi \left[ G_t \mid S_t = s \right] $$

where $G_t$ is the return — the total discounted reward from time step $t$ onward:

$$ G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} $$

Here:

If $\gamma$ is close to 0, the agent is “myopic” and only cares about immediate rewards. If $\gamma$ is close to 1, the agent is “farsighted” and values long-term rewards almost as much as immediate ones.

Why Discounting Matters

Discounting isn’t just a mathematical trick — it also has a very intuitive justification. A reward today is generally worth more than the same reward in the distant future (think of it like interest rates in finance). Discounting also keeps the sum in the return finite when the environment doesn’t naturally terminate.

3. The Bellman Equation for State Values

The real power of the state-value function comes from the fact that it can be expressed recursively. This recursive relationship is known as the Bellman Equation, named after Richard Bellman, who pioneered dynamic programming.

$$ V^\pi(s) = \sum_{a} \pi(a \mid s) \sum_{s’, r} p(s’, r \mid s, a) \left[ r + \gamma V^\pi(s’) \right] $$

Let’s break this down piece by piece:

In plain English: the value of a state equals the expected immediate reward plus the discounted value of wherever you end up next. This recursive structure is what allows algorithms to compute value functions iteratively, without having to simulate infinite trajectories.

4. Optimal State-Value Function

Every policy $\pi$ has its own value function $V^\pi$. But what we usually care about is the optimal value function $V^*(s)$ — the best possible expected return achievable from state $s$, over all possible policies:

$$ V^*(s) = \max_{\pi} V^\pi(s) $$

The optimal value function satisfies the Bellman Optimality Equation:

$$ V^(s) = \max_{a} \sum_{s’, r} p(s’, r \mid s, a) \left[ r + \gamma V^(s’) \right] $$

This equation says: the optimal value of a state is obtained by choosing the action that maximizes the expected immediate reward plus the discounted optimal value of the next state.

Once you know $V^*(s)$ for every state, extracting the optimal policy is straightforward — just pick the action that leads to the highest expected value.

5. State-Value Function vs Action-Value Function (Q-function)

A common point of confusion for beginners is the difference between $V(s)$ and $Q(s, a)$.

AspectState-Value Function $V(s)$Action-Value Function $Q(s,a)$
InputJust the stateState and action pair
MeaningExpected return from a state, following policy $\pi$Expected return from taking action $a$ in state $s$, then following $\pi$
Use caseUseful when the model of the environment is known (used in policy evaluation)Useful when the model is unknown (used in Q-learning, model-free control)
Relationship$V^\pi(s) = \sum_a \pi(as) Q^\pi(s,a)$

The relationship between the two is important:

$$ V^\pi(s) = \sum_{a} \pi(a \mid s) , Q^\pi(s, a) $$

This tells us that the value of a state is simply the weighted average of the values of all actions available in that state, weighted by how likely the policy is to choose them.

6. How Is the State-Value Function Computed?

There are three broad families of methods for estimating $V(s)$:

a) Dynamic Programming (Policy Evaluation)

If we have a complete model of the environment (transition probabilities and rewards), we can compute $V^\pi(s)$ exactly using iterative policy evaluation:

$$ V_{k+1}(s) = \sum_{a} \pi(a\mid s) \sum_{s’,r} p(s’,r\mid s,a)\left[r + \gamma V_k(s’)\right] $$

We start with an arbitrary $V_0(s)$ (often all zeros) and repeatedly apply this update until the values converge.

b) Monte Carlo Methods

When we don’t know the environment’s dynamics, we can estimate $V(s)$ by running many episodes, observing the actual returns $G_t$ from each visit to state $s$, and averaging them:

$$ V(s) \approx \frac{1}{N(s)} \sum_{i=1}^{N(s)} G_t^{(i)} $$

This is simple and unbiased but requires complete episodes and tends to have high variance.

c) Temporal-Difference (TD) Learning

TD learning combines the best of both worlds — it doesn’t need a model of the environment (like Monte Carlo), but it updates estimates after every single step (like dynamic programming), using bootstrapping. The most well-known TD method is TD(0):

$$ V(S_t) \leftarrow V(S_t) + \alpha \left[ R_{t+1} + \gamma V(S_{t+1}) – V(S_t) \right] $$

Here, $\alpha$ is the learning rate, and the term in brackets is called the TD error:

$$ \delta_t = R_{t+1} + \gamma V(S_{t+1}) – V(S_t) $$

The TD error tells us how surprised the agent is — the difference between what it expected the value to be and what it now believes it should be, given the new information.

7. Visualizing the Relationship

Below is a simplified flow of how a state-value function gets updated during learning:

flowchart LR
    A["Agent observes the current state"] --> B["Agent selects an action"]
    B --> C["Environment returns reward and next state"]
    C --> D["Compute the temporal difference error"]
    D --> E["Update the value function"]
    E --> F["Move to the next state"]
    F --> A

This loop repeats over thousands or millions of steps until the value estimates stabilize and accurately reflect the long-term consequences of being in each state.

8. A Simple Python Implementation

Let’s implement TD(0) prediction for a simple gridworld-like environment to make this concrete.

import numpy as np

# Simple environment: 5 states in a line, terminal at state 4
n_states = 5
V = np.zeros(n_states)          # Initialize state values to 0
gamma = 0.9                     # Discount factor
alpha = 0.1                     # Learning rate
episodes = 1000

def step(state, action):
    """action: 0 = left, 1 = right"""
    if action == 1:
        next_state = min(state + 1, n_states - 1)
    else:
        next_state = max(state - 1, 0)
    reward = 1 if next_state == n_states - 1 else 0
    done = next_state == n_states - 1
    return next_state, reward, done

for episode in range(episodes):
    state = 0
    done = False
    while not done:
        action = np.random.choice([0, 1])   # random policy
        next_state, reward, done = step(state, action)

        # TD(0) update
        td_target = reward + gamma * V[next_state]
        td_error = td_target - V[state]
        V[state] += alpha * td_error

        state = next_state

print("Estimated state values:", V)

Running this will show that states closer to the terminal (rewarding) state end up with higher estimated values, exactly as we’d expect intuitively.

9. Advanced: State-Value Functions in Deep Reinforcement Learning

In small, discrete environments, we can represent $V(s)$ as a table (one entry per state). But in complex environments — like Atari games, robotics, or self-driving car simulations — the state space is enormous or continuous, so tables become impossible.

This is where function approximation comes in. Instead of storing $V(s)$ directly, we approximate it using a parameterized function, typically a neural network:

$$ V(s) \approx V_\theta(s) $$

where $\theta$ represents the weights of the neural network. Training becomes a supervised-learning-like problem where we minimize the mean squared TD error:

$$ L(\theta) = \mathbb{E}\left[ \left( R_{t+1} + \gamma V_\theta(S_{t+1}) – V_\theta(S_t) \right)^2 \right] $$

This is exactly what happens in algorithms like Advantage Actor-Critic (A2C/A3C) and Proximal Policy Optimization (PPO), where a neural network called the “critic” learns $V_\theta(s)$ while another network (the “actor”) learns the policy.

A minimal PyTorch-style critic network might look like this:

import torch
import torch.nn as nn

class ValueNetwork(nn.Module):
    def __init__(self, state_dim, hidden_dim=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )

    def forward(self, state):
        return self.net(state)

This network takes a state as input and outputs a single scalar — the estimated value of that state.

10. Advantages of Using State-Value Functions

11. Disadvantages and Limitations

12. Real-World Applications

State-value functions aren’t just academic — they power some very tangible systems:

13. Best Practices When Working with State-Value Functions

  1. Choose gamma thoughtfully. A discount factor too close to 1 can make training unstable; too close to 0 makes the agent short-sighted.
  2. Normalize rewards and states before feeding them into neural value function approximators — this stabilizes gradient-based learning.
  3. Use bootstrapping carefully. TD methods introduce bias if the value function is poorly initialized; consider techniques like TD(λ) to balance bias and variance.
  4. Combine with baseline subtraction in policy gradient methods. Using $V(s)$ as a baseline in the advantage function $A(s,a) = Q(s,a) – V(s)$ significantly reduces variance in policy gradient estimates.
  5. Monitor for overestimation. Especially in deep RL, value networks can overestimate returns; techniques like double learning or target networks help mitigate this.
  6. Use experience replay when training value networks with off-policy data to improve sample efficiency and stability.

14. Summary

The state-value function $V(s)$ is one of the most fundamental building blocks in reinforcement learning. It answers a deceptively simple question — “how good is it to be in this state?” — but that question underlies almost every algorithm in the field, from classic dynamic programming to cutting-edge deep RL systems like AlphaZero and PPO.

We covered:

If you take away one thing from this article, let it be this: almost every advanced RL algorithm you’ll encounter is, at its heart, just a clever way of estimating and using this one function. Once you internalize the state-value function, concepts like actor-critic methods, advantage estimation, and even Q-learning will click into place much faster.

References

Exit mobile version