When I first started learning reinforcement learning (RL), one idea kept showing up everywhere before I even understood what it meant: the state-value function. Every textbook, every lecture, every research paper leaned on it as if it were the most obvious thing in the world. It took me a while to realize that this single concept is actually the backbone of almost everything else in RL — from Q-learning to policy gradients to deep RL algorithms like PPO and A3C.
In this article, I want to walk through the state-value function from the ground up. I’ll start with the intuition, move into the math, show you how it’s computed, and then connect it to real code and real-world applications. By the end, you should be comfortable enough with the idea that when you see $V(s)$ in a paper, you won’t flinch.
1. The Big Picture: What Problem Are We Solving?
Reinforcement learning is about an agent that interacts with an environment. At each time step, the agent is in some state $s$, takes an action $a$, receives a reward $r$, and transitions to a new state $s’$. The agent’s goal is to learn a policy — a strategy for choosing actions — that maximizes the total reward it collects over time.
But here’s the catch: rewards aren’t just about the immediate payoff. An action might give a small reward now but set the agent up for a much bigger reward later (or vice versa — a big reward now might lead to disaster later). So the agent needs some way to judge “how good” it is to be in a particular state, considering everything that might happen afterward.
That’s exactly what the state-value function answers: if I’m in this state and I follow my current strategy, how much total reward can I expect to collect from here onward?
2. Defining the State-Value Function
The state-value function, usually written $V^\pi(s)$, is defined as the expected return (cumulative future reward) starting from state $s$ and following policy $\pi$ thereafter.
Formally:
$$ V^\pi(s) = \mathbb{E}_\pi \left[ G_t \mid S_t = s \right] $$
where $G_t$ is the return — the total discounted reward from time step $t$ onward:
$$ G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} $$
Here:
- $R_t$ is the reward received at time $t$.
- $\gamma \in [0, 1]$ is the discount factor, which controls how much the agent cares about future rewards compared to immediate ones.
- $\pi$ is the policy being followed — a mapping from states to actions (or a probability distribution over actions).
If $\gamma$ is close to 0, the agent is “myopic” and only cares about immediate rewards. If $\gamma$ is close to 1, the agent is “farsighted” and values long-term rewards almost as much as immediate ones.
Why Discounting Matters
Discounting isn’t just a mathematical trick — it also has a very intuitive justification. A reward today is generally worth more than the same reward in the distant future (think of it like interest rates in finance). Discounting also keeps the sum in the return finite when the environment doesn’t naturally terminate.
3. The Bellman Equation for State Values
The real power of the state-value function comes from the fact that it can be expressed recursively. This recursive relationship is known as the Bellman Equation, named after Richard Bellman, who pioneered dynamic programming.
$$ V^\pi(s) = \sum_{a} \pi(a \mid s) \sum_{s’, r} p(s’, r \mid s, a) \left[ r + \gamma V^\pi(s’) \right] $$
Let’s break this down piece by piece:
- $\pi(a \mid s)$: the probability of taking action $a$ in state $s$ under policy $\pi$.
- $p(s’, r \mid s, a)$: the probability of transitioning to state $s’$ and receiving reward $r$, given that action $a$ was taken in state $s$.
- $r + \gamma V^\pi(s’)$: the immediate reward plus the discounted value of the next state.
In plain English: the value of a state equals the expected immediate reward plus the discounted value of wherever you end up next. This recursive structure is what allows algorithms to compute value functions iteratively, without having to simulate infinite trajectories.
4. Optimal State-Value Function
Every policy $\pi$ has its own value function $V^\pi$. But what we usually care about is the optimal value function $V^*(s)$ — the best possible expected return achievable from state $s$, over all possible policies:
$$ V^*(s) = \max_{\pi} V^\pi(s) $$
The optimal value function satisfies the Bellman Optimality Equation:
$$ V^(s) = \max_{a} \sum_{s’, r} p(s’, r \mid s, a) \left[ r + \gamma V^(s’) \right] $$
This equation says: the optimal value of a state is obtained by choosing the action that maximizes the expected immediate reward plus the discounted optimal value of the next state.
Once you know $V^*(s)$ for every state, extracting the optimal policy is straightforward — just pick the action that leads to the highest expected value.
5. State-Value Function vs Action-Value Function (Q-function)
A common point of confusion for beginners is the difference between $V(s)$ and $Q(s, a)$.
| Aspect | State-Value Function $V(s)$ | Action-Value Function $Q(s,a)$ |
|---|---|---|
| Input | Just the state | State and action pair |
| Meaning | Expected return from a state, following policy $\pi$ | Expected return from taking action $a$ in state $s$, then following $\pi$ |
| Use case | Useful when the model of the environment is known (used in policy evaluation) | Useful when the model is unknown (used in Q-learning, model-free control) |
| Relationship | $V^\pi(s) = \sum_a \pi(a | s) Q^\pi(s,a)$ |
The relationship between the two is important:
$$ V^\pi(s) = \sum_{a} \pi(a \mid s) , Q^\pi(s, a) $$
This tells us that the value of a state is simply the weighted average of the values of all actions available in that state, weighted by how likely the policy is to choose them.
6. How Is the State-Value Function Computed?
There are three broad families of methods for estimating $V(s)$:
a) Dynamic Programming (Policy Evaluation)
If we have a complete model of the environment (transition probabilities and rewards), we can compute $V^\pi(s)$ exactly using iterative policy evaluation:
$$ V_{k+1}(s) = \sum_{a} \pi(a\mid s) \sum_{s’,r} p(s’,r\mid s,a)\left[r + \gamma V_k(s’)\right] $$
We start with an arbitrary $V_0(s)$ (often all zeros) and repeatedly apply this update until the values converge.
b) Monte Carlo Methods
When we don’t know the environment’s dynamics, we can estimate $V(s)$ by running many episodes, observing the actual returns $G_t$ from each visit to state $s$, and averaging them:
$$ V(s) \approx \frac{1}{N(s)} \sum_{i=1}^{N(s)} G_t^{(i)} $$
This is simple and unbiased but requires complete episodes and tends to have high variance.
c) Temporal-Difference (TD) Learning
TD learning combines the best of both worlds — it doesn’t need a model of the environment (like Monte Carlo), but it updates estimates after every single step (like dynamic programming), using bootstrapping. The most well-known TD method is TD(0):
$$ V(S_t) \leftarrow V(S_t) + \alpha \left[ R_{t+1} + \gamma V(S_{t+1}) – V(S_t) \right] $$
Here, $\alpha$ is the learning rate, and the term in brackets is called the TD error:
$$ \delta_t = R_{t+1} + \gamma V(S_{t+1}) – V(S_t) $$
The TD error tells us how surprised the agent is — the difference between what it expected the value to be and what it now believes it should be, given the new information.
7. Visualizing the Relationship
Below is a simplified flow of how a state-value function gets updated during learning:
flowchart LR
A["Agent observes the current state"] --> B["Agent selects an action"]
B --> C["Environment returns reward and next state"]
C --> D["Compute the temporal difference error"]
D --> E["Update the value function"]
E --> F["Move to the next state"]
F --> AThis loop repeats over thousands or millions of steps until the value estimates stabilize and accurately reflect the long-term consequences of being in each state.
8. A Simple Python Implementation
Let’s implement TD(0) prediction for a simple gridworld-like environment to make this concrete.
import numpy as np
# Simple environment: 5 states in a line, terminal at state 4
n_states = 5
V = np.zeros(n_states) # Initialize state values to 0
gamma = 0.9 # Discount factor
alpha = 0.1 # Learning rate
episodes = 1000
def step(state, action):
"""action: 0 = left, 1 = right"""
if action == 1:
next_state = min(state + 1, n_states - 1)
else:
next_state = max(state - 1, 0)
reward = 1 if next_state == n_states - 1 else 0
done = next_state == n_states - 1
return next_state, reward, done
for episode in range(episodes):
state = 0
done = False
while not done:
action = np.random.choice([0, 1]) # random policy
next_state, reward, done = step(state, action)
# TD(0) update
td_target = reward + gamma * V[next_state]
td_error = td_target - V[state]
V[state] += alpha * td_error
state = next_state
print("Estimated state values:", V)
Running this will show that states closer to the terminal (rewarding) state end up with higher estimated values, exactly as we’d expect intuitively.
9. Advanced: State-Value Functions in Deep Reinforcement Learning
In small, discrete environments, we can represent $V(s)$ as a table (one entry per state). But in complex environments — like Atari games, robotics, or self-driving car simulations — the state space is enormous or continuous, so tables become impossible.
This is where function approximation comes in. Instead of storing $V(s)$ directly, we approximate it using a parameterized function, typically a neural network:
$$ V(s) \approx V_\theta(s) $$
where $\theta$ represents the weights of the neural network. Training becomes a supervised-learning-like problem where we minimize the mean squared TD error:
$$ L(\theta) = \mathbb{E}\left[ \left( R_{t+1} + \gamma V_\theta(S_{t+1}) – V_\theta(S_t) \right)^2 \right] $$
This is exactly what happens in algorithms like Advantage Actor-Critic (A2C/A3C) and Proximal Policy Optimization (PPO), where a neural network called the “critic” learns $V_\theta(s)$ while another network (the “actor”) learns the policy.
A minimal PyTorch-style critic network might look like this:
import torch
import torch.nn as nn
class ValueNetwork(nn.Module):
def __init__(self, state_dim, hidden_dim=64):
super().__init__()
self.net = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1)
)
def forward(self, state):
return self.net(state)
This network takes a state as input and outputs a single scalar — the estimated value of that state.
10. Advantages of Using State-Value Functions
- Compact representation of long-term outcomes: Instead of tracking entire trajectories, the agent only needs one number per state.
- Enables efficient policy evaluation and improvement: Once you know $V(s)$, you can compare policies and improve them systematically (this is the foundation of policy iteration).
- Reduces variance compared to raw Monte Carlo returns: Especially when combined with TD learning, value functions smooth out noisy reward signals.
- Forms the basis for actor-critic methods: Modern deep RL algorithms rely heavily on learned value functions to stabilize training.
11. Disadvantages and Limitations
- Requires a policy to be meaningful: $V(s)$ is always defined with respect to a policy; it says nothing universal about a state unless you specify which policy is being followed (unless you’re referring to the optimal value function).
- Doesn’t tell you which action to take: Unlike $Q(s,a)$, the state-value function alone doesn’t directly indicate the best action — you need a model of the environment or an additional mechanism (like an actor) to extract a policy.
- Function approximation errors: When using neural networks to approximate $V(s)$, errors can compound and destabilize training, especially in high-dimensional or sparse-reward environments.
- Sample inefficiency: Estimating accurate values in large state spaces can require an enormous number of interactions with the environment.
12. Real-World Applications
State-value functions aren’t just academic — they power some very tangible systems:
- Game-playing AI: DeepMind’s AlphaGo and AlphaZero used value networks to evaluate board positions in Go and Chess, effectively learning a state-value function over billions of possible positions.
- Robotics: Robots learning to walk or manipulate objects use value functions to judge how promising a given configuration or pose is.
- Autonomous driving simulations: Value functions help evaluate how “safe” or “efficient” a driving state is, guiding path-planning algorithms.
- Finance and portfolio management: Some algorithmic trading systems use RL where the “state” is the portfolio and market conditions, and the value function estimates expected future returns.
- Recommendation systems: A user’s browsing state can be evaluated for long-term engagement value rather than just immediate click-through rate.
13. Best Practices When Working with State-Value Functions
- Choose gamma thoughtfully. A discount factor too close to 1 can make training unstable; too close to 0 makes the agent short-sighted.
- Normalize rewards and states before feeding them into neural value function approximators — this stabilizes gradient-based learning.
- Use bootstrapping carefully. TD methods introduce bias if the value function is poorly initialized; consider techniques like TD(λ) to balance bias and variance.
- Combine with baseline subtraction in policy gradient methods. Using $V(s)$ as a baseline in the advantage function $A(s,a) = Q(s,a) – V(s)$ significantly reduces variance in policy gradient estimates.
- Monitor for overestimation. Especially in deep RL, value networks can overestimate returns; techniques like double learning or target networks help mitigate this.
- Use experience replay when training value networks with off-policy data to improve sample efficiency and stability.
14. Summary
The state-value function $V(s)$ is one of the most fundamental building blocks in reinforcement learning. It answers a deceptively simple question — “how good is it to be in this state?” — but that question underlies almost every algorithm in the field, from classic dynamic programming to cutting-edge deep RL systems like AlphaZero and PPO.
We covered:
- The formal definition of $V^\pi(s)$ and the return $G_t$.
- The Bellman Equation and Bellman Optimality Equation.
- How $V(s)$ relates to $Q(s,a)$.
- Three ways to estimate value functions: dynamic programming, Monte Carlo, and TD learning.
- How neural networks extend this idea to complex, high-dimensional environments.
- Practical code, advantages, limitations, and real-world use cases.
If you take away one thing from this article, let it be this: almost every advanced RL algorithm you’ll encounter is, at its heart, just a clever way of estimating and using this one function. Once you internalize the state-value function, concepts like actor-critic methods, advantage estimation, and even Q-learning will click into place much faster.
References
- Sutton, R. S., & Barto, A. G. Reinforcement Learning: An Introduction (2nd Edition). MIT Press.
- Silver, D. UCL Course on Reinforcement Learning (Lecture notes and slides).
- Mnih, V. et al. “Playing Atari with Deep Reinforcement Learning.” DeepMind, 2013.
- Schulman, J. et al. “Proximal Policy Optimization Algorithms.” arXiv:1707.06347.
- OpenAI Spinning Up documentation: https://spinningup.openai.com/
