Learn · Intermediate

Markov Decision Processes: The Math Behind How AI Learns to Act

A Markov Decision Process, or MDP, is the mathematical framework that formalizes how an agent makes a sequence of decisions to achieve a goal. It has four ingredients: states (the situations the agent can be in), actions (the choices available in each state), a reward signal (a number telling the agent how well it is doing), and transitions (the rules, often probabilistic, for how actions move you from one state to the next). Nearly all of reinforcement learning -- the technique behind game-playing systems, robot controllers, and the training that shapes modern chatbots -- is, underneath, the problem of solving an MDP. It was introduced by Richard Bellman in 1957, and it remains the single most useful lens for thinking about AI that acts over time.

Start with the word 'Markov.' The Markov property says the future depends only on the present state, not on the entire history of how you arrived there. If you are playing chess, the board in front of you tells you everything you need to decide your next move -- it does not matter in what order the pieces got there. This is a powerful simplifying assumption: it means the agent can carry a single summary of its situation (the state) rather than an ever-growing memory of everything that has happened. When the assumption holds, the whole problem becomes tractable.

The agent's job is to find a policy -- a rule that maps each state to an action -- that maximizes not immediate reward but total reward over the long run. This is where MDPs get their teeth. A move that pays off now might lead somewhere terrible later; a move that costs you now might set up a win. To balance this, MDPs use the idea of a value: the value of a state is the total future reward you can expect if you start there and act well from then on. Bellman's famous equation ties it together recursively -- the value of where you are equals the reward you get now plus the (discounted) value of where you end up next. A discount factor makes near-term rewards count for more than distant ones, which keeps the math well-behaved and mirrors how a dollar today beats a dollar in a decade.

How do you actually solve an MDP? If you know the transition and reward rules exactly, classical methods like value iteration and policy iteration -- dynamic programming -- can compute the optimal policy directly. But in most interesting problems the agent does not know the rules; it has to learn them by trying things and observing what happens. That is reinforcement learning, and the definitive treatment is Sutton and Barto's Reinforcement Learning: An Introduction. Methods like Q-learning estimate the value of state-action pairs from experience; policy-gradient methods adjust the policy directly. The distinction between learning from your own current behavior versus from stored past behavior is the on-policy versus off-policy split, and it is a choice about how to squeeze learning out of an MDP you cannot see all of.

That last phrase points to the crucial complication: the plain MDP assumes the agent can see the true state. Real agents usually cannot. A robot sees camera pixels, not the exact positions of every object; a self-driving car reads noisy sensors; a poker player cannot see opponents' cards. This is a Partially Observable Markov Decision Process, or POMDP, formalized in the landmark 1998 paper by Leslie Kaelbling, Michael Littman and Anthony Cassandra. In a POMDP the agent never observes the state directly -- it receives observations that only hint at it -- so it must maintain a 'belief,' a probability distribution over what state it might be in, and act on that belief instead. POMDPs are far harder to solve exactly, but they describe the world honestly.

This framework quietly underlies a lot of today's AI. When Fei-Fei Li's 2026 essay on world models argued that the many things now called 'world models' -- renderers, simulators, planners -- are all different projections of one underlying loop, that loop is the POMDP: the agent takes actions, actions change a hidden state, the agent sees only observations (photons, sensor readings, pixels), and new observations inform new actions. A planner, in her taxonomy, is exactly the piece that solves the decision problem: given an observation and a goal, what should the agent do next. Even the reinforcement learning that fine-tunes large language models frames a conversation as an MDP, where the state is the dialogue so far, the actions are the next tokens, and the reward comes from human preferences.

Why learn this? Because once you see the states-actions-rewards-transitions skeleton, a huge swath of AI stops looking like a grab-bag of tricks and starts looking like variations on one problem: how should an agent behave, over time, under uncertainty, to get the most of what it wants? The MDP is the sentence that problem is written in -- and knowing it makes everything from game-playing agents to robot policies to RLHF legible.

Ground Truth. (2026, July 17). Markov Decision Processes: The Math Behind How AI Learns to Act. Ground Truth. https://groundtruth.day/learn/markov-decision-processes.html

@misc{groundtruth:markov-decision-processes,
  title  = {Markov Decision Processes: The Math Behind How AI Learns to Act},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/markov-decision-processes.html}
}

Topics: reinforcement-learning · mdp · pomdp · decision-making · fundamentals