Markov Decision Process (MDP)
Imagine you are playing a video game where you control a character moving through different rooms, collecting coins, and avoiding traps. You want to find the best strategy to win the game while scoring the highest points.
How MDP Works?
In simple terms, an MDP is a game plan that helps an agent (like your game character) make the best decisions based on its current situation. It has five key elements:
- States (S) – Where you are in the game 🎮
- Example: Your character is in Room 1.
- Actions (A) – What you can do 🚶♂️
- Example: Move left, right, jump, or pick up a coin.
- Transition Probability (P) – What happens when you act 🎲
- Example: If you jump, there’s a 90% chance you land safely and a 10% chance you fall.
- Rewards (R) – Points you gain or lose 🏆
- Example:
- +10 points for collecting a coin.
- -5 points if you hit a trap.
- -1 point for just walking.
- Example:
- Discount Factor (γ) – How much you care about future rewards ⏳
- Example: If γ = 0, you only focus on the next move.
- If γ = 0.9, you think ahead to maximize total points over time.
Example: Finding the Best Strategy
Imagine your game world looks like this:
| Room 1 | Room 2 (💰) |
|---|---|
| Trap (⛔) | Exit (🏁) |
- If you move right, you reach Room 2 and get +10 points (coin 💰).
- If you move down, you hit the trap (⛔) and lose -5 points.
- The best strategy? Move right → then down to exit safely.
The goal of MDP is to find the best path (called a policy) to maximize your score over time.
Why is MDP Important?
MDP helps in decision-making under uncertainty, like:
✅ Self-driving cars choosing the safest route 🚗
✅ Robots learning how to clean a room efficiently 🤖
✅ AI assistants deciding what to recommend next 📱
