Train an AI Agent with Reinforcement Learning

Reinforcement learning (RL) trains an agent to choose actions by interacting with an environment and receiving rewards or penalties. This beginner project uses a small grid world to show the full loop: define a task, train a tabular agent, and check whether it learned a useful policy. You will need basic Python familiarity; no GPU or large dataset is required.

What You’ll Need

• Python 3 and a terminal or notebook, with the ability to install packages.
• NumPy, for storing and updating the agent’s action values.
• A modest amount of time to run experiments and inspect results. Training is not guaranteed to work just because code runs.

Step-by-Step

1. Choose a small, measurable task. Start with a simple grid world rather than a game with images or a robot. The agent occupies one cell, can move up, down, left, or right, and aims to reach a goal while avoiding a hazard. This keeps states and actions easy to inspect. Decide what success means before coding: for example, reaching the goal in fewer moves without hitting the hazard.
2. Define the environment. Represent each valid cell as a state, and the four moves as actions. Write a reset function that places the agent at the start and returns the starting state. Write a step function that accepts an action, updates the position, and returns the next state, reward, and a done flag. End an episode when the agent reaches the goal, hits the hazard, or exceeds a move limit. A move might earn -1, the goal +10, and the hazard -10; choose values that make progress toward the goal preferable to wandering.
3. Make the task reproducible. Use a fixed random seed while debugging. Check that every action produces a valid next state, terminal states really end episodes, and rewards match the intended outcomes. Try a few actions by hand before training. If walls block movement, specify whether an attempted move leaves the agent in place and whether it still incurs the step cost.
4. Create an action-value table. For a small discrete grid, store one value, Q(s,a), for every state and action. Initialize the values to zero. Q estimates the expected future reward for taking action a in state s and then following the current policy. For larger or continuous problems, a table may be impractical; that is when function approximation and libraries such as Gymnasium-compatible tools become relevant, but those add complexity.
5. Implement Q-learning. After taking action a in state s and observing reward r and next state s′, update the table with: Q(s,a) ← Q(s,a) + α[r + γ max_a′ Q(s′,a′) − Q(s,a)]. Here α is the learning rate, between 0 and 1, and γ is the discount factor, also between 0 and 1. If the transition ends the episode, use no future-value term: the target is just r. A practical starting point is α = 0.1 and γ = 0.95, then adjust based on repeated runs rather than treating these as universal settings.
6. Balance exploration and exploitation. During training, choose a random valid action with probability ε; otherwise choose the action with the highest current Q value. This is called ε-greedy exploration. Begin with a relatively high ε, such as 1.0, and gradually reduce it toward 0.05 over episodes. Keep some exploration while learning; if ε is zero from the start, the agent may never discover a useful route. When Q values tie, select randomly among the best actions to avoid a fixed directional bias.
7. Train across episodes. For each episode, reset the environment and repeatedly select an action, call the step function, apply the Q-learning update, and move to the returned state. Stop on the done flag or a maximum step count. Run many episodes—perhaps a few thousand for a tiny grid—and record the total reward, number of steps, and whether the goal was reached. These measurements help distinguish learning from a lucky episode.
8. Evaluate without exploration. After training, set ε to zero and run separate evaluation episodes from the start. Do not update Q during this check. Report the percentage that reach the goal and the average number of steps. Test multiple random seeds or start positions if the task has randomness. A strong training reward alone is not proof of success: the agent may have memorized one favorable sequence.
9. Inspect and improve carefully. Print the best action for each cell or draw the learned route. Check that it avoids hazards and reaches the goal efficiently. If performance is weak, change one thing at a time: verify rewards and transitions, increase training episodes, adjust the exploration schedule, or compare a few learning rates. Keep the evaluation setup unchanged so results remain comparable.

Common Mistakes

A common error is rewarding the goal but forgetting a step cost or episode limit, allowing aimless behavior to look acceptable. Another is updating from the next-state maximum after a terminal transition; this incorrectly assumes future rewards exist after the episode ends. Keep training exploration separate from evaluation, and do not judge learning from one run. Finally, avoid making the state representation omit information that affects what happens next. If two visually identical states have different consequences, the agent needs a way to tell them apart.

Tips

Start with the smallest environment that demonstrates the idea, then add complexity gradually. Log episode return and success rate, and plot them if convenient; a moving average can make trends easier to see. Fix seeds during debugging, but evaluate with several seeds before trusting a result. Save the learned table and the environment settings together so you can reproduce a run. Use rewards that express the actual objective, not merely a metric that is easy to optimize. If the agent finds a strange shortcut, inspect the reward design and transition rules before changing the algorithm.

Troubleshooting

The agent never reaches the goal: test whether the goal is reachable and whether the step function returns the correct state. Confirm that exploration remains active early in training and that episodes last long enough to reach the goal.

Training reward improves but evaluation fails: check that evaluation starts from the intended state, uses the same environment rules, and disables learning updates. Try multiple seeds to detect a result that depends on luck.

Rewards fluctuate heavily: this can be normal while actions are still exploratory. Track success rate over a window of episodes and evaluate with ε = 0. If values swing wildly, reduce the learning rate or slow the exploration decay.

The agent repeats an unhelpful action: check tie-breaking, blocked-move handling, and whether the state table includes all relevant information. Verify that terminal transitions do not bootstrap from another Q value.

The code is too slow or complicated: reduce the grid size and remove rendering during training. For a discrete table, the computation should be light; slow runs often come from repeated visualization or excessive logging.

Frequently Asked Questions

Do I need a GPU?
No. A small tabular problem runs on an ordinary computer. GPUs are useful for some larger neural-network-based environments, not this starter example.

How many episodes should I train?
There is no universal number. Start with a few thousand for a tiny grid, then evaluate success rate and increase training if performance is still improving.

What is the difference between a reward and a Q value?
A reward is the immediate feedback for a transition. A Q value estimates the total discounted future reward associated with an action from a state.

Why does the agent need random exploration?
Without trying alternatives, it may settle on an early action that seems good but leads to poor outcomes. Exploration lets it discover other routes.

When should I use a neural network?
Consider one when the state or action space is too large for a table, such as inputs made of images. Neural methods require more tuning and careful evaluation, so learn the tabular loop first.

Does a trained agent always find the best policy?
No. Results depend on the environment, reward design, exploration, and training. Test the policy on separate runs and inspect its behavior before relying on it.

Related

• Data Center Power and Water: Which Public Numbers Can You Trust?
• AI-Powered Phishing vs Traditional Phishing: How to Tell
• One-Minute Platformer Levels Players Want to Replay