Exploration vs. Exploitation in Reinforcement Learning
Published:
You have two routes to work. Your usual route takes about 35 minutes. A colleague suggests a shortcut that could get you there in 25, but you’ve never tried it. If you have a meeting first thing, you might stick with the road you know. On a morning when you have time to spare, testing the shortcut could save you time on future trips.
That choice is the exploration–exploitation trade-off. Exploitation means using what you’ve learned to pick the option that currently looks best. Exploration means trying an option to learn more about it. The shortcut could cost you time today and turn out to be the best route for the rest of the week.
Why the first result can mislead
Suppose you try the shortcut on a quiet Friday and get to work in 25 minutes. That gives you one data point. If you leave at the same hour on Monday, the same route might take 45 minutes. You need more trips under comparable conditions to tell whether Friday’s result came from the route or the traffic that day.
A reinforcement learning agent faces the same problem. Early in training, the turn with the highest estimated value may simply be the one that got lucky first. If the agent always picks that turn, it may gather little evidence about the alternatives. It can also get stuck with a partial picture of the road network. A turn at the first junction might lead to a whole set of streets the agent will only discover if it takes that turn.
Exploring on every trip has a cost too. Once the agent has found a reliable route, random turns can send it back into traffic. The question is how often to spend a trip learning something new and how often to use the route its experience already supports.
A simple balance with epsilon-greedy
Epsilon-greedy is one way to make that choice. At each decision, the agent picks a random action with probability ε (epsilon). The rest of the time, it picks the action with the highest estimated value. In reinforcement learning, that second choice is called greedy because it takes the option that looks best at the moment.
Here’s the action selection method from my SARSA notebook. It draws a uniform random number, then uses ε to decide whether to sample an action.
def select_action(self, state):
if np.random.rand() < self.epsilon:
return np.random.choice(self.num_actions)
return np.argmax(self.Q[state])
Suppose you set ε = 0.2. At a junction, the agent chooses randomly about 20% of the time and follows its best estimate about 80% of the time. If there are four legal turns and one clear favourite, the random choice will still pick the favourite a quarter of the time. So the agent takes its favourite turn about 85% of the time overall. Epsilon controls how often it picks randomly.
In a commute, you might decide to try an alternative only at the junction where you’re unsure. Basic epsilon-greedy applies the same random chance at each decision, even at a junction the agent knows well. On a ten-turn trip, several chances to explore can interrupt a route the agent already knows how to follow. That’s one reason to change ε as training progresses.
Explore more early, then test what the agent learned
With decaying epsilon, the agent starts with a higher chance of exploring and reduces it as training goes on. For example, you could start at ε = 1.0, multiply it by 0.999 after each episode, and keep a minimum of 0.01. Those numbers are an example. The same schedule can behave differently on a task where the goal is easy to stumble across and one where reaching it takes a long sequence of good decisions.
If ε drops too quickly, the agent may commit to a route after only a handful of useful trips. If it stays high too long, the agent may already know a good route while its training trips keep looking slow. To tell those cases apart, periodically pause training and test the learned policy with ε = 0. That lets you see how it performs when it follows its current best estimates at every junction. In the commute example, traffic can still vary, so compare average travel times over several trips under similar conditions.
When comparing schedules, give each agent the same number of training trips. Look at how quickly each one starts finding good routes and how well its learned policy performs at the end. A schedule that explores heavily might look worse early on while it maps out more options. The evaluation trips tell you whether that extra exploration paid off.
I tried this trade-off in FrozenLake, a grid game where an agent earns a reward for reaching a goal while avoiding holes. On slippery ice, a chosen move can send it to an unintended square. In my experiments, SARSA often struggled with lower ε values, while Q-learning fared better at those settings. That observation came from my setup. It gave me a concrete reason to test exploration settings for each algorithm and compare how often their learned policies reached the goal. See the FrozenLake experiments in the SARSA and Q-learning notebooks.
