Intrinsic rewards are designed to guide exploration in reinforcement learning by assigning value to an agent’s experience, for example through prediction error or learning progress. However, maximizing these rewards need not produce the most informative experience available. We propose a formal criterion for exploration based on the comparison of statistical experiments. It compares policies (rules for choosing actions from past actions and observations) by the counterfactual information they acquire: how well their histories can substitute for experience under alternative policies. We construct a single, simple environment in which specified count-based, prediction-error, empowerment, and information-gain objectives have maximizing policies that are strictly dominated under this criterion. We explain these failures and establish conditions under which existing intrinsic rewards successfully encourage optimal exploration. We also construct an objective that assigns a higher value whenever exploration strictly improves under our criterion.
The talk drew on the paper of the same name, with Laura Gomezjurado Gonzalez and Clark Barrett.