Information Uncertainty Influences Learning Strategy from Sequentially Delayed Rewards

Abstract

The problem of temporal credit assignment has long been posed as a nontrivial obstacle to identifying signal from data. However, human solutions in complex environments, involving repeated and intervening decisions, as well as uncertainty in reward timing, remain elusive. To this end, our task manipulated uncertainty via the amount of information given in their feedback stage. Using computational modeling, two learning strategies were developed that differentiated participants’ updates of sequentially delayed rewards: eligibility trace whereby previously selected actions are updated as a function of the temporal sequence - and tabular update - whereby additional feedback information is used to only update systematically-related rather than randomly related past actions. In both models, values were discounted over time with an exponential decay. We hypothesized that higher uncertainty would be associated with (i) a switch from tabular to eligibility strategy and (ii) higher rates of discounting. Participants’ data (N = 142) confirmed our first hypothesis, additionally revealing an effect of the starting condition. However, our discounting hypothesis had only weak evidence of an effect and remains an open question for future studies. We explore potential explanations for these effects and possibilities of future directions, models, and designs.