Loughborough University
Browse

Solving the distal reward problem with rare correlations

Download (1.55 MB)
journal contribution
posted on 2015-03-13, 13:46 authored by Andrea SoltoggioAndrea Soltoggio, Jochen Steil
In the course of trial-and-error learning, the results of actions, manifested as rewards or punishments, occur often seconds after the actions that caused them. How can a reward be associated with an earlier action when the neural activity that caused that action is no longer present in the network? This problem is referred to as the distal reward problem. A recent computational study proposes a solution using modulated plasticity with spiking neurons and argues that precise firing patterns in the millisecond range are essential for such a solution. In contrast, the study reported in this letter shows that it is the rarity of correlating neural activity, and not the spike timing, that allows the network to solve the distal reward problem.In this study, rare correlations are detected in a standard rate-based computational model by means of a thresholdaugmented Hebbian rule. The novel modulated plasticity rule allows a randomly connected network to learn in classical and instrumental conditioning scenarios with delayed rewards. The rarity of correlations is shown to be a pivotal factor in the learning and in handling various delays of the reward. This study additionally suggests the hypothesis that short-term synaptic plasticity may implement eligibility traces and thereby serve as a selectionmechanism in promoting candidate synapses for long-term storage.

Funding

This work was supported by the European Community’s Seventh Framework Programme FP7/2007-2013, Challenge 2 Cognitive Systems, Interaction, Robotics (Grant No. 248311—AMARSi).

History

School

  • Science

Department

  • Computer Science

Published in

Neural Computation

Volume

25

Issue

4

Pages

940 - 978

Citation

SOLTOGGIO, A. and STEIL, J., 2013. Solving the distal reward problem with rare correlations. Neural Computation, 25 (4), pp. 940-978

Publisher

© Massachusetts Institute of Technology Press

Version

  • VoR (Version of Record)

Publisher statement

This work is made available according to the conditions of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) licence. Full details of this licence are available at: https://creativecommons.org/licenses/by-nc-nd/4.0/

Publication date

2013

Notes

This article is © Massachusetts Institute of Technology Press.

ISSN

0899-7667

eISSN

1530-888X

Language

  • en

Usage metrics

    Loughborough Publications

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC