1 - Maintaining stimulus representations

The degree to which a stimulus i at time t is “active” in memory is denoted by:

$$ \tag{Eq.1} E_i(t) = \Sigma_{t_i \leq t}e^{-(\frac{t-t_i}{t\_constant})} $$

where t_i are all the time steps up to time t, and t_constant is a time constant (usually meant to be the inter-reward rate)¹

2 - Learning stimulus associations:

The model learns retrospective associations after meaningful causal targets occur. Whether event j is a meaningful causal target is given by:

$$ \tag{Eq.2} \Phi_j = \begin{cases} 1,& \text{if } \Phi_j(t) = 1\\ 1,& \text{if }DA_j + \beta_j > \theta\\ 0,& \text{otherwise} \end{cases} $$

where Φ plays the role of an indicator function, DA_j is the total dopamine activity at the time of event j, β_j is the unconditioned value of event j and θ is a global threshold parameter.² Note that the indicator function is self-preserving: once a stimulus becomes a meaningful causal target, it does not stop being so.

After stimulus j is observed, the predecessor representation contingency, or PRC, for each stimulus i is updated via:

PRC_i ← j = M_i ← j − M_i

where M_i ← j is the predecessor representation of i given j has occurred, and M_i is the base rate with which i occurs. Both of these quantities are given by:

M_i ← j = M_i ← j′ + Φ_jα(E_i ← j − M_i ← j′)

and

M_i = M_i′ + kα(E_i − M_i′)

where M_i ← j′ and M_i′ are the quantities before j was observed, k and α are learning rate parameters, and E_i ← j is the eligibility trace of stimulus i at the time j occurs (see Eq. 1).

Then, the PRC can be used to derive the prospective association, aptly named the successor representation contingency, or SRC via Bayes rule:

$$ \tag{Eq.5} SRC_{i \rightarrow j} = PRC_{i \leftarrow j} \frac{M_j}{M_i} $$

The base rate for j, M_j is calculated via Eq.4b.

3 - Releasing Dopamine

The model postulates that dopaminergic signaling encodes the adjusted net contingencies for causal relations between stimuli, or ANCCRs. The total dopaminergic activity at the time of event i is equal to:

DA_i = Σ_j(ANCCR_i → jΦ_j)

And the ANCCR from stimulus i to stimulus j is given by:

ANCCR_i → j = NC_{i ↔︎ j}CW_i → j − ∑_k ≠ i(ANCCR_{k ↔︎ j}Δ_k ← iΦ_{k ↔︎ i})

where NC_{i ↔︎ j} is the net contingency between stimuli i and j, CW_i → j is the causal weight that i has with j, Δ_k ← i is the recency of stimulus k with respect to stimulus i, and Φ_{k ↔︎ i} is an indicator function denoting whether k and i have a putative causal relationship with each other.

The net contingency between stimuli i and j, NC_{i ↔︎ j}, is given by:

NC_{i ↔︎ j} = wSRC_i → j + (1 − w)PRC_i ← j

or a weighted sum of successor and predecessor representation contingencies.

The net contingency is used to calculate the indicator function above, as:

$$ \tag{Eq.9} \Phi_{k \leftrightarrow i} = \begin{cases} 1,& \text{if } NC_{i \leftrightarrow j} > \theta\\ 0,& \text{otherwise} \end{cases} $$

where θ is the same threshold parameter used in Eq.2³, and the indicator function for a stimulus and itself, Φ_{i ↔︎ i}, is 0.

The recency term, Δ_k ← i, is given by:

$$ \tag{Eq.10} \Delta_{k \leftarrow i} = e^{-(\frac{t_j-t_i}{t\_constant})} $$

where t_constant is the same parameter used in Eq.1. Note however that Eq.9 does not include the sum term in Eq. 1. Finally, the causal weight from stimulus i to stimulus j is given by:

CW_i → j = CW_i → j′ + α_rewardδ_i → j

where CW_i → j′ is the previous causal weight, α_reward is a learning rate parameter exclusive for causal weights, and δ_i → j is a delta term depending on the sign of the total dopaminergic activity, given by:

$$ \tag{Eq.12} \delta_{i \rightarrow j} = \begin{cases} CW_{j \rightarrow j} - CW_{i \rightarrow j}, & \text{if } DA_j \ge 0\\ (0-CW_{i \rightarrow j})\frac{n_i^{-1}\Delta{i \leftarrow j} \Phi_{i \leftrightarrow j}}{\Sigma_{k \neq j}(n_k^{-1}\Delta_{k \leftarrow j} \Phi_{k \leftrightarrow j})},& \text{otherwise} \end{cases} $$

where CW_j → j above is the reward magnitude of stimulus j. In plain words, when dopaminergic activity is positive, causal weights (from all present and absent stimuli) strengthen. Conversely, when dopaminergic activity is negative, causal weights (from all present and absent stimuli) weaken, proportional to their normalized frequency and recency (as long as they have putative causal relations with j).

4 - Generating responses

Responding in ANCCR is lightly specified. The value of responding upon presentation of stimulus i is given by:

Q_i = Σ_k(SRC_i → kCW_i → k)

which can then be mapped onto probabilities via a softmax function⁴.