chapter-09.md

bibfile
ccnlab.bib

Executive Function {#sec:ch-exec}

We have now reached the top of the cognitive neuroscience hierarchy: the "executive" level. In a business, an executive makes important decisions and plans, based on high-level information coming in from all the different divisions of the company, and with a strong consideration of "the bottom line." In a person, the executive level of processing, thought to occur primarily within the prefrontal cortex (PFC), similarly receives high-level information from posterior cortical association areas, and is also directly interconnected with motivational and emotional areas that convey "the bottom line" forces that ultimately guide behavior. Although many of us walk around with the impression (delusion?) that our actions are based on rational thought and planning, instead it is highly likely that basic biological motivations and affective signals play a critical role in shaping what we do. At least, this is what the underlying biology of the PFC and associated brain areas suggests. And yet, it is also clear that the PFC is critical for supporting more abstract reasoning and planning abilities, including the ability to ignore distraction and other influences in the pursuit of a given goal. We will try to unravel the mystery of this seemingly contradictory coexistence of abilities in the PFC in this chapter.

Evidence for the importance of the PFC in higher-level cognitive control comes from the environmental dependency syndrome associated with damage to PFC. In one classic example, a patient with PFC damage visited a researcher's home and, upon seeing the bed, proceeded to get undressed (including removal of his toupee!), got into bed, and prepared to sleep. The environmental cues overwhelmed any prior context about what one should do in the home of someone you don't know very well. In other words, without the PFC, behavior is much more reflexive and unthinking, driven by the affordances of the immediate sensory environment, instead of by some more abstract and considered plan or goals. You don't need actual PFC damage to experience this syndrome --- certainly you have experienced yourself absent-mindedly doing something cued by the immediate sensory environment that you hadn't otherwise planned to do (e.g., brushing your teeth a second time before going to bed because you happened to see the toothbrush). We all experience lapses in attention --- the classic stereotype of an absent-minded professor is not explained by the lack of a PFC in professors, but rather that the PFC is apparently working on something else and thus leaves the rest of the brain to fend for itself in an environmentally-dependent manner.

Another great source of insight into the cognitive contributions of the PFC is available to each of us every night, in the form of our dreams. It turns out that the PFC is one of the brain areas most inactivated during dreaming phases of sleep. As a result, our dreams often lack continuity, and seem to jump from one disconnected scene to another, with only the most tangential thread connecting them. For example, one moment you might be reliving a tense social situation from high school, and the next you're trying to find out when the airplane is supposed to leave, with a feeling of general dread that you're hopelessly late for it.

So what makes the PFC uniquely capable of serving as the brain's executive? Part of the answer is its connectivity, as alluded to above --- it sits on top of the overall information processing hierarchy of the brain, and thus receives highly-processed "status reports" about everything important going on in your brain. In this sense it is similar to the hippocampus as we saw in the Memory Chapter, and indeed these areas appear to work together. However, the PFC is also especially well placed to exert control over our actions --- the PFC is just in front of the frontal motor areas (see the Motor Chapter), and has extensive connectivity to drive overt (and covert) motor behavior. Furthermore, the medial and ventral areas of PFC are directly interconnected with affective processing areas in subcortical regions such as the amygdala, thus enabling it to be driven by, and reciprocally, to amplify or override, motivational and affective signals.

In addition to being in the right place, the PFC also has some special biological properties that enable it to hold onto information in the face of distraction, e.g., from incoming sensory signals. Thus, with an intact PFC, you can resist the idea of laying down in someone else's bed, and remain focused on the purpose of your visit. We refer to this ability as robust active maintenance because it depends on the ability to keep a population of neurons actively firing over the duration needed to maintain a goal or other relevant pieces of information. This ability is also referred to as working memory, but this latter term has been used in many different ways in the literature, so we are careful to define it as synonymous with robust active maintenance of information in the PFC, in this context. We will see later how active maintenance works together with a gating system that allows us to hold in mind more than one item at a time, to selectively update and manipulate some information while continuing to maintain others, in a way that makes the integrated system support more sophisticated forms of working memory.

Recordings of neurons in the PFC of monkeys in the 1970's showed that they exhibit this robust active firing over delays (aka delay period activity). One of the most widely-used tasks is the oculomotor delayed response task, where a stimulus is flashed in a particular location of a video display, but the monkey is trained to maintain its eyes focused on a central fixation cross until that cross goes off, at which point it must then move its eyes to the previously flashed location in order to receive a juice reward. Neurons in the frontal eye fields (an area of PFC) show robust delay-period firing that is tuned to the location of the stimulus, and this activity terminates just after the monkey correctly moves its eyes after the delay. There are many other demonstrations of this robust active maintenance in the PFC of humans as well.

The computational models we explore in this chapter show how these two factors of connectivity and robust active maintenance can combine to support a wide range of executive function abilities that have been attributed to the PFC. The goal is to provide a unifying model of executive function, as compared to a laundry list of cognitive abilities that it is thought to support.

One of the most important executive function abilities is the ability to rapidly shift behavior or thought in a strategic manner (often referred to as cognitive flexibility). For example, when attempting to solve a puzzle or other challenging problem, you often need to try out many different ideas before discovering a good solution. Without the PFC, behavior is repetitive and stereotypical (banging your head against the wall again and again), lacking this hallmark flexibility. The ability to rapidly update what is being actively maintained in the PFC is what enables the PFC system to rapidly shift behavior or thought --- instead of requiring relatively slow synaptic weight modification to change how the system behaves, updating the pattern of active neural firing in PFC can change behavior immediately. In short, the PFC system contributes to behavioral adaptation by dynamically updating activation states, which then shape posterior cortical representations or motor actions via top-down biasing of the associated patterns of activity. In contrast, behavioral adaption in the posterior cortex or basal ganglia relies much more on slowly adapting weight changes. Evidence for this difference comes from task switching paradigms, including the widely-studied Wisconsin card sorting task (WCST) in adults, and the dimensional change card sorting task (DCCS) in children.

The computational models in this chapter show how the basal ganglia (BG) and midbrain dopamine areas (specifically the ventral tegmental area, VTA) play a critical role in the rapid, strategic updating of PFC activity states. Specifically, we'll see that robust active maintenance requires an additional control signal to switch between maintaining existing information vs. updating to encode new information. The BG, likely in conjunction with dopaminergic signals from the VTA, play this role of dynamic gating of the maintenance of information in PFC. This dynamic gating function is identical to the role the BG plays in gating motor actions, as we saw in the Motor Chapter. Furthermore, the BG learning process is also identical to that in the Motor chapter based on reinforcement learning principles. Specifically, dopamine (from the SNc, which is next door to the VTA) shapes BG learning and thereby enables the gating mechanism to deal with the challenging problem of deciding what is important to maintain (and as such is task-relevant and therefore predictive of intrinsic reward), vs. what can be ignored (because it is not predictive of good task performance). These mechanisms embody the general notion that the PFC-BG cognitive system evolved by leveraging existing powerful mechanisms for gating motor behavior and learning. From this perspective, cognition cannot be divorced from motivation, as dopaminergic learning signals play a central and intimate role in the basic machinery of PFC/BG function. The analogous functions of BG and dopamine in cognitive and motor action selection and learning have been strongly supported by various data over the last 10 or 20 years, including evidence from monkey studies, and in humans, effects of disease impacting BG and/or dopamine, pharmacological manipulations, functional imaging, and genetics.

Biology of PFC/BG and Dopamine Supporting Robust Active Maintenance

{#fig:fig-pbwm-architecture-bio width=40% }

The overall connectivity of the areas that are particularly important for executive function are shown in [@fig:fig-pbwm-architecture-bio], in relation to the sensory and motor processing associated with posterior cortex (temporal, parietal and occipital lobes) and motor frontal cortex (i.e., frontal cortex posterior to the prefrontal cortex). The PFC is interconnected with higher-level association cortical areas in posterior cortex where highly processed and abstracted information about the sensory world is encoded. It also interconnects with higher-level motor control areas (premotor cortex, supplementary motor areas), which coordinate lower-level motor control signals to execute sequences of coordinated motor outputs. With this pattern of connectivity, PFC is in a position to both receive from, and exert influence over, the processing going on in posterior and motor cortex.

{#fig:fig-bg_loops_ads86-9 width=75% }

The Basal Ganglia (BG), which consists principally of the striatum (caudate, putamen, nucleus accumbens), globus pallidus, and subthalamic nucleus, is densely interconnected with the PFC by way of specific nuclei of the thalamus ([@fig:fig-bg_loops_ads86-9]). As described in detail in the Motor Control and Reinforcement Learning Chapter, the BG provides a dynamic, adaptive gating influence on the frontal cortex, by disinhibiting the excitatory loop between PFC and the thalamus. In the context of the PFC, this gating influence controls the updating of information that is actively maintained in the PFC, using the same mechanisms that control the initiation of motor actions in the context of motor control. Also, top-down projections from PFC to the subthalamic nucleus support a type of inhibitory control over behavior by detecting conditions under which ongoing action selection should be halted or switched and preventing the rest of the BG circuitry from gating the planned motor action.

The final major component of the executive control system consists of the substantia nigra pars compacta (SNc) and several other associated brain areas that together drive phasic dopamine neuromodulation of the BG, resulting in reinforcement learning of its gating actions. This system, summarized computationally using the PVLV model as described in the Motor Control and Reinforcement Learning Chapter, interacts with the active maintenance of information in PFC to be able to reinforce a gating signal in the BG that leads to subsequent good performance and reward later in time. This time-travel property of the phasic DA reinforcement learning is essential for training a system that maintains information over time.

In the following subsections, we summarize the biological properties of each of these systems and their relevance to executive function. In a model of the A-not-B task, we'll explore how developing active maintenance abilities can explain otherwise puzzling behaviors of infants. We'll see how a model of the Stroop task captures a remarkably wide range of aspects of executive function (and dysfunction) in terms of robust active maintenance. To explore how the system can determine which information should be actively maintained, we'll turn to the prefrontal cortex basal ganglia working memory model (PBWM), which integrates basal ganglia and midbrain dopamine areas to perform complex executive function tasks.

Robust Active Maintenance in the PFC

{#fig:fig-SommerWurtz00-fig2 width=60% }

The ability of PFC neurons to exhibit sustained active firing over delays, as initially discovered by [@FusterAlexander71; @KubotaNiki71], is shown in [@fig:fig-SommerWurtz00-fig2], panel B ("Neuron with delay signal"), in the context of the delayed saccading task described in the introduction. Other subsets of PFC neurons also exhibit other firing patterns, such as responding transiently to visual inputs (Panel C) and initiating movements (Panel D). This differentiation of neural response patterns in PFC has important functional implications that we capture in the PBWM model described later.

{#fig:fig-pfc-trc-reverb-loops width=50% }

There are two primary biological mechanisms that enable PFC neurons to exhibit sustained active firing over time:

• Recurrent excitatory connectivity: Populations of PFC neurons have strong excitatory interconnections ([@fig:fig-pfc-trc-reverb-loops]), such that neural firing reverberates back-and-forth among these interconnected neurons, resulting in sustained active firing. There are two types of such connections: 1) a corticocortical loop among pyramidal cells in the same PFC stripe, and; 2) a corticothalamocortical loop between lamina VI pyramidal cells in PFC and the thalamic relay cells that project to that particular group of cells.

• Intrinsic excitatory maintenance currents: At the synapses formed by both of the recurrent excitatory loops there are NMDA and metabotropic glutamate (mGluR) receptors that, once opened by high frequency activity, provide a longer time window of increased excitability so as to keep reverberant activity going. Recall from the Learning Chapter that the NMDA channel requires the neuron to be sufficiently depolarized to remove the Mg+ (magnesium) ions that otherwise block the channel. This activity-dependent nature of the NMDA channel makes it ideally suited to providing a "switched" or dynamically gated form of active maintenance --- only those neurons that have already been sufficiently activated will benefit from the increased excitation provided by these NMDA channels. This provides a "hook" for the basal ganglia system to control active maintenance: when the thalamic neurons are disinhibited via a BG gating action, the ensuing burst of activity enables a subset of PFC neurons to get over their NMDA Mg+ block thresholds, and thereby continue to fire robustly over time.

Development of PFC Active Memory Strength and the A-not-B Task

The developmental process can provide important insights into various cognitive phenomena, often by making cognitive failures particularly stark and shedding light on the processes involved. A great example of this is the A-not-B task developed by pioneering developmental researcher Jean Piaget [@Piaget54]. An infant is repeated shown a toy hidden in one location (labeled A), and when the toy is then hidden in a different location (B), they continue to reach back to A. The behavior is striking --- the infant just saw the toy being hidden, tracking the experimenter's movements with great attention (typically novel, interesting toys are used). And yet they appear to forget all about this in a flash, reverting back to the previously established "habitual" behavior. Success on this task relies upon the development and integrity of dorsolateral prefrontal cortical regions [@Diamond90].

The computational model we explore here [@Munakata98] shows how a range of behavioral phenomena, some of it quite subtle and complex, can be captured with a relatively simple model that shares much in common with the Stroop model we'll explore later in the chapter. Development in this model is operationalized simply as the strength of the reverberant excitatory connections among PFC neurons, which are the only mechanism for active maintenance in this simplified model. The "older" networks can hold onto information (like where a toy was just hidden) for a longer period of time due to their stronger recurrent connections, while information is much more fleeting in the "younger" networks with weaker recurrent connections.

Exploration of Developing Active Maintenance

To see how this all plays out, open the a-not-b model in CCN Sims.

Functional Specialization Across PFC Areas

The mechanisms for robust active maintenance exist across the PFC, but different PFC areas have been associated with different contributions to overall executive function. We will explore the idea later in this chapter that these different functional contributions can be explained in terms of differences in connectivity of these PFC areas with other parts of the brain, within the context of the unifying framework that all PFC areas share robust active maintenance as a critical feature.

{#fig:fig-Fuster01-2 width=40% }

Anatomically, the frontal lobes constitute those cortical areas anterior to the central sulcus. Immediately anterior to the central sulcus, and thus most posteriorly in frontal cortex, is the primary motor cortex (M1), which is most prominently seen on the lateral surface but extends all the way over the dorsal surface and onto the medial side. Contiguous tissue roughly anterior to M1 makes up planning motor areas, the premotor (PM) cortex (laterally) and supplementary motor areas (SMA, pre-SMA; medially). Then, anterior to that are the PFC areas, labeled with their Brodmann numbers in [@fig:fig-Fuster01-2].

At the broadest level, the PFC areas can be divided along the major axes of medial vs. lateral and dorsal vs. ventral. Generally speaking, the lateral PFC areas are interconnected with sensory and motor areas in more posterior cortex, and are thought to play a role in controlling the processing in these areas. In contrast, the medial PFC areas are more strongly interconnected with subcortical brain areas associated with affective and motivational functions. Functionally we can characterize the lateral areas as being important for "cold" cognitive control, while the medial areas are important for "hot" emotional and motivational processing [@fig:fig-cortical_fun-9]. However, this distinction is not as clear cut as it sounds, as even the lateral areas are subject to modulation by motivational variables and BG/dopamine gating signals based on the extent to which maintained cognitive information is predictive of task success (a form of reward).

{#fig:fig-cortical_fun-9 width=50% }

The functional significance of the dorsal vs. ventral distinction has been considerably more controversial in the literature, but anatomically it is clear that dorsal PFC areas interconnect more with the dorsal pathway in the posterior cortex, while ventral PFC interconnects with the ventral posterior cortex pathway. As we saw in the Perception Chapter, the dorsal pathway in posterior cortex is specialized for perception-for-action (How processing): extracting perceptual signals to drive motor control, while the ventral pathway is specialized for perception-for-identification (What processing). This functional specialization in posterior cortex can be carried forward to the associated dorsal and ventral areas of PFC [[@fig:fig-cortical_fun-9]], such that dorsal lateral PFC (DLPFC) areas are particularly important for executive control over motor planning and the parietal cortex pathways that drive motor control, while ventral lateral PFC (VLPFC) areas are particularly important for control over the temporal lobe pathways that identify entities in the world, and also form rich semantic associations about these entities.

On the medial side, the dorsal medial PFC is also known as the anterior cingulate cortex (ACC), which has been shown to encode the affective aspects of motor control variables (e.g., how much effort will an action take, what is its probability of success, how much conflict and uncertainty is there in selecting a response), which is consistent with a "hot how" functional specialization. Dorsomedial PFC areas also project to the subthalamic nucleus within the BG, and serve to delay motor responding to prevent impulsive choice under difficult response selection demands [@Frank06; @AronBehrensSmithEtAl07; @CavanaghWieckiCohenEtAl11]. The ventromedial areas of PFC (VMPFC) including the orbital frontal cortex (OFC) have been shown to encode the affective value of different sensory stimuli, consistent with the idea that they are the "hot what" areas. See [@OReilly10] for more discussion of the What/How functional specialization idea.

Top-down Cognitive Control from Sustained PFC Firing: The Stroop Model

We now turn to perhaps the single most studied task used to test for executive function, the Stroop task, named after John Ridley Stroop, who first described the basic phenomenon [@Stroop35]. The computational model of this task, developed initially by [@CohenDunbarMcClelland90], has been applied (with appropriate change of labels) to a remarkably wide range of different phenomena. Thus, this deceptively simple task and model capture the most critical features of executive function.

{#fig:fig-stroop width=10% }

In the Stroop paradigm ([@fig:fig-stroop]) subjects are presented with color words (e.g., "red", "green") one at a time and are required to either read the word (e.g., "red"), or name the color of the ink that the word is written in. Sometimes the word "red" appears in green ink, which represents the incongruent or conflict condition. The "Stroop effect" is that error rates and response times are larger for this incongruent condition, especially in the case of color naming ([@fig:fig-stroop-data]). That color naming is particularly difficult in the incongruent condition has been attributed to the relatively "automatic", well-practiced nature of reading words, so that the natural tendency to read the word interferes with attending to, and naming, the color of the ink.

The Cohen et al. (1990) Stroop model showed how a maintained PFC task representation can provide a strong top-down bias to support the weaker color processing channel in the face of the stronger word-reading pathway. They were able to establish the difference between word reading and color naming simply as a function of the amount of training provided on each of these tasks. Our simulation reproduces these same core features.

{#fig:fig-stroop-data width=40% }

The Stroop model helps clarify the role of inhibition in executive function. Many people describe the Stroop task as requiring people to inhibit the prepotent word reading pathway, in order to focus on the ink color, and the model also does involve inhibitory dynamics. However, the PFC in the model does not provide a directed form of inhibition to the word reading pathway specifically. Instead, it provides excitatory top-down support to the weaker pathway (color naming), which then enables this pathway to better compete (via lateral inhibitory interactions) with the more dominant word reading pathway. Thus, inhibition is seen as a more collateral, automatic process operating throughout the cortex, and top-down biasing is involved in exciting relevant information, rather than inhibiting irrelevant information. Similarly, people have viewed the A-not-B task as requiring inhibition of the prepotent response to repeat a previously-successful action, but the A-not-B model and other models of developing executive function [@MortonMunakata02; @StedronSahniMunakata05] instead succeed via excitation of competing relevant options.

Exploration of Top-down Cognitive Control

Open the stroop model in CCN Sims.

Substructure within PFC Areas: Stripes

Within each functional PFC area, there is some interesting topographic organization of neurons into hypercolumns, macrocolumns or stripes (each of these terms is typically associated with a similar type of neural organization, but in different parts of the cortex, with stripes being specific to the PFC; [@LevittLewisYoshiokaEtAl93]). In all areas of cortex, one can identify the smallest level of neural topological organization as a cortical column or microcolumn (to more clearly distinguish it from the larger macrocolumn), which contains roughly 20 pyramidal neurons in a region that is roughly 50 microns across. A stripe contains roughly 100 of these microcolumns, generally organized in an elongated shape that is roughly 5 microcolumns wide (250 microns) by 20 microcolumns long (1000 microns or 1 millimeter). Each such stripe is interconnected with a set of roughly 10 or more other stripes, which we can denote as a stripe cluster. Given the size of the human frontal cortex, there may be as many as 20,000 stripes within the frontal cortex (including motor areas).

In PFC and other areas, neurons within a microcolumn tend to encode very similar information, and may be considered equivalent to a single rate-coded neuron of the sort that we typically use in our models. We can then consider an individual stripe as containing roughly 100 such rate-coded neuron-equivalents, which provides sufficient room to encode a reasonably large number of different things using sparse distributed representations across microcolumns.

Functionally, we hypothesize in the PBWM model that each stripe can be independently updated by a corresponding stripe-wise loop of connectivity with an associated stripe of neurons through the BG system. This allows for very fine-grained control by the BG over the updating and maintenance of information in PFC, as we describe next.

Basal Ganglia and Dynamic Gating

{#fig:fig-bg-gating-science width=75% }

As we discussed in the Motor Control and Reinforcement Learning Chapter, the Basal Ganglia (BG) is in a position to modulate the activity of the PFC, by virtue of its control over the inhibition of the thalamic neurons that are bidirectionally connected with the PFC ([@fig:fig-bg-gating-science]). In the default state of no striatal activity, or firing of indirect (NoGo) pathway neurons, the SNr (substantia nigra pars reticulata) or GPi (globus pallidus internal segment) neurons tonically inhibit the thalamus. This prevents the thalamocortical loop from being activated, and it is activation of this loop that is thought to be critical for initiating motor actions or updating PFC active memory representations. When the striatal Go (direct) pathway neurons fire, they inhibit the tonic SNr/GPi inhibition, thereby allowing the excitatory thalamocortical loop to be activated. This wave of excitatory activation can activate a new population of PFC neurons, which are then actively maintained until a new Go signal is fired.

Phasic DA and Temporal Credit Assignment

Another critical biological mechanism for executive function, which we also discussed in the Motor Control and Reinforcement Learning Chapter, is the firing of phasic dopamine (DA) neurons in the midbrain (ventral tegmental area (VTA) and substantia nigra pars compacta (SNc)). These neurons initially respond to primary rewards (e.g., apple juice), but then learn to fire at the onset of conditioned stimuli (CSs) that reliably predict these primary rewards. This amounts to a form of time travel that solves a critical problem for the PFC active maintenance system: how does the system learn what to maintain, given that the decision for what to maintain typically occurs well in advance of the subsequent value of having maintained something useful. If you think of the maintenance of useful information in the PFC as a kind of CS (because they should reliably be associated with positive outcomes), then the dopamine neurons will learn to fire at the onset of such a CS. Having this phasic DA signal at CS onset can then reinforce the decision to maintain this information in the first place, thus solving the time travel problem.

The computational model described next incorporates this key idea, by having the phasic DA signal at CS onset drive learning of the BG Go neurons that update new information into PFC active maintenance. The model shows that this core idea is sufficient to support the learning of complex executive function tasks.

The PBWM Computational Model

{#fig:fig-pbwm-architecture-mnt-out width=40% }

The biological properties of the PFC/BG system that we reviewed above are captured in a computational model called PBWM (prefrontal cortex basal ganglia working memory) [@OReillyFrank06; @HazyFrankOReilly06; @HazyFrankOReilly07] ([@fig:fig-pbwm-architecture-mnt-out]). The PFC neurons in this model are organized into separately-updatable stripes, and also into separate functional groups of maintenance and output gating (described more below). Furthermore, each PFC stripe is represented in terms of superficial layers (2,3) and deep layers (5,6) --- the deep layer neurons specifically have the ability to sustain firing over time through a variety of mechanisms, representing the effects of NMDA and mGluR channels and excitatory loops through the thalamus. The flow of activation from the superficial to deep layers of a given PFC stripe is dependent on BG gating signals, with the BG layers also organized into corresponding maintenance and output gating stripes. The Matrix layer of the BG (representing the matrisomes of the striatum) has separate Go and NoGo neurons that project to a combined GPi and thalamus (GPiThal) layer with a single neuron per stripe that fires if the Go pathway is sufficiently stronger than the NoGo (this mechanism abstracts away from the detailed BG gating circuitry involving the GPe, GPi/SNr, STN and thalamus, as simulated in the motor chapter, and simply summarizes functionality in a single GPiThal layer). A GPiThal Go signal will update the PFC deep layer activations to reflect the current superficial layer activations, while a NoGo leaves the PFC alone to continue to maintain prior information (or nothing at all).

The PVLV phasic dopamine system drives learning of the BG Go and NoGo neurons, with positive DA bursts leading to facilitation of Go and depression of NoGo weights, and vice-versa for DA dips --- using the same reinforcement learning mechanisms described in the Motor Control and Reinforcement Learning chapter.

Perhaps the single most important key for understanding how the PBWM system works is that it uses trial and error exploration of different gating strategies in the BG, with DA reinforcing those strategies that are associated with positive reward, and punishing those that are not. In the current version of the model, Matrix learning is driven exclusively by dopamine firing at the time of rewards, and it uses a synaptic-tag-based trace mechanism to reinforce/punish all prior gating actions that led up to this dopaminergic outcome. Specifically, when a given Matrix unit fires for a gated action, synapses with active input establish a synaptic tag, which persists until a subsequent phasic dopaminergic outcome signal. Extensive research has shown that these synaptic tags, based on actin fiber networks in the synapse, can persist for up to 90 minutes, and when a subsequent strong learning event occurs, the tagged synapses are also strongly potentiated [@RedondoMorris11; @Rudy15; @BoschHayashi12]. This form of trace-based learning is very effective computationally, because it does not require any other mechanisms to enable learning about the reward implications of earlier gating events. In earlier versions of the PBWM model, we relied on CS (conditioned stimulus) based phasic dopamine to reinforce gating, but this scheme requires that the PFC maintained activations function as a kind of internal CS signal, and that the amygdala learn to decode these PFC activation states to determine if a useful item had been gated into memory. Compared to the trace-based mechanism, this CS-dopamine approach is much more complex and error-prone. Instead, in general, we assume that the CS's that drive Matrix learning are more of the standard external type, which signal progress toward a desired outcome, and thus reinforce actions that led up to that intermediate state (i.e., the CS represents the achievement of a subgoal).

The presence of multiple stripes is typically important for the PBWM model to learn rapidly, because it allows different gating strategies to be explored in parallel, instead of having a single stripe sequentially explore all the different strategies. As long as one stripe can hit upon a useful gating strategy, the system can succeed, and it quickly learns to focus on that useful stripe while ignoring the others. Multiple stripes are also critical when more than one piece of information has to be maintained and updated in the course of a task --- indeed, it is this demand that motivated the development of the original PBWM model to supersede earlier gating models, which used phasic dopamine signals to directly gate PFC representations but did not support multiple gating and hence was limited to a capacity of a single item. One interesting consequence of having these multiple stripes is that "superstitious" gating can occur in other stripes --- if that gating happens to reliably enough coincide with the gating signals that are actually useful, it too will get reinforced. Perhaps this may shed light on our proclivity for being superstitious?

Output Gating

{#fig:fig-mnt2out width=75% }

As we saw in [@fig:fig-SommerWurtz00-fig2], some PFC neurons exhibit delay-period (active maintenance) firing, while others exhibit output response firing. These populations do not appear to mix: a given neuron does not typically exhibit a combination of both types of firing. This is captured in the PBWM framework by having a separate set of PFC stripes that are output gated instead of maintenance gated, which means that maintained information can be subject to further gating to determine whether or not it should influence downstream processing (e.g., attention or motor response selection). We typically use a simple pairing of maintenance and output gating stripes, with direct one-to-one projections from maintenance to output PFC units, but there can be any form of relationship between these stripes. The output PFC units are only activated, however, when their corresponding stripe-level BG/GPiThal Go pathway fires. Thus, information can be maintained in an active but somewhat "offline" form, before being actively output to drive behavior. [@fig:fig-mnt2out] illustrates this division of labor between the maintenance side and the output side for gating and how a "handoff" can occur.

For more PBWM details, including further considerations for output gating, how maintained information is cleared when no longer needed (after output gating), and gating biases that can help improve learning, see the PBWM Details Appendix, which also includes relevant equations and default parameters.

Dynamic Updating of PFC Active Memory: The SIR Model

Having seen in the Stroop and A-not-B models how sustained PFC activity can influence behavior through top-down biasing, we now turn to the more complex aspects of PFC function, involving the dynamic gating of PFC representations by the basal ganglia, and the ability to rapidly update and robustly maintain information. As a first introduction to this functionality, captured by the PBWM model, we use the simple SIR (Store, Ignore, Recall) task. Here is a sample sequence of trials in this task:

• S - A --- this means that the network should store the A stimulus for later recall --- network responds A.

• I - C --- ignore the C stimulus, but you still have to respond to it --- network responds C.

• I - B --- ignore the B stimulus --- network responds B.

• R --- recall the most recently stored stimulus --- network responds A.

The BG maintenance gating system has to learn to fire Go to drive updating of PFC on the Store trials to encode the associated stimulus for later recall. It also must learn to fire NoGo to the ignore stimuli, so they don't overwrite previously stored information. Finally, on recall trials, the output BG gating mechanism should drive output of the stored information from PFC. It is critical to appreciate that the network starts out knowing nothing about the semantics of these various inputs, and has to learn entirely through trial-and-error what to do with the different inputs.

Exploration of Dynamic Updating of PFC Active Memory

To see this learning unfold, open the sir model in CCN Sims.

While we don't consider it here for simplicity, the real power of this PBWM model comes when it is augmented to have multiple parallel stripes. It can thus learn to separately update and maintain multiple pieces of information in working memory and to retrieve the correct information when needed. A good example of this demand is summarized by the SIR-2 task, where instead of involving a single store and recall task control signal, there are two such signals (i.e. S1 and S2 and R1 and R2). Thus, the network has to learn to separately store two stimuli, update them into separate buffers, and appropriately respond based on the maintained information in the correct buffer when cue to recall R1 vs. R2. More recent variants of this model have been applied to working memory for continuous stimuli (e.g., colors) allowing the model to account for varying precision of memory representations (Soni & Frank, 2024; see https://elifesciences.org/reviewed-preprints/97894 ). This model further learns an adaptive gating strategy wherein it can merge several memory representations together into "chunks", which improves performance (even beyond a model that is endowed with a larger number of stripes). This matches human performance and provides an explanation for the limitations of working memory capacity based on the difficulty of learning how to manage multiple items in memory, which is mitigated by chunking.

More Complex Dynamic Updating of PFC Active Memory: The N-Back Task

The N-back task has become one of the most widely used measures of complex working memory function in the PFC, in part because it so reliably drives the activation of the PFC in functional MRI (fMRI) experiments. Chatham et al (in press) developed a PBWM-based model of this task, which shows how a more complex cognitive task can be learned by PBWM. This model makes contact with a range of important findings in the cognitive neuroscience literature as well.

Hierarchical Organization of PFC: Subtasks, Goals, Cognitive Sequencing

{#fig:fig-frank-badre12-pfc-hierarch width=75% }

For related models simulating hierarchical control over action across multiple PFC-BG circuits, see [@ReynoldsOReilly09; @FrankBadre12; @CollinsFrank13]. The latter model considers situations in which there are multiple potential rule sets signifying which actions to select in particular sensory states, and where the appropriate rule set might depend on a higher level context. (For example, your tendency to greet someone with a hug, kiss, handshake, or wave might depend on the situation: your relationship to the person, whether you are in the street or at work, etc. And when you go to a new country (or city), the rule set to apply may be the same as that you've applied in other countries, or it might require creating a new rule set). More generally, we refer to the higher level rule as a "task-set" which contextualizes how to act in response to many different stimuli. Hierarchical PFC-BG networks can learn to create these PFC task-sets, and simultaneously, which actions to select in each task-set ([@fig:fig-frank-badre12-pfc-hierarch]).

Critically, with this hierarchical representation, the learned PFC representations are abstract and independent of the contexts that cue them, facilitating generalization and transfer to other contexts, while also identifying when new task-sets need to created. They also allow for new knowledge to be appended to existing abstract task structures, which then can be immediately transferred to other contexts that cue them (much like learning a new word in a language: you can immediately then re-use that word in other contexts and with other people). To see this network in action, including demonstrations of generalization and transfer, see the Collins & Frank network linked here. Various empirical data testing this model have shown that indeed humans (including babies!) represent such task-sets in a hierarchical manner (even when not cued to do so, and even when it is not beneficial for learning) in such a way that facilitates generalization and transfer; and that the extent of this hierarchical structure is related to neural signatures in PFC and BG [@BadreFrank12; @CollinsFrank16a].

To put many of the elements explored above to their most important use, we explore how the coordinated interactions of various regions of the PFC (including the affective areas explored previously), together with BG gating, enable the system to behave in a coherent, task-driven manner over multiple sequential steps of cognitive processing. This is really the hallmark of human intelligence: we can solve complex problems by performing a sequence of simpler cognitive steps, in a flexible, adaptive manner. More abstract cognitive models such as ACT-R provide a nice characterization of the functional properties of this level of cognition. The goal with the model we explore here is to understand how more detailed neural mechanisms can work together to produce this functionality.

• Higher (more anterior) levels of PFC encode context/goals/plans to organize sequence of cognitive actions, which are driven by more lower, more posterior PFC areas. Critically, these higher areas do not specify rigid sequences of actions, but rather encode the desired outcome states of the sequence of actions, and provide appropriate context so that appropriate lower-level steps will be selected.

• Each step in a sequence of actions involves a consideration of the reward outcomes and effort costs of the action relative to other possible options.

Affective Influences over Executive Function: Roles of the OFC and ACC

One of the most important features of the PFC and executive function is that it integrates emotional and motivational influences together with high-level cognitive control and planning. The medial and ventral regions of the PFC are particularly important for processing emotional and motivational factors, with considerable data converging on the idea that the ventral medial areas including the orbital prefrontal cortex (OFC) are important for encoding the affective value of stimuli, while the dorsal medial areas (principally the anterior cingulate cortex (ACC)) is important for encoding the affective value of motor actions and plans.

Other Executive Functions

The main models explored above are intended to cover some of the most central and important aspects of executive function, but this is a very large space and there are many important phenomena that we unfortunately cannot cover (though we plan to expand the scope of what is covered in future revisions, with optional models covering various of the following topics).

For many people, particularly in an academic setting, the first things that may come to mind if asked to name some higher-level cognitive functions might be things like: learning and/or using formal mathematics (like calculus or statistics); or, perhaps, the use of careful logical reasoning to make a major decision. But, in addition to these highly formalized domains, there are many other day-to-day, but none the less important, mental activities that also involve a highly sophisticated level of processing, activities like: planning one's day or a work project, or; resisting the temptation to have dessert when you are trying to lose ten pounds before bathing suit season, or; counting cards in working memory while playing blackjack. All these kinds of mental activities are now known to rely upon the frontal cortex and related structures for their optimal expression. Here is a list of some major categories of distinctive executive functions:

• Highly structured cognitive activities, often involving formal symbol systems --- Mental activities like learning and/or using mathematics, formal logic, computer programming, creative and/or non-fiction writing, and structured, rational decision-making. All of these require temporally-extended maintenance of task-relevant information, especially of a highly abstract, symbolic nature. The role of language in these and many other executive functions is a very important aspect --- language provides a highly flexible mental currency for active maintenance and control over behavior --- by remembering specific words or phrases, we can remind ourselves of what we want to achieve, or what we have derived in an initial processing step, etc.

• Control over encoding and retrieval of episodic information in the hippocampus --- it is highly likely that the hippocampus and PFC/BG systems interact significantly in many forms of executive function, with the rapid learning abilities of the hippocampus complementing the transient, flexible active maintenance properties of the PFC. If the PFC gets distracted, the information is typically gone from active maintenance forever, but the hippocampus can encode and retrieve information in terms of long-lasting synaptic changes. Often, it may be more efficient to use this hippocampal encoding and retrieval instead of persistent active maintenance of information in PFC.

Alternative frameworks and modeling approaches

In this chapter we have focused on one particular theoretical framework, but there have been many other approaches described over the years. Probably the most influential model came from [@Baddeley86], who especially focused on working memory, but also argued for a "central executive." In particular, he postulated two specific forms of working memory: 1) a phonological loop for maintaining verbal information and; 2) a visuospatial scratchpad for spatial information. Another highly influential theoretical approach came from [@Shallice88] who described a supervisory attentional system (SAS) framework. Finally, there is also the very influential traditional AI approach, which we will discuss briefly below.

Motivated largely by the kinds of cognitive functions listed above, traditional AI has largely focused on a design-oriented approach using symbols that has focused on trying to figure out what it would take to solve a particular kind of problem, and then designing a model that does things that way. There is an irony in this approach in that researchers taking this approach are using the very higher-level cognitive functionality they are trying to explain in order to design a system that will reproduce it. A fundamental problem with this kind of approach is that it basically designs in the very functionality it aims to explain. This is not to say that these kinds of approaches are without merit, only that they are limited in what they can ultimately explain. Perhaps for obvious reasons, these kinds of models of cognitive function have been most successful in dealing with the kinds of cognitive function that we listed as being at the highest level - that is, in modeling systems able to do formal mathematics and logic. What they have done less well in has been in accounting for many of the kinds of things that might be considered less high-level, or even lower-level, things which we often take to be automatic. It is for these latter areas that the biologically informed neural network approach has been most helpful. Thus, these two approaches can be nicely complementary and hybrid approaches are being pursued. For example, the Leabra approach is being hybridized with the ACT-R approach in an architecture called SAL.

All of these approaches are not mutually exclusive, but instead share many common ideas and can be complementary in many ways. In particular, the traditional AI approach, by going straight to solving a high level problem (e.g., arithmetic) can address some of our most advanced cognitive functions. On the other hand, the goal of the neural network approach we advocate is to provide a more bottom-up model that tries to provide a reductionist account for the emergence of control-like processing based on underlying automatic mechanisms. This is the approach we take with the PBWM framework.

Summary of Key Points

• The prefrontal cortex (PFC) encodes information in an active state through sustained neural firing, which is more flexible and rapidly updatable than using synaptic weight changes.

• The basal ganglia (BG) drives updating (dynamic gating) of PFC active memory states, enhancing flexibility.

• Phasic dopamine signals from midbrain nuclei have the right properties for training BG gating, by transferring reward associations earlier in time to the onset of stimuli that predict subsequent rewards.

• The PFC influences cognitive processing elsewhere in the brain via top-down excitatory biasing. The A-not-B model demonstrates how the emergence of PFC active maintenance abilities can support behavioral changes that infants show as they develop.

*The Stroop model demonstrates how the active maintenance of task information can guide behavior and help to overcome conflicting options.

• BG dynamic gating can support flexible cognitive function by dynamically encoding some information while ignoring other irrelevant information, and updating the contents of active memory. The SIR and n-back models demonstrate these abilities.

• Medial and ventral areas of PFC (orbital prefrontal cortex (OFC) and anterior cingulate cortex (ACC)) convey affective information about stimuli and actions, respectively, and are important for properly evaluating potential actions to be taken (decision making, problem solving, etc).