Research Project: 'A computational model to simulate visual stability from eye movements and spatial attention'

Current research in the problem of visual stability explore specific aspects such as predictive remapping, saccadic suppression of displacement and saccadic compression of space. While this reductionist method has allowed a greater understanding of many mechanisms involved in the overall visual system it can be difficult to incorporate these results into an encompassing theory. The best current theories suggest either a global visual mechanism that assume visual stability (Deubel, 1998) or a low level, local mechanism that stitches perisaccadic information at the neural level (i.e., remapping, Wurtz, 2008). We propose to test these theories by building a large scale computational model of the oculomotor and visual attention systems to test these theories. We will begin with established models of the human visual system with both external (salience, bottom up) and internal (cognitive state, top down) information and extend these to create a full simulation of the visual system built on new and existing experimental data.

Hypothesis one. Treating the computational model as a modular system will allow an interchange of multiple computational methods. For example, the initial test will use a diffusion model for saccadic generation, but switch to a deep learning neural network (Krizhevsky, Sutskever, & Hinton, 2012) for increased biological plausibility. These two solutions will be compared in terms of predictive accuracy as well as how well they reflect the known underlying cognitive and biological processes.

Hypothesis two. The final overall model can be tested as an eye movement simulator in classic eye movement experiments. Data from existing experiments and new experiments will be used to test and improve the overall model. The model will be used as a saccadic simulator to determine the degree it performs in similar way to human participants. By simulating multiple aspects of the vision and attention networks, we will get a clearer understanding of whether global or local models of visual stability fit the full range of experimental and neuroimaging data.

The project was implemented during 2017.

Research goals and objectives

1. Integrated model of visual-attention networks.
   Objective a: Integrate best examples of computational models including bottom-up salience in early striate cortex, top down modulation of spatial attention in the posterior parietal cortex, saccade generation in superior colliculus and saccade control in the frontal eye fields.
   Objective b: Test the integrated model against existing experimental data as a human visual simulation. Experimental data will be selected that specifically tests the role of attention across eye movements (MacInnes et al., 2014a, 2014b; MacInnes & Hunt, 2015; MacInnes, 2017; Gordienko & MacInnes, in preparation; Malevich et al., in press).
2. Established algorithms replaced with new machine learning approaches.
   Objective a) Implement a new model of saccade generation.
   Objective b) Replace diffusion model of saccade generation from goal 1 by the new model.
   Objective c) Compare the two models in their ability to predict experiment results and match current theories of visual stability.
3. New experiments.
   Objective a) use predictions from model to design and implement new experiments to test global vs local theories of visual stability.
   Experiments will include saccadic suppression of displacement (Objective b), predictive remapping (Objective c) and specifically comparing the remapping data with interference to similar components of both model and human participants (Objective d).

Research results and their application

1. We have developed and integrated models of different aspects of attention and vision.
   First, orienting has been implemented as an extension of the classic model of bottom up salience as suggested by Itti & Koch (2000). We have improved this model to include top down attention mechanisms and feedback from other models of attention.
   Second, alerting has been implemented as a temporal diffusion model to simulate the role that the locus Coeruleus, an area located in the brainstem, plays in alerting attention (Posner & Fan, 2008).
   Third, executive functions have been implemented as deep belief and deep convolution networks enabling transitions without the need of attentional homunculus for control.
   Finally, integration of these networks has been based on interactions proposed by Callejas et al., 2004).
2. Model integration:
   We have proposed a modular modeling framework, which allows for interchangeable submodels, which can be implemented at many levels of granularity. This framework exists in the spaces between biologically accurate neural level projects like the European human brain project and current cognitive architectures. Key components of its structure are a modular design for algorithm integration, a visual interface for intuitive model flow specification and allow for sufficient computational complexity to allow for the best of breed parallel and temporal machine learning algorithms.
3. Experiment data and model testing:
   Experiment data on remapping of attention, saccadic generation and the role of attention in guiding saccadic integration is a scientific contribution in their own right and results in top tier scientific publication. In addition, this data served to train and test the final model for accuracy and its ability to test the multiple theories of visual stability.

Applied and practical uses of this model are certainly possible in the field of robotic vision, face recognition and display devices that optimize for human vision and attention.  Computational models of human vision and attention have different goals than applied computer vision, but the techniques and algorithms often have much overlap.  Models and simulations of human vision will help bridge the gap between psychophysiology and these practical applications faster than experimental data alone.

The results have been presented in publications in psychology / neuroscience journals, at conferences and workshops, and during the lecture courses delivered by Dr MacInnes in the HSE.