Methods for comparing scanpaths and saliency maps: strengths and weaknesses

Abstract

Eye tracker provides objective and quantitative evidence of eye movements. These data are used in a wide variety of applications. In this paper, we are interested in the computational modelling of visual attention. We report on methodologies commonly used to assess performance of these kinds of models. We survey the strengths and weaknesses of common assessment methods. Following this review, we illustrate the use of some methods to benchmark computational models of visual attention.

Supplementary materials

• Eye tracking datasets: the following datasets are all freely available on the web. However, we have reformatted all data in order to have a common framework as descrived below:
  For each picture, the raw eye tracking is given with the following formalism in a text file:
  
  • one line per observer;
  • a triplet is given for each visual fixation of the current observer. (px, py) corresponds to the spatial coordinates in the picture plane of the visual fixation, fd corresponds to the fixation duration for this point;
  • the end of line is given by the triplet <-1 -1 -1>

  For instance, the following text file gives the coordinates and the durations of visual fixations for a given picture: 171 253 200 187 241 420 245 235 260 689 361 500 459 191 700 213 247 180 -1 -1 -1 235 219 380 247 209 220 257 209 400 457 179 220 469 177 160 -1 -1 -1 329 231 240 145 267 280 197 257 260 415 229 160 511 195 360 459 197 140 331 235 160 181 257 200 -1 -1 -1 -1 -1 -1 Therefore, for this picture, there are three observers (three lines in the file). The first oberver (first line) made 6 fixations.

Five eye tracking datasets:

Characteristics of eye-tracking datasets freely available on the web. Download original pictures (ppm, type 6 or bmp) and eye data with the appropriate format (If you intend to use one of these datasets, please cite the paper accordingly. Bibtex of these papers are given at the end of this page.):


Le Meur's database [1]	[original pictures] (22Mb), [visual fixations data]
MIT's database [2]	[original pictures] (1260Mb), [visual fixations data]
Bruce's database [3]	[original pictures] (75Mb), [visual fixations data]
Kootstra's database [4]	[original pictures] (38Mb), [visual fixations data]
Ehinger's database [5]	[original pictures] (800Mb), [visual fixations data]

From left to right, top to bottom: original picture, fixation map, highlighted map, average scan path, heat map, hot map.

Software

• User interface: an example is given below. The version V2.1 is now available. A password is requested to unzip the archive. Please contact me (olemeur at irisa.fr), I will send you it as quick as possible.
  
  NEW ! [fixation analysis software] (7Mo) NEW !
  
  [READ ME]
  
  Please cite the following paper in any published work if you use it:
  
  O. Le Meur and T. Baccino, Methods for comparing scanpaths and saliency maps: strengths and weaknesses, Behavior Research Methods, 2012 (http://dx.doi.org/10.3758/s13428-012-0226-9).
• A simple program to convert SMI file into STAT files (required by the fixation analysis software). [smiToStat] (Many thanks to Rubiela Silva (LUTIN UserLab)for her software implementation!)

BibTex

@Article{LeMeur_2013,
  author = {Le Meur, Olivier and Baccino, Thierry},
  title = {Methods for comparing scanpaths and saliency maps: strengths and weaknesses},
  journal = {Behavior Research Methods},
  volume={45},
  number={1},
  pages = {251-266},
  note = {10.3758/s13428-012-0226-9},
  affiliation = {Université de Rennes 1. IRISA, Campus universitaire de Beaulieu,
	35042 Rennes, France},
  issn = {1554-351X},
  keyword = {Behavioral Science},
  publisher = {Springer New York},
  url = {http://dx.doi.org/10.3758/s13428-012-0226-9}
}

Dataset's reference

[1]

@Article{LeMeur_2006,
	author  = {Olivier Le Meur and Patrick Le Callet and Dominique Barba and Dominique Thoreau},
	title   = {A coherent computational approach to model the bottom-up visual attention},
	journal = {IEEE Pattern Analysis and Machine Intelligence},
	volume	= {28},
	pages 	= {802--817},
	year    = {2006}
}	
	

[2]

@InProceedings{Judd_2009,
	author    = {Tilke Judd and Krista Ehinger and Fr{\'e}do Durand and Antonio Torralba},
	title     = {Learning to Predict Where Humans Look},
	booktitle = {IEEE International Conference on Computer Vision (ICCV)},
	year      = {2009}
}
	

[3]

@InProceedings{Bruce_2006,
	author 	= {Neil Bruce and John Tsotsos},
	title 	= {Saliency Based on Information Maximization},
	booktitle = {Advances in Neural Information Processing Systems 18},
	publisher = {MIT Press},
	year 	= {2006},
	editor 	= {Y. Weiss and B. Sch\"{o}lkopf and J. Platt},
	pages 	= {155--162},
	address = {Cambridge, MA}
}
	

[4]

@Article{Kootstra_2011,
	author    = {Gert Kootstra and Bart de Boer and Lambert Schomaker},
	title     = {Predicting Eye Fixations on Complex Visual Stimuli Using Local Symmetry},
	journal   = {Cognitive Computation},
	volume    = {3},
	number    = {1},
	year      = {2011},
	pages     = {223--240}  
}
	

[5]

@Article{Ehinger_2009,
	author 	= {Ehinger, Krista A. and Hidalgo-Sotelo, Barbara and Torralba, Antonio and Oliva, Aude},
	title 	= {Modeling Search for People in 900 Scenes: A combined source model of eye guidance},
	journal = {Visual cognition},
	year 	= {2009},
	volume 	= {17},
	pages 	= {945--978},
	number = {6-7} 
}