A Review of Eye Movement Tracking Research
By Matt Langeman
99346078
For SYDE 740
Introduction
Back in 1947 the group of Fitts, Jones, and Milton used motion picture cameras to study the eye movements of pilots as they used cockpit controls and instruments while landing a plane. This is the first know study to use eye movements in the field of usability engineering. However, based on citation records, it seems to have received little interest at the time. Instead Fitts would go on to become famous for Fitts’ Law, which he published in a paper in 1954.
Eye movement tracking is a research method that has a long history. The following paper will attempt to provide an historical overview of eye movement tracking, follow the progression of the method up to current day, propose areas of research, and overview the best practices for future studies.
In order to do a thorough historical search on eye movement tracking, Yarbus’ Eye Movement and Vision was chosen as a seminal article. First printed in 1965, it is obviously not the earliest study of eye movement. However, it is frequently cited and provides an extensive review of previous methods of eye movement tracking. Furthermore, it gives a detailed account of Yarbus’ original invention of a device he used for eye movement tracking.
The following is a summary of Yarbus’ findings on early methods of eye movement study. Direct visual observation was the first method used to study eye movements. Some experimenters (*Javal 1879) stood behind the subject and used a mirror to unobtrusively observe the subject’s eye movement. This method was used to observe eye movement during reading and only allowed for a general characterization of large eye movements. Later research (*Newhall, 1928) used lens to magnify the image of the eye in order to increase the accuracy of direct visual observation. Other methods used as far back as 1898 involved a mechanical connection between the eye and a recording instrument. Orschansky (*1899) was the first known experimenter to use the reflection of a beam of light to project eye movements onto a screen. A cup similar to a contact lens was place on the eye and a small mirror was attached to the cup. All of these early methods were complicated, required anesthetization of the eye’s surface and had low accuracy. Furthermore, the subject’s head had to be held still, usually being placed in a headrest. Dodge and Cline (*1901) are credited as the first to use photographic methods to study eye movement. Further refinements of this photographic method often involved placing a small spot on the subject’s eye in order to more accurately measure eye movement. Barlow (*1952) apparently placed a small drop of mercury on the subject’s cornea and another on the subject’s forehead. A microscope was then used to project and record the drops onto a moving film. These photographic methods were less invasive than mechanical methods and were successful in recording large eye movements. The main disadvantage of the method was the tedious and lengthy nature of analyzing the recorded data. Schott (*1922), Meyers (*1929), and Jacobson (*1930) pioneered the use of electrooculographic methods. These methods used the potential difference between the outer and inner sides of the retina or between the cornea and the sclera. During eye movements the potential difference changes. These changes can be detected by electrodes attached to the skin to the right and left of the eye. The main advantages of these methods are that it does not involve attaching anything directly to the eye, it allows for full freedom of head movement and it measures both vertical and horizontal movement at the same time. However it is only recommended for experiments that do not require a great deal of accuracy.
The following is a summary of Yarbus’ personal research and experimentation. The lack of viable options for accurately measuring eye movements led Yarbus to develop his own method. The most innovative aspect of the method was the use of a suction device that was affixed to surface of the subject’s eye. The suction device, referred to as a “cap” by English translations, was designed to be lightweight in order to minimize its affect on eye movement. A small mirror was attached to the caps in order to record eye movements using reflected beam of light. Experiments using the caps required anesthetizing the eye’s surface and taping open the subject’s eyelids. Yarbus includes detailed instructions on building these caps as well as descriptions and diagrams of the recommended processes to use the cap device.
Yarbus used his method to conduct a variety of experiments including: perception of objects stationary relative to the retina, eye movements during fixation of stationary objects, saccadic eye movements, eye movements during change of stationary points of fixation in space, eye movements during perception of moving objects, and eye movements during perception of complex objects. Some of his most interesting findings were related to these last experiments on perception of complex objects. For one set of experiments, subjects were told to examine a specific picture and their eye movements were recorded during this free examination. Analyses of the recordings show that “the eye rests much longer on some of [the pictures elements] than on others” (Yarbus, p 171). Yarbus postulates that these elements “may contain information useful and essential for perception.” He further states that “eye movements reflect the human thought processes; so the observer’s thought may be followed to some extent from records of eye movements” (Yarbus, p 190). In an extension of these picture studies, Yarbus used Repin’s picture “An Unexpected Visitor” and recorded the eye movements of subjects while they performed seven different tasks including such things as giving the ages of the people, remembering the clothes worn by the people and surmising what the family had been doing before the arrival of the “unexpected visitor” (Yarbus, p 174). The recordings show that “depending on the task facing the subject, the eye movements varied” (Yarbus, p 192). Yarbus notes that “eye movements after instruction are interesting because they help in the analysis of the significance of eye movements during the free examination of a picture; they show that the importance of the elements giving information is determined by the problem facing the observer, and that this importance may vary within extremely wide limits” (Yarbus, p 193). In concluding his analysis of these experiments Yarbus stresses that “the distribution of points of fixation on an object, the order in which the observer’s attention moves from one point of fixation to another, the duration of the fixations, the distinctive cyclic pattern of examination, and so on are determined by the nature of the object and the problem facing the observer at the moment of perception” (Yarbus, p 196).
Evolution of Method
At the time of Yarbus’ research, eye movement tracking was still very complex and methods used for measurement had large trade-offs between accuracy of information and freedom and unobtrusive use for the subjects. Other researchers such as Shackel (1960) and Mackworth & Thomas (1962) were also making advances in head-mounted eye tracking systems that were slightly less obtrusive.
Jacob and Kern (2003) cite two events as crucial in transforming eye movement tracking into a more viable method of research. First was the discovery by Cornsweet and Crane (1973) “that multiple reflections from the eye could be used to dissociate eye rotations from head movements.” This discovery increased the precision of eye movement tracking and paved the way for two military/industry teams (U.S. Airforce/Honeywell Corporation and U.S. Army/EG&G Corporation) to develop remote eye tracking systems. These new systems drastically reduced obtrusiveness and increased freedom of movement for subjects. Secondly, the two military/industry teams then developed systems to automate the analysis of eye tracking data. This advancement, aided by the invention of the minicomputer, reduced the labor needed for data analysis and allowed for real-time analysis of eye tracking studies. Thus methods of eye movement tracking were beginning to overcome the disadvantages raised by Yarbus: the lack of accuracy, the amount of intrusiveness, and the laborious nature of data collection and analysis. As a result, researchers found more areas to incorporate eye movement studies.
Research attempting to relate eye movements to cognitive processes continued throughout the 1970’s (Monty 1975, Just 1976a, 1976b). Many of these studies were related to reading and information processing. A comprehensive history of these studies was performed by Rayner (1998) and is beyond the scope of this paper. One interesting subset of this area is the relationship between eye movement and attention (Posner, 1980). Movement of attention does not always correspond to a movement of eyes. However for complex tasks such as reading, the link between locus of attention and location of eyes in likely very close.
Studies continued on scene perception, similar to the experiments performed by Yarbus.
One study (Loftus, 1981) focusing on scene perception had the following propositions: (1) A normal fixation on a picture is designed to encode some feature of the picture, (2) the duration of a fixation is determined by the amount of time required to carry out the intended feature encoding, and (3) the more features are encoded from a picture, the better the recognition memory will be from the picture. Loftus had difficulties with his method of simulating saccades using tachistoscopic flashes. It was unclear whether the flashes caused subjects to move their location of fixation as intended, or whether they held their eyes steady throughout the flash (Duchowski, 2002). Further study (Rayner and Pollatsek, 1992) indicates that information about the background or setting of a scene is obtained during the initial fixation. Furthermore, while information about details and objects located far from the point of fixation can be obtained, important objects usually receive fixation.
In 1981, Bolt proposed the use of eye fixation and movement as an input device for computers. Stating that “the machine had ought to have instrumentation to capture the modes of expression natural to people,” Bolt and his colleagues were “concentrating upon displays that respond to what you are saying, where you are pointing, and most recently, displays that know where you are looking.” Two of the first working eye-based interactive systems where introduced in 1990, one by Bolt & Starker and another by Jacob. Jacob’s paper introduced a methodological problem of determining user intent when using eye-tracking in place of a mouse for cursor movement. His resolution at the time was to use dwell time as the method of selection to replace the mouse click. Eye-based interaction continues to be researched and has been shown to be significantly faster than hand pointing (Tanriverdi and Jacob, 2000). It also clearly has significant practical benefits for disabled people. However, technical issues cause trade-offs between obtrusive head-gear and desktop systems that restrict head movement. This is still a problem that limits widespread adoption outside of experimental settings (Vertegaal, 2002).
While eye movement research in the usability engineering field does not appear in publication throughout the 1970’s, Jacob and Karn (2003) attribute the lull in usability engineering research using eye movement tracking to the huge amounts of effort and time required for data collection and analysis. As mentioned, the computer helped ease the burden of data collection and analysis. The computer also brought a whole new area for eye movement research. Computer interfaces have received a significant amount of study using eye movement tracking. Card (1984) studied people’s eye movements when they search pull-down computer menus. He found that searches were initially faster using alphabetically arranged menus as opposed to menus grouped by function. The difference was related to difficulty visually finding target items. However, with experience all menus arrangements tended to be equivalent, requiring only one saccade movement to find a menu item. Card also concluded that people learn item locations using in perceptual chunks. An item’s location was often remembered along with what items came directly before and after it. Also, if menu items were divided into a box, an item would be remembered along with the other items in that box. MacGregor and Lee (1987), Hendrickson (1989), Hornof and Kieras (1997), and McCarthy, Sasse, and Riegelsberger (2003) continued the use of eye movement tracking to research computer-based menus.
Current Research
The study by McCarthy, Sasse, and Riegelsberger
(2003) examines the conflicting advice given for the best position of the
navigational menu on websites. Convention dictates that the navigational menu
should appear on the left side of the screen. Nielsen (1999) found that “success rate for product
search is 80% when menu labels conformed to expectations. This drops to 9% with
unfamiliar menu labels.” Based on this finding Jacob Nielsen proposed “Jacob’s
Law of web user experience”: “Users spend most of their time on other sites. Thus, anything that is a
convention and used on the majority of other sites will be burned into the
users’ brains and you can only deviate from it on pain of major usability
problems.” However, the National Cancer Institute Guidelines suggest placing
the navigational menu on the right-hand side of the page. This advice is based
on a study by Baily, Koyani and Nall (1999) indicating clicking of the
navigational menu was more efficient when it was located closer to the web
browser’s scroll bar. In an attempt to resolve this conflict, McCarthy, Sasse
and Riegelsberger conducted studies of three menu placements: left, right, and
top. Each subject performed nine tasks. Besides measuring task time, eye
movements were recorded in order to determine where subjects looked on the
screen. It was found that use of the left menu initially led to faster task
times. However, after a few uses, the differences in task times were
insignificant. It terms of eye glances, when first using pages with the
right-hand menu placement, subjects tended to glance towards the left-hand side
of the page. These left-hand glances continued but were reduced over subsequent
uses. Furthermore they did not appear to affect task completion time. The
conclusion of the study was that violating the convention of left-hand menu
placement does not have long-term effects. Thus alternative menu placements
should be explored, especially if studies indicate better possibilities.
The area of computers and the web have recently received a large amount of research interest. This trend will likely continue in the near future. Jacob and Kern (2003) suggest future studies related to apparent differences found between novices, and experienced participants when performing tasks. It has been found that eye movements of experienced users differ significantly from those of novice users. Study of these differences could contribute to understanding how users progress from a novice to an expert. Understanding of this progression could have implications for increasing user efficiency in terms of email usage and web search. Jacob and Kern (2003) also suggest the need for search studies that are more realistic noting that “in a typical human-computer interface, [uses] do not have a good representation of the target. Most of the literature in visual search starts with the participant knowing the specific target. We need more basic research in visual search when the target is not known completely.”
Eye-tracking also has a bright future in the area of marketing and advertising. This field offers the opportunity of continuous new studies based on different websites and products. One interesting study was performed on Google’s results page. Conducted by search marketing firms Enquiro and Did-it and eye tracking firm Eyetools, the study found that results ranked lower than seventh were viewed by less than 50% of the views (Enquiro, 2005). Studies based on sites such as Yahoo or Ebay could also lead to interesting conclusions. If fact most websites could benefit from an eye movement tracking study, although the benefit may not outweigh the cost.
Another area of research for eye-tracking studies is that of mobile devices. As the usage of cell phones, PDA’s, mp3 players, etc. continues to grow, it would be interesting to investigate the eye movements of users while using these devices. Clearly there would be significant technical difficulties in simulating the environment of mobile device use. However it also seems likely that eye movement data could indicate areas for usability improvement. Furthermore, given the propensity for these devices to be used during situations such as driving, eye movement tracking could lead to recommendations for improving safety.
The preceding is by no means a comprehensive history of research using eye movement tracking. An overwhelming body of research exists around this topic and it continues to grow rapidly. The review is useful in beginning to understanding how eye movement tracking has evolved as a research method and how it can be used in the future. In terms of current practices it is important to note that there are still technical problems related to capturing eye movement data. McCarthy, Sasse, and Riegelsberger (2003) note that they “used an eye tracking system that does not require head restraint. This meant that when participants changed body posture too quickly or moved out of the tracking field, data was lost.” Thus in order to “ensure high-quality data, [they] excluded from analysis any participant who was tracked for less than 90% of the time.” These technical difficulties mean that careful decision is still needed to determine the trade-offs between different eye tracking systems. The need for accurate data, the environment of study, and the effect of the tracking device on users are all significant factors in choosing the appropriate system.
Collecting eye tracking is often still a relatively labor-intensive process, depending upon equipment and experiment design. Studies based on subjects performing unstructured tasks may require looking at frame-by-frame captures of video data.
Current research stresses the benefits of eye tracking as being a small window into what the user is thinking. While not a complete solution, it does offer information about a user’s thought process while completing a task. In human-computer interaction, eye tracking data is most useful when combined with data such as mouse movements and clicks.
It is also useful to note the most frequent metrics used when conducting eye movement tracking studies. Jacob and Karn used a sampling of twenty eye movement based usability studies to identify the six most frequently used metrics. Their findings are summarized below.
(1) Number of fixations, overall: this metric is thought to be negatively related to search efficiency. Experimenters should account for task length as longer tasks require more fixations.
(2) Gaze percentage on each area of interest: this metric indicates the importance of the various areas of interest. However it should be noted that Fitts, Jones & Milton (1950) treated duration of fixation and number of fixations separately, stating that longer durations relate to difficulty and more frequent durations relate to importance.
(3) Fixation duration mean, overall: this metric indicated the difficulty of extracting information. Longer fixations and longer gazes imply greater difficulty.
(4) Number of fixations on each area of interest: this metric, similarly to gaze percentage, corresponds to the importance of an element or area of interest.
(5) Gaze duration mean, on each area of interest: this metric is also related to difficulty of information extraction. Elements that are difficult to process are predicted to receive longer gazes.
(6) Fixation rate overall: this metric is related to fixation duration. The time between fixations is small compared to time spent fixating. Thus fixation rate should be inversely related to fixation duration.
Jacob and Karn also designate several metrics as being promising for future research. A summary of these is listed below:
· Scan path: indicate efficiency of arrangement of interface elements
· Number of gazes on each area of interest: gazes (a group of successive fixations within the same area of interest) can be more meaningful than the number of individual fixations.
· Number of involuntary and number of voluntary fixations: Graf & Kruger (1989) have proposed a distinction between short and long gazes. More research is needed on this classification method.
· Percentage of participants fixating an area of interest: indicates the “attention-getting” properties of an element or area of interest.
· Time of first fixation on target area of interest: used when there is a specific search target.
Conclusion
In conclusion, eye movement tracking has been a research method with over 100 years of history. It has been applied to a wide range of research areas and has evolved significantly over the years. It is interesting to note that many of these metrics suggested by Jacob and Karn (2003) are based on the aircraft pilot studies of Fitts, Jones, and Milton (1950) and many were also measured in the studies by Yarbus (1967). Thus as eye movement tracking lends new insight into areas such as computer-human interaction, it is important to understand and appreciate the early work that laid essential groundwork for future studies.
Works Cited
Bailey, R.W., Koyani, S. and Nall, J. (2000), Usability testing of several health information Web sites, National Cancer Institute Technical Report, September 7-8.
Bolt,
R.A. (1982). Eyes at the Interface, Proceedings
of the ACM Human Factors in Computer SystemsConference, 360-362.
Card, S.K. (1982). User percpetual mechanisms in the search of computer command menus. Proceedings of Human Factors in Computer Systems. SIGCHI, 190-196.
Cornsweet
and Crane (1973). Accurate two-dimensional eye tracker using first and fourth
Purkinje images. Journal of the Optical
Society of America. 63:921-928.
Enquiro: Search Smarter Marketing. Did-it,
Enquiro, and Eyetools uncover Search’s Golden Triangle. Retrieved April 24,
2005 from http://www.enquiro.com/eye-tracking-pr.asp.
Duchowsi,
A.T. (2002). A Breadth-First Survey of Eye Tracking Applications. Behavior Research Methods, Instruments,
& Computers (BRMIC). 33 (4).
Fitts, P. M. (1954). The Information Capacity of the Human
Motor System in Controlling the Amplitude of Movement. Journal of Experimental
Psychology, 47, 381-391.
Fitts,
P. M., Jones, R.E. & Milton, J.L. (1950). Eye movements of aircraft pilots
during instrument-landing approaches. Aeronautical
Engineering Review, 9(2), 24-29.
Graf, W. & Krueger, H. (1989). Ergonomic evaluation of
user-interfaces by means of eye-movement data. In Smith, M.J. & Salvendy,
G. (eds.) Work with Computers: Organizational, Management, Stress and Health
Aspects, 659-665.
Hendrickson, J.J. (1989). Performance, preference, and
visual scan patterns on a menu-based system: implications for interface design.
Proceedings of the SGCHI conference on
Human factors computing systems. 217-222.
Hornof, AJ., and Kieras, D.E. Cognitive modeling reveals menu search is both random and systematic, in Proceedings of CH1’97 (Atlanta GA, March 1997), ACM Press, 107-l14.
Jacob, R. J. K & Karn, K. S. (2003). Eye Tracking in
Human-Computer Interaction and Usability Research: Ready to Deliver the
Promises. The Mind's Eyes: Cognitive and
Applied Aspects of Eye Movements.
Javal,
E. (1878). Essai sur la physiologie de la lecture. Annales d'Oculistique, 79, 97-117, 155-167, 240-274; 80 (1879),
61-73, 72-81, 157-162, 159-170, 242-253.
Just,
M.A., and Carpenter, P.A. (1976a). Eye Fixations and Cognitive Processes,
Cognitive Psychology, 8: 441-480.
Just,
M.A., and Carpenter, P.A. (1976b). The role of eye-fixation research in
cognitive psychology. Behavior Research
Methods & Instrumentation. 8: 139-143.
Loftus,
G.R. (1981). Tachistoscopoic simulations of eye fixations on pictures. Journal of Experimental Psychology: Human
Learning & Memory. 7, 369-376.
MacGregor,
J., and Lee, E. Menu search: random or systematic? Int. J. Man-Machine Studies
26 (1987), 627-63 1.
Mackworth,
N.H. & Thomas, E.L. (1962). Head-mounted eye-marker camera. Journal of the Optical Society of America.
52, 713-716.
Monty, R.A. (1975). An advanced eye-movement measuring and
recording system. American Psychologist.
30, 331-335.
Nielsen,
J. (1999). Do Interface Standards Stifle Design Creativity? Jakob Nielsen’s Alertbox, August 22.
Available at http://www.usit.com/alertbox/990822.html
Shackel,
B. (1960). Note on mobile eye viewpoint recording. Journal of the Optical Society of America. 59, 763-768.
Posner,
M. I. (1980). Orienting of attention. Quarterly
Journal of Experimental
Psychology, 32, 3-25.
Rayner,
K. (1998). Eye movements in reading and information processing: 20 years of research.
Psychological Bulletin, 124, 372-422.
Rayner,
K, & Pollatsek, A. (1992). Eye movements and scene perception. Canadian Journal of Psychlog, 46,
342-376.
Tanriverdi,
V. and Jacob, R.J.K. (2000). Interacting with eye movements in virtual
environments. In the Proceedings of the
ACM CHI 2000 Human Factors in Computing Systems Conference. 265-272.
Vertagaal, Roel
(2002). What do the eyes behold for human-computer interaction? In the Proceedings of the
symposium on Eye tracking research & applications. 59-60.
Yarbus, A.L. (1967) Eye Movements and Vision. (B.
Haigh Trans.). New York: Plenum Press.(Original work published 1965).
*Note: for references under Historical Search, see the back of Yarbus (1967).