SILC Showcase

Showcase July 2014: The order of attentional shifts determines what visual relations we extract

Share this article using our url:

The order of attentional shifts determines what visual relations we extract

Audrey L. Michal, David H. Uttal, and Steven L. Franconeri

Northwestern University


Related research:

  • Franconeri, S. L., Parrott, S., & Uttal, D. H. (2012, January). Visual processing of spatial relationships. Spatial Intelligence and Learning Center (SILC) [Northwestern University]. [URL]

Consider the graph below, which plots levels of friction for various materials. If you wanted to know the relationship between the friction levels for carpet and vinyl, how would you explore the graph? Linguistically, there are two ways to frame this relationship: that carpet has higher friction than vinyl, or that vinyl has lower friction than carpet. Our recent experiments demonstrate that the choice of which ordering is constructed is tightly linked with the way that students explore the graph with their eyes, and that these 'perceptual routines' can have substantial impact on the efficiency of relational extraction.

Figure 1
Figure 1: A typical bar graph depicting levels of friction for multiple materials.

A relationship between two items can be framed in multiple ways depending on which item is the target and which item is the referent. For instance, the relationship between carpet and vinyl can either be framed as “Carpet has higher friction than vinyl” (target = carpet, referent = vinyl) or as “Vinyl has lower friction than carpet” (target = vinyl, referent = carpet; Clark & Chase, 1972). Although this distinction is subtle, it has important consequences for how we interpret relationships depicted by graphs and other symbolic representations. Despite our intuition that these alternative interpretations exist simultaneously our visual system, recent evidence from our lab suggests that we can only visually process one of these framings at a time. While we can quickly pull out surface-level information from the graph (e.g., one of the bars is very tall, all the bar heights are different, etc.), we cannot simultaneously extract that the carpet bar is higher than the vinyl bar and that the vinyl bar is lower than the carpet bar (Franconeri et al., 2012). This limitation may stem from the need to deploy singular spatial attention to an object in order to match its visual properties to its spatial location (Treisman & Gelade, 1980), and this type of matching is critical to relational processing (Hummel & Biederman, 1992).

In support of this, in a previous study, we used eye tracking to show that people judge size relations of bars (e.g., is bar A > bar B?) by inspecting the bars one at a time. On each trial, we showed participants two bars that differed in height, and we varied the relative left/right locations of the shorter and taller bar from trial to trial. Participants responded whether the depicted relation had a [short tall] or [tall short] configuration. We assessed the order in which participants inspected the bars by measuring where participants looked first (i.e., toward the taller or shorter bar). Most participants were biased to look toward the taller bar first, regardless of whether the taller bar was on the left or right. This suggests that most participants framed the size relationship using the taller bar as the target and the shorter bar as the referent. Thus, the order in which people inspected the bars depended on how individuals interpreted the relation.

In the previous study, we did not specify how the size relation should be interpreted; in other words, it was up to the individual to decide whether the taller bar or the shorter bar was the target. How might attentional strategies change if people are instructed to interpret a relation in a specific way? For instance, if a person is asked “Does carpet have higher friction than vinyl?” for the graph shown in Figure 1, are they more likely to look toward the carpet bar first since the relation is framed with carpet as the target?

To test this prediction, we recorded eye movements and response times (RTs) while participants viewed simple graphs depicting relative quantities of fruits (limes, blueberries or oranges). The experimental procedure is outlined in Figure 2. On each trial, participants viewed a question (e.g., “Are there more blueberries than limes?”; “Are there fewer oranges than limes?”), followed by a simple bar graph. Participants then answered the question using a “yes” or “no” response. As in the previous study, the relative left/right locations of the bars representing the target (i.e., first queried fruit) and referent (i.e., second queried fruit) were varied from trial to trial; thus, any biases to look toward the bar representing the target first should occur regardless of whether the target bar is on the left or right. In addition to measuring biases for looking towards the target bar first, we also measured biases for looking towards the left bar first, as left-to-right processing is ubiquitous in Western culture (e.g., Dickinson & Intraub, 2009).

As a secondary hypothesis, we also tested for age group differences in the task to see whether attentional strategies for relation judgments develop over time. Thus, we tested separate groups of six year-olds, eight year-olds and undergraduates on the same experiment.

Figure 2
Figure 2: Experimental procedure and visual routines of interest.

Our results are shown in Figure 3. Biases toward one bar or another (left/right on the top row, and referent/target in the bottom row) can be seen in the shift of the dot clouds toward one bar or another. In addition to measuring participants’ overall biases to look towards the left bar or target bar first, we plot each participant's average RT as their dot height, allowing a view of the correlation of the strength of each individual’s bias with RTs.

Figure 3
Figure 3: Correlations between response time (RT) and first saccade biases as a function of spatial location (left/right bar) and relational role (target/referent).

Since inspecting the target bar first should frame the size relationship in the correct way, we predicted that the target-bar-first routine would be the more efficient routine. In contrast, since the target bar was equally likely to appear on the left and right side, a consistent left-bar-first routine would only lead to a correct framing of the relation on half of the trials (i.e., only for trials in which the target bar appeared on the left). Thus, we predicted that participants with strong target-bar-first biases would be more likely to extract the correct relation framing and respond more efficiently (i.e., faster responses overall), whereas participants with strong left-bar-first biases would be less likely to extract the correct relation framing and respond less efficiently (i.e., slower responses overall).

The six year-old and eight year-old groups showed a strong bias to look towards the left bar first, whereas undergraduates did not show a bias toward either the left or right bar. In line with our predictions, RTs were negatively correlated with leftward biases for both groups of young children, suggesting that children with stronger leftward biases were slower to respond. In contrast, there was no relationship between leftward bias and RT for undergraduates.

For eye movement patterns defined by target-first routines, only the eight year-old and undergraduate groups showed a significant bias to look towards the target bar first. Additionally, RTs correlated with the strength of target-first biases for both of these groups, such that stronger target-first biases were associated with faster RTs. Six year-old participants showed neither an overall target-first bias nor a relationship between target-first bias and RT.

Taken together, these results suggest that there are both optimal (selecting the target object first) and sub-optimal routines (selecting the left object first) for extracting visual relations in graphs, with children showing a greater tendency to attend sub-optimally. The order of attentional shifts accounted for a significant amount of RT variability and thus appears critical for framing 'directionality' of visual relations. Importantly, these results suggest that even seemingly simple graphs are not always intuitive; young students in particular may benefit from guidance about how to inspect graphs in ways that best support comprehension.


  • ♦ Clark, H. H., & Chase, W. G. (1972). On the process of comparing sentences against pictures. Cognitive Psychology, 3, 472–517.
  • ♦ Dickinson, C. A., & Intraub, H. (2009). Spatial asymmetries in viewing and remembering scenes: Consequences of an attentional bias? Attention, Perception & Psychophysics, 71, 1251-1262.
  • ♦ Franconeri, S. L., Scimeca, J. M., Roth, J. C., Helseth, S. A., & Kahn, L. E. (2012). Flexible visual processing of spatial relationships. Cognition, 122, 210–227.
  • ♦ Hummel, J. E., & Biederman, I. (1992). Dynamic binding in a neural network for shape recognition. Psychological Review, 99(3), 480–517.
  • ♦ Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97–136.
You are here: SILC Home Page SILC Showcase Showcase July 2014: The order of attentional shifts determines what visual relations we extract