Individual Differences in Spatial Abilities and the Visualization of Conditional Probabilities

Working Paper

June 2006

 

Vince Kellen

DePaul University

vkellen@depaul.edu

 

Susy Chan

DePaul University

schan@cti.depaul.edu

 

Xiaowen Fang

DePaul University

xfang@cti.depaul.edu

ABSTRACT

In tasks such as disease diagnosis, interpretation of evidence in criminal trials and management of security and risk data, people need to process conditional probabilities to make critical judgments and decisions. As dual-coding theory and the cognitive theory of multimedia learning (CTML) would predict, visual representations (VRs) should aid in these tasks. Conditional probability problems are difficult and require subjects to build a mental model of set inclusion relationships to solve them. Evidence from neurological research confirms the distinction between the “what” and the “where” pathways in visual processing and working memory.  It further confirms that mental model construction relies on the “where” pathway. To solve conditional probability problems, subjects need to build a mental model of the problem, which involves visual spatial processing (the “where” pathway). Prior research has revealed that individuals differ in their ability to perform spatial processing tasks. Do visualization interface designers need to take into account the nuances of spatial processing and individual differences for these problems?  Therefore, this study will use a 3x2 factorial design to determine the relationship between  subject’s spatial abilities (high or low) and representations on user performance and satisfaction (spatial VR, an iconic VR or pure text) and their impact on user performance and satisfaction in solving conditional probability problems. This study will provide guidance on how visualization interfaces should be designed to facilitate this type of task.

Keywords

Information visualization, Bayesian reasoning, conditional probabilities, dual-coding, cognitive theory of multimedia learning, mental models, individual differences, spatial ability

INTroduction

The medical, legal and security disciplines use Bayesian inference in many contexts that require many people who are naïve in Bayesian inference to make critical judgments and decisions involving disease diagnosis and treatment, interpretation of evidence in criminal and civil cases and the management of security data. Prior research has tangentially examined visual representations (VRs) in these Bayesian inference tasks involving conditional probabilities, usually to see how the VR facilitates teaching Bayes’ theorem or to test specific theoretical claims made in prior research. Very little research has compared different VRs and their impact on naïve Bayesian subjects’ performance and none have examined the role individual differences in spatial abilities play in this task.  This research will examine three treatments of a conditional probability Bayesian inference task (text, and spatial and iconic VRs) and how differences in spatial ability affect subject performance with each of these representations.

Prior Research

Information visualization techniques have been applied in a variety of domains. More recently, they have been integrated into general business applications (Mirel, 1998), are finding its way to consumers via popular web sites (Peet’s Coffee Selector, SmartMoney) that use visualization techniques, and are common in popular applications such as SPSS, SAS, Microsoft Outlook and Adobe Photoshop (Plaisant, 2004). Citing several studies in the 1980s and the 1990s, Dull and Tegarden contend that visualization can improve problem-solving capabilities (1999, also Tegarden 1999). Card, Mackinlay, & Shneiderman (1999) propose that information visualization amplifies cognition through a knowledge crystallization process. Spence (2001) defines information visualization as a process by which the visual presentation of information leads to a mental model, or cognitive map; a cyclical process of human activity creates and recreates this mental model. Tufte (1997) provides a compelling argument for proper data visualization to support decision-making in his retrospective analysis of the documents NASA engineers used to unsuccessfully convince NASA managers to scrub the doomed Challenger launch in January 1986. On the day of the launch, managers and engineers discussed canceling the launch due to unusually cold weather. Engineers familiar with the problem had correctly anticipated the problem – the now infamous O-rings would fail in the cold weather – but did not make an appropriate visual presentation of the data to management.

For Bayesian inference tasks involving processing conditional probabilities, most research has tangentially examined the use of VRs, usually to resolve other research problems. Over the past decade or so, researchers have engaged in a vigorous debate regarding the failure of subjects to properly solve Bayesian inference problems, with most of the debate concerning explanations for the apparent poor performance with textual presentations, including the role numeric format plays (e.g., natural frequencies versus probabilities, Gigerenzer & Hoffrage, 1995); the varying performance levels at boundary conditions of the problem: manipulations of reference class size, prevalence, sensitivity and specificity rates (Mellers and McGraw, 1999; Brase, 2002);  and explanations of performance based on theories of mental models (Johnson-Laird, et al., 1999) and the use of heuristics in judgment and decision making (Kahneman, Slovic & Tversky, 1988).

While researchers have examined the use of VRs in teaching Bayesian reasoning skills to students, very little prior research focused on the use of VRs in Bayesian inference for naïve subjects. The list of textual and visual representations discussed  or tested in the body of research includes: contingency tables (Cole,  1989), trees (Martignon & Wassner, 2002), frequency bars or “beam cut” diagrams (Gigerenzer & Hoffrage, 1995), Euler circles (Sloman, Over, Slovak, & Stibel, 2003), signal detection, probability curves (Cole, 1989), frequency grids (Cole, 1989; Gigerenzer & Hoffrage, 1995) and Bayesian boxes (Burns, 2004). No research to-date has examined the role individual differences in spatial abilities plays in subject performance with conditional probability tasks. 

One popular version of this problem, the “mammography problem,” asks subjects to infer a probability to a medical disease problem (Gigerenzer & Hoffrage, 1995). Prior research indicates subject accuracy is typically poor for the text-only probability version of the problem (Gigerenzer & Edwards, 2003; Gigerenzer & Hoffrage, 1995; Kurzenhauser & Hoffrage, 2002; Sedlmeier, 2000; Sloman, Over, Slovak, & Stibel, 2003) and in some cases accuracy can be improved with use of visuals.

The study of individual differences in cognitive abilities has a long history (Woodcock, 2002; Stanovich & West, 2000). Research into self reports of vividness of imagery and spatial abilities have conflicting results and explanations (Dean & Morris, 2003) regarding the relationship between imagery and spatial abilities. However, Kozhevnikov, Kossyln & Shephard (2005) and Kozhevnikov, Hegarty & Mayer (2002) provide some new insight clarifying prior conflicts. Subjects with low spatial abilities show an inclination towards constructing object or iconic VRs and have difficulty interpreting more abstract spatial VRs whereas subjects with high spatial abilities show an inclination towards constructing spatial or abstract VRs and a facility with interpreting more abstract spatial VRs. Imagery is not a unitary ability and may contain separate abilities reflecting the distinction between the “what” and the “where” pathways. The “what” pathway (the ventral stream) is crucial for the identification of objects and is sensitive to shape and color. The “where” pathway (the dorsal stream) is crucial for identifying spatial relations among objects and for visually guiding movements towards objects (Kastner & Ungerleider, 2000). The dorsal and ventral streams are used not only for perception – processing external representations, but for mental imagery – constructing internal representations in the absence of external representations.  

According to dual-coding theory (Paivio, 1990), VRs should facilitate mental model construction by building referential connections between verbal and nonverbal representations of the same problem. These connections aid in accessing or encoding in long term memory. Dual coding theory, however, does not specify what kind of VR helps best (Schnotz & Bannert, 2003). Findings from mental models (Knauff, et al., 2003) and cognitive neuroscience research (Kossyln, Ganis & Thompson, 2001) suggests a distinction between a visual working memory for object imagery and a spatial working memory for spatial relations and mental model construction, also reflecting the “what” and the “where” pathways (Schnotz, 2005). Conditional and relational reasoning tasks activate regions of the brain that make up the “where” pathway of spatial perception and working memory (Knauff & Johnson-Laird, 2002; Ruff, C., Knauff, M., Fangmeier, T. & Spreer, J. 2003). Because cognitive resources are finite, spatial activities can interfere with inference tasks in the “where” pathway when these tasks are performed concurrently (Schnotz, 2005).

CTML has a rich research track that explains subject learning (retention and transfer) via multimedia displays. However, no prior CTML research has focused on the conditional probabilities problem. Some prior CTML researched has examined the role of individual differences in spatial abilities on multimedia problems, but not in depth (Mayer, 2005). According to the individual differences principle of CTML, multimedia displays help those with low domain knowledge and those with high spatial abilities. Subjects with high domain knowledge can solve problems without the aid of the VR. Subjects with high spatial abilities can accommodate more visual information within working memory and thus can make use of the VR.  (Mayer, 2005). Conditional probability problems are ones in which nearly all subjects have little prior knowledge in solving.

What theoretical model prevails?

Interestingly, prior research into Bayesian inference and conditional probability problem solving has used both spatial and object VRs (hereafter referred to as iconic VRs) but without making a distinction between these two kinds of displays or measuring any linkage to differing spatial abilities. Spatial displays are more abstract representations of the relationships between sets of items without depicting individual items. Iconic displays use more concrete images to refer to specific items within a set, highlighting the use of shape and shading to make clear the distinction between kinds of items. Spatial displays are designed to take advantage of the “where” system and iconic displays are designed to take advantage of the “what” system. Figures 1 and 2 show the same conditional probability problem represented as a spatial and an iconic VR.

To help naïve subjects in solving this problem, what should interface designers do? The literature suggests different approaches.

First, as previously discussed, research into object and spatial visualizers suggests that the type of VR may matter; opposite their high-spatial counterparts, subjects with low spatial ability may not be able to effectively process spatial VRs (Kozhevnikov, et al., 2005). High spatial subjects should do well with spatial VRs and low spatial subjects may do better with less abstract iconic VRs. Mental models research suggests that a spatial VR showing the relationships between the concepts would help reasoning and representations that contain irrelevant imagery details (iconic VRs), should impede reasoning (Knauff & Johnson-Laird, 2002).

However, the type of conditional probability problems used in our study is difficult and takes subjects several minutes to solve, despite the fact that it only involves two numbers and two basic mathematical operations (addition and division)., e.g., A / (A+B). Since subjects struggle to find a relevant strategy from long term memory, perhaps the type of VR should aid in referential processing between the verbal (text) and the visual coding. Dual coding theory suggests that a display with more concrete concepts would elicit better referential processing between the two codings and long-term memory. Also, if visual working memory is composed of two distinct parts (the “what” and the “where” system) that are used to both solve the problem (build an internal representation or mental model) and perceive the problem (by scanning the external representation and maintaining its properties in either visual system), should the display be mindful of cognitive overload of the “where” system? This suggests a second approach. An iconic VR should facilitate mental model construction without interfering with that construction.

HYPOTHESES

Based on dual-coding theory and mental models research, while working with a spatial VR (an external representation) on a conditional probability problem, subjects must visually process explicit relations between concepts, thus activating the “where” pathway. Subjects need to maintain or rebuild this representation in visual working memory. While working with an iconic VR, subjects must process shape and texture more than spatial relations, thus activating the “what” pathway. Iconic VRs, being more concrete and less abstract, are more likely to facilitate referential processing which is needed for this difficult problem. With either spatial or iconic VRs, during mental model construction, subjects must alternate between constructing a mental model (an internal representation) of the problem and examining the VR (the external representation). Since a spatial VR must now compete for limited cognitive resources, in this case the “where” pathway, a spatial VR should be inferior to an iconic VR.  Therefore, we hypothesize the following effects of the visualization of conditional probability problems:

H1. VRs will generate more correct answers than textual representations

H2. Iconic VRs will generate more correct answers than spatial VRs

H3. VRs will result in a higher level of user satisfaction than textual representations.

H4. Iconic VRs will result in a higher level of satisfaction than spatial VRs

Based on the CTML research, subjects with high spatial ability, should perform better in all forms of representation in solving conditional probability problems (text, spatial or iconic VRs) and VRs (spatial or iconic) should benefit subjects with high spatial ability more so than subjects with lower spatial ability.  Therefore, we hypothesize that:

H5. Users with higher spatial abilities generate more correct answers than subjects with lower spatial abilities under all three forms of representation (text, spatial or iconic VRs).

H6. VRs will have stronger effects for those users with high spatial abilities than users with low spatial ability.

Hypothesis H1, H2, H3 and H4 are supported by dual coding theory and mental models research.  Hypotheses H2 and H4 are predicated on the split between “what” and “where” processing in visual working memory and use of the “where” pathway during mental model construction. Hypothesis H5 and H6 are supported in CTML research; specifically the individual differences principle.

Experiment design

This research will use a between-subjects 2X3 factorial design to test the above hypotheses:

 

Representation

Spatial ability

Iconic

Spatial

Text

High

 

 

 

Low

 

 

 

 

There are two independent variables—spatial ability and representation.  Spatial ability will be measured by the card rotation and paper folding tests from Ekstrom, French & Hardon (1976).  High and low level of spatial ability will be determined by a median split.  Representation will be defined by three forms of presenting conditional probability problems – textual, iconic visual, and spatial visual.  The conditional probability problems use frequency formats and a ‘short’ menu (Gigerenzer & Hoffrage, 1995), which facilitate more normative answers than more complicated versions of these problems. In keeping with CTML design principles, the VRs are presented in incremental, integrated displays in which the user can control the pace of the problem description, which is presented on the same screen as the visual treatment.

Dependent variables are user performance and satisfaction.  Performance is measured by accuracy (the number of correct answers).  User satisfaction is measured by a 14-item questionnaire based to Davis’ (1989) technology acceptance model (TAM). In this model, perceived ease of use and perceived usefulness are two key determinants on user adoption of technology.

A total of 200 subjects will be recruited from a Midwest university, classified as subjects of high and low spatial ability and randomly assigned to each of the three treatments. Subjects will fill out a background questionnaire regarding their education level, prior skill in probabilities, age, gender and any visual impairment. Subjects will then complete a timed test of card rotation and paper folding (Ekstrom, et al., 1976). Afterwards, they will receive a brief explanation about the task and also complete a battery of six conditional probability problems and a post-test survey designed to measure satisfaction.

We will use a “write-aloud” process tracing protocol (Gigerenzer & Hoffrage, 1995), in which the subjects’ recorded notes, diagrams and calculations will be analyzed. Consistent with Gigerenzer and Hoffrage (1995), scoring of the correct answers follows a strict rounding criteria which allows for answers to be rounded up or down to the nearest integer. The combination of the rounding criteria and the process tracing data is used to score a correct answer. This eliminates math errors from disqualifying normatively correct processes.

Implications

The implications for this research are important. Information and decision support systems may need a broader repertoire of visualization techniques to be able to improve performance in naïve subjects that takes into consideration the task demands for these kinds of problems. This research may suggest that VRs may facilitate other difficult judgment and decision problems, provided designers are mindful of the cognitive limitations within the “what” and the “where” pathways of visual processing.  Although CTML is a theory that explains subject learning (retention and transfer) with multimedia displays, because of its inclusion of theories of memory and cognition, it may be robust enough to extend to non-learning environments and contribute to the rich and conflicting research into judgment and decision making (JDM). Finally, we hope this research can clarify how subjects perform with VRs in these tasks. User interface designers may find that for difficult tasks like these, representing a problem visually in spatial external representation may actually interfere with the mental model construction in working memory and impede reasoning.

 References

Brase, G. L. (2002). Which statistical formats facilitate what decisions? The perception and influence of different statistical information formats. Journal of Behavioral Decision Making. 15, 381-401.

Burns, K. (2004). Painting pictures to augment advice. AVI ’04, May 25-28. ACM. 344-349.

Card, S. K., Mackinlay, J. D. & Shneiderman, B. (1999). Information Visualization: Using Vision to Think. Academic Press

Cole, William G. Understanding Bayesian reasoning via graphical displays. CHI ’89 Proceedings, May 1989. ACM. 381-386.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use and user acceptance of information technology. MIS Quarterly. Sep. 319-340.

Dull, R. B., & Tegarden, D. P. (1999). A comparison of three visual representations of complex multidimensional accounting information. Journal of Information Systems. 13, 117-131

Dean, G. M. & Morris, P. E. (2003). The relationship between self-reports of imagery and spatial ability. British Journal of Psychology. 94, 245-73.

Ekstrom, R. B., French, J. W., & Harman, H. H. (1976). Manual for kit of factor-referenced cognitive tests. Princeton: Educational Testing Service.

Gigerenzer, G. & Edwards, A. (2003). Simple tools for understanding risks: from innumeracy to insight. BMJ. 327, 741-744.

Gigerenzer, G. & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: frequency formats. Psychological Review. 102, 684-701.

Girotto, V. & Gonzalez, M. (2001). Solving probabilistic and statistical problems: a matter of information structure and question form. Cognition. 78, 247-276.

Johnson-Laird, P.N., Legrenzi, P., Girotto, V., Legrenzi, M.S., & Caverni, J. (1999). Naïve probability: a mental model theory of extensional reasoning. Psychological Review. 106(1), 62-88.

Kahneman, Daniel, Slovic, Paul & Tversky, Amos. (1988). Judgement Under Uncertainty: Heuristics and Biases. Cambridge University Press.

Kastner, S. & Ungerleider, L. G. (2000). Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience. 23, 315-341.

Kossyln, S. M., Ganis, G., & Thompson, W. L. (2001). Neural foundations of imagery.  Nature Review, Neuroscience.  2(9), 635-42.

Knauff, M., Fangmeier, T., Ruff, C., & Johnson-Laird, P.N. (2003). Reasoning, models, images: behavioral measures and cortical activity. Journal of cognitive neuroscience. 15(4). 559-573.

Knauff, M. & Johnson-Laird, P.N. (2002). Visual imagery can impede reasoning. Memory & Cognition.  30(3), 363-371.

Kozhevnikov, M., Hegarty, M. & Mayer, R. E. (2002). Revising the visualizer-verbalizer dimension: evidence for two types of visualizers. Cognition and Instruction. 20(1), 47-77.

Kozhevnikov, M., Kossyln, S. & Shephard, J. (2005). Spatial versus object visualizers: a new characterization of visual cognitive style. Memory and Cognition. 33(4), 710-726.

Kurzenhauser, S. & Hoffrage, U. (2002). Teaching Bayesian reasoning: An evaluation of a classroom tutorial for medical students. Medical Teacher. 24, 516-521.

Martignon, L. & Wassner, C. (2002). Teaching decision making and statistical thinking with natural frequencies. ICOTS6.

Mirel, B. (1998). Visualization for data exploration and analysis: a critical review of usability research. Technical Communication. Fourth Quarter, 1998. 491-509.

Mayer, R. E. (2002). Multimedia Learning. The Psychology of Learning and Motivation. 41, 85-139.

Plaisant, C. (2004). The challenge of information visualization evaluation. AVI ’04, ACM. May 25-28.

Paivio, A. (1990). Mental Representation: A Dual Coding Approach.  Oxford University Press. New York.

Ruff, C., Knauff, M., Fangmeier, T. & Spreer, J. (2003). Reasoning and working memory: common and distinct neuronal processes. Neuropsychologia. 41, 1241-1253.

Schnotz, W. & Bannert, M. (2003). Construction and interference in learning from multiple representation. Learning and Instruction. 13. 141-156.

Schnotz, W. (2005). An integrated model of text and picture comprehension. In R. E. Mayer (Eds), The Cambridge Handbook of Multimedia Learning. Cambridge University Press.

Sedlmeier, P. (2000). How to improve statistical thinking: Choose the task wisely and learn by doing. Instructional Science. 28, 227-262.

Sloman, S. A., Over, D., Slovak, L., & Stibel, J. M. (2003). Frequency illusions and other fallacies. Organizational Behavior and Human Decision Processes. 91, 296-301.

Spence, R.. (2001).  Information Visualization. Addison-Wesley.

Stanovich, K. E. & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences. 23, 645-726.

Tegarden, D. P. (1999). Business information visualization. Communications of the AIS. Vol. 1, Article 4 (January).

Tufte, E. R. (1997). Visual Explanations: Images and Quantities, Evidence and Narrative. Graphics Press.

Woodcock, R.W. (2002). New looks in the assessment of cognitive ability. Peabody Journal of Education. 77(2), 6-22.


Figure 1. A spatial VR

Figure 2. An iconic VR