Mapping Visual and Verbal Information: Integrated Hints in Geometry (Aleven & Butcher)

From LearnLab
Jump to: navigation, search

Using Elaborated Explanations to Support Geometry Learning

Vincent Aleven and Kirsten Butcher 

Summary Table

PIs Vincent Aleven & Kirsten R. Butcher
Other Contributers Graduate Students: Carl Angioli (CMU HCII), Michael Nugent (Pitt, Computer Science)

Research Programmers/Associates: Octav Popescu (Research Programmer, CMU HCII), Grace Lee Leonard (Research Associate, CMU HCII), Thomas Bolster (Research Associate, CMU HCII)

Study Start Date Planned Start: April 24, 2007
Study End Date Expected End: June 1, 2007
LearnLab Site Central Westmoreland Career & Technology Center (CWCTC)
LearnLab Course Geometry
Number of Students Approx. 120
Total Participant Hours Approx. 480
Data available in DataShop

Dataset:Contiguity CWCTC Spring 2007

  • Pre/Post Test Score Data: No
  • Paper or Online Tests: Paper
  • Scanned Paper Tests: Yes
  • Blank Tests: No
  • Answer Keys: No


Does integration of visual and verbal knowledge during learning support deep understanding? Can robust learning be supported implicitly by representations that link relevant knowledge components in visual and verbal materials? The overall goal of this project is to gain a better understanding of 1) visual and verbal knowledge components in a problem-solving environment and, 2) how instructional support to promote connections between visual and verbal knowledge components can support the development of deep understanding. Ultimately, we are interested in coordination and integration processes in learning with visual and verbal knowledge components, and how these processes may support robust learning.

We are using the Geometry Cognitive Tutor as a research vehicle for our project. In geometry, visual information is represented pictorially in a problem diagram and verbal/symbolic information is represented in text that contains given and goal information as well as in conceptual rules/principles of geometry. The goal of the research described here is to determine if implicit instructional events that use visual cues to map between text and diagrams can support knowledge retention and transfer. These visual cues are instantiated in the Geometry Cognitive Tutor by using colored highlighting to connect textual references of geometric features in instructional hints to the visual depictions of those features in the geometry diagram (screen shots provided below, in the Independent Variables section). This in vivo study will take place April - June, 2007.

Background & Significance

A central question in theories of learning with multimedia sources, and for the Coordinative Learning research cluster, is how students coordinate between multiple representations. Existing theories of multimedia learning (e.g., Mayer, 2001; Schnotz, 2002) assume that successful learning is supported by cognitive processes that operate between separately encoded visual and verbal representations. Laboratory research in learning with scientific diagrams has shown that simplified diagrams support [[retention] and mental model development more than detailed diagrams, and that simplified representations better support integration of textual information (Butcher, 2006). Indeed, other research has found that a multimedia interface that required learners to integrate text labels in a diagram improved knowledge retention and transfer for complex information (Bodemer, Ploetzner, Feuerlein, & Spada, 2004).

Taken together, these studies raise an important question regarding coordinative learning: Is coordination best supported by supporting successful mapping between text and visual information or by the construction of an integrated visual-verbal representation? The current study addresses this question using a 2 (Integrated Hint vs. Standard Hints) X 2 (Contiguous Representation vs. Noncontiguous Representation).

In the current study, successful mapping is supported with Integrated Hints. Integrated Hints use color-coded highlighting to link diagrammatic features to corresponding textual instructional explanations. In geometry, learners must map between relevant text and pictorial representations (e.g., to connect the text "angle ABC" to the visual depiction of this angle in a diagram) as they learn to recognize and apply relevant geometric principles/rules. Thus, one might expect mapping cues to support robust learning by helping students to form an integrated representation of critical knowledge components.

The current study supports construction of an integrated visual-verbal representation with a contiguous problem-solving interface (see Contiguous Representations for Robust Learning (Aleven & Butcher)). One might expect the contiguous representation to support robust learning in geometry because it eases the cognitive load required to maintain and use the location and value of solved quantities to reason about related features.


See Visual-Verbal Learning Project Glossary

Research questions

  1. Does mapping support in the form of Integrated Hints support students' retention and transfer of knowledge components?
  2. Do contiguous representations in geometry support students' retention and transfer of knowledge components?
  3. Are the effects of Integrated Hints or contiguous representations stronger for transfer than for retention?
  4. Do Integrated Hints and contiguous representations interact in their support for robust learning, as measured by performance on transfer items?

Independent Variables

Hint Integration: Standard vs. Integrated Hints

  • Standard Hints include plain text in a pop-up window, without highlighting of the hint text or the associated diagram features.
  • Integrated Hints (below) include the same text as the standard hints, but with color-coded highlighting that provides a visual link between relevant text and diagram features.

Diagram IntHints.jpg

Contiguity: Noncontiguous vs. Contiguous Representation

  • The noncontiguous representation supports student interaction with a table that is separate from the diagram. Accepted answers and feedback are displayed in the table, and not in the diagram.
  • The contiguous representation supports student interaction with the diagram itself. Feedback is provided near the relevant diagram feature and accepted answers are displayed in the correct location in the diagram. For more information and screen shots, see Contiguous Representations for Robust Learning (Aleven & Butcher).

Dependent variables

  • Log data collected during tutor use, used to assess:


We expect both the Integrated Hints and the contiguous representations to support robust learning, as measured by transfer items. Further, we expect best performance on transfer items when students receive both types of support, because optimal coordination between text and diagrams should require both mapping and Visual-verbal Integration.

Findings (Preliminary)

  • Problem Solving Answers

ProblemSolvingAnswers Contig4 Prelim.jpg

  • Application of Geometry Rules to Diagram

TrueFalseAnswers Contig4 Prelim.jpg

Preliminary results show strong benefits for diagram interaction, but not when integrated hints are provided. It should be noted that these results are based on overall analysis of experimental conditions. However, there is no guarantee that students in all conditions made equivalent use of the hints-on-demand. Thus, results need to be analyzed when hint use (based on log file analysis) is taken into account.


From a Coordinative Learning Cluster perspective, coordination between visual and verbal information supports foundational skill building, because attending to both representations simultaneously associates features from both with the learned knowledge components. This association increases feature validity and promotes robust learning.

When students are already receiving coordination support in the form of diagram interaction, additional assistance in the form of integrated hints may not be optimal. Although more analyses are necessary to fully explore the effects of integrated hints on student learning, these results may highlight the need to resolve the assistance dilemma. Providing more coordinative support during learning may not always be optimal for robust learning.

Further Information


Interactive Communication as Support for Visual-Verbal Integration:
Our research is investigating multiple methods with which student learning can be supported by interactions with pictorial information during geometry learning (also see our work on contiguous representations in geometry: Contiguous Representations for Robust Learning (Aleven & Butcher). However, our work also includes more a more explicit method for supporting student integration visual and verbal knowledge components. This method involves interactive support for students' elaborated explanations during geometry learning. Research investigating this explicit support is part of the Interactive Communication Cluster: Using Elaborated Explanations to Support Geometry Learning (Aleven & Butcher)

Efforts to Improve Self-Supervised Learning Using Instructional Hints: Our Integrated Hints can be considered a specialized method to improve the instructional quality of hints (based on a Coordinative Learning perspective). Thus, our work is relevant to other PSLC efforts to improve the quality and usefulness of hints during intelligent tutoring (e.g., Help Lite (Aleven, Roll)).

Visual Representations for Robust Learning in Other Domains: Our efforts to support students' integration of visual and verbal knowledge are informed by and related to efforts investigating the use of visual representations to support robust learning in other domains. A closely related PSLC project is Visual Representations in Science Learning, in which researcher are exploring whether coordination between verbal and visual representations can help students refine initially shallow understandings into meaningful chemical concepts.

Annotated Bibliography


  • Bodemer, D., Ploetzner, R., Feuerlein, I., & Spada, H. (2004). The active integration of information during learning with dynamic and interactive visualisations. Learning and Instruction, 14, 325-341.
  • Butcher, K. R. (2006). Learning From Text With Diagrams: Promoting Mental Model Development and Inference Generation. Journal of Educational Psychology, 98(1), 182-197.
  • Mayer, R. E. (2001). Multimedia Learning. Cambridge: Cambridge University Press.
  • Schnotz, W. (2002). Commentary: Towards an integrated view of learning from text and visual displays. Educational Psychology Review, 14(1), 101-120.

Future Plans: June 2007 - December 2007

  1. (Carnegie Learning): Gather log data and assessment data from classrooms.
  2. (Carnegie Learning): Anonymize log data and assessments, then provide to DataShop
  3. Score student performance on assessments
  4. Analyze log data and learning outcomes
  5. Prepare manuscript
  6. Integrate results into final project report for PSLC