Roll IPL

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Invention as preparation for learning

Ido Roll, Vincent Aleven, Bruce M. McLaren, Kenneth Koedinger

Can invention activities prepare students to better learn from subsequent instruction, compared with instruction-and-practice only?

Meta-data

PI's: Ido Roll, Vincent Aleven, Dan Schwartz, Ken Koedinger

Other Contributers: David Klahr

Study # Start Date End Date LearnLab Site # of Students Total Participant Hours DataShop?
1 4/2007 4/2007 North Hills 20 40 No, paper-and-pencil only
2 9/2007 12/2007 Community Day 4 48 No, paper-and-pencil only
3 4/2008 5/2008 Steel Valley 125 900 No, paper-and-pencil only
4 4/2009 5/2009 Steel Valley 140 560 Some of it in DataShop, the rest is getting there

Abstract

The assistance dilemma asks what form of assistance is most appropriate for different stages of learning. While direct instruction and practice have been shown to be efficient for novices, students often acquire shallow knowledge components and lack robust understanding. Some evidence suggests that invention using contrasting cases, prior to instruction and practice, can accelerate future learning, compared with instruction and practice alone (Schwartz & Martin, 2004). The invention process as described in Schwartz & Martin (2004) includes the following stages: Design (of a mathematical model to solve a class of problems); calculation (of the solution based on the model); evaluation (of its correctness); and debugging (of the faulty model). Notably, most students fail to invent mathematically valid models, so the goal is not for students to discover the correct solution. At the same time, students do make models that capture deep features of the class of problems, which prepares them to learn and understand the significance of expert solutions for handling such situations. Following the invention, students receive instruction on the expert solution (that is, formulas) and practice it. This procedure is based on the hypothesis that students’ own inventions, together with subsequent instruction, are sources for coordinative learning. By attempting to create a model that correctly distinguishes the “contrasting cases” (carefully selected instances within a class of problems) students notice (and to some degree invent) the problem features that an adequate model must take into account, and they attend to them during subsequent instruction. However, alternative explanations for the effectiveness of the IPL process are possible, with different instructional implications. A “debugging hypothesis” suggests that evaluation and debugging of pre-designed models are sufficient to promote future learning by directing students’ attention to the short-comings of the designed models, and thus to the deep features of the domain. Alternatively, an “unfinished goals” hypothesis suggests that the effect is caused by students reaching impasses during invention. According to this hypothesis, calculation-evaluation are sufficient for preparing for future learning. We propose to investigate this in a series of ablation studies with the goal of better defining the invention process and identifying the cognitive processes involved. This includes a combination of in-vivo and lab studies within the Algebra LearnLab, contributing to the Coordinative Learning theoretical framework. Following the ablation studies we plan to implement the procedure in a Cognitive Tutor, which will be evaluated in a lab study. This will allow us to better operationalize the process, do a micro-genetic analysis of it, and identify productive patterns of learning trajectories using log mining.

Background and Significance

One of the main challenges of education is to help students reach meaningful and robust learning. The assistance dilemma raises the question of what form (and ‘amount’) of assistance are most effective with different learners in different stages of the learning process (Koedinger & Aleven, 2007). Instruction followed by practice is known to be very efficient for teaching novices (e.g., Koedinger, Anderson, Hadley & Mark, 1997); yet, students often acquire shallow procedural skills, and fail to acquire conceptual understanding (Aleven & Koedinger, 2002). This can be attributed, at least in part, to students using superficial features and not encoding the deep features of the domain (Chi, Feltovich & Glaser, 1981). One approach to getting students to attend and encode the deep features is to add an invention phase prior to instruction. Invention as preparation for leaning (IPL) was shown to help students better cope with novel situations that require learning (Schwartz & Martin, 2004; Sears, 2006). In this process students are presented with a dilemma in the form of contrasting cases, and attempt to invent a mathematical model to resolve this dilemma. For example, Figure 1 shows four possible pitching machines. Students are asked to invent a method that will allow them to pick the most reliable machine. The concept of contrasting cases comes from the perceptual learning literature, since these cases, when appropriately designed, emphasize differences in the deep structure of the examples (Gibson & Gibson, 1955). The invention process includes designing a model, applying it to the given set of contrasting cases, evaluating the result, and debugging the model. This iterative process is very similar to the debugging process as described by Klahr and Carver (1988; Figure 2). Unlike other inquiry-based manipulations (cf. Lehrer et al., 2001; de Jong & van Joolingen, 1998), the goal of the IPL process is not for students to discover the correct model, but to prepare them for subsequent instruction. During the instruction students share their models, critic their peers’ models, and learn the expert solutions (A similar classroom critic process was shown to be effective by White & Frederiksen, 1998). Preparation for learning from the instruction is evaluated using accelerated future learning assessment. The accelerated future learning assessment includes an embedded instruction in the test in the form of solved example. Schwartz (2004) found that only students who invented prior to the test were able to take advantage of that instruction in order to solve novel problem, while students who practiced a given visual method prior to the test did not take advantage of the embedded learning resource and thus could not solve the target problem. This shows that the IPL process has a positive effect on students’ ability to independently learn from the solved examples. In the case of the contrasting cases given in Figure 1, subsequent instruction will introduce the students with the notion (and formulas) of variance. While the invention group was superior to instruction-and-practice group on accelerated future learning measure, there was no direct comparison of normal or transfer measures between invention and instruction-and-practice conditions (though invention students showed pre-to-post gains, and were shown to outperformed college students). Also, it is not yet clear how robust this pedagogy is and what its key features are.

Example for contrasting cases (topic - variability)

The overall IPL process:

See IPL Glossary

Research questions

1. What is the overall effect of Invention tasks on students domain knowledge, sense-making skills, and motivation, compared with direct instruction?
2. What elements of invention contribute to that effect? What cognitive processes do they drive? In what ways does knowledge acquired following invention differ from knowledge acquired in direct instruction alone?
3. Can the IPL process be scaled-up using technology?

In addition, the project makes the following contributions:

1. It compares different measures of robust learning, in order to understand what aspect of knowledge can be assessed using what type of measure.

Independent Variables

Different studies manipulated different stages of the Invention task: - Observation (a.k.a. comparative reasoning): comparing contrasting cases that vary along deep features, with regard to target concepts - Generative reasoning: designing novel mathematical procedures to compare the contrasting cases with regard to the target concept - Evaluation: Evaluation of the models

Dependent variables

Domain knowledge (in increasing 'distance' from instruction):

• Normal measures
• Transfer measures
• New strategy items (with learning resource)
• New strategy items (without learning resource)

Motivation and affect:

• Behavioral measure: % of students who kept working during breaks
• Self reports

Findings

Domain knowledge:

• IPL students in advanced classes were more capable of solving new strategy items without learning resource. In fact, in the absence of a learning resource, direct instruction students performed at floor, while IPL students performed as well as with the source.
• This effect holds when controlling for simple domain knowledge (performance on normal items in the same test).
• This was found in multiple new-strategy items. However, all results were found in a single topic (central tendency and graphing). The single test item on the topic of variability failed to capture differences between conditions.

Motivation:

• IPL students reported to have benefited more (marginally significant: F=3.3, p<.07)
• There was a significant interaction between condition and test anxiety. Text anxiety was assessed using the MSLQ (Pintrich 1999) before the study had begun. Students who reported to have higher test anxiety also reported to have benefited more from IPL instruction compared to high-anxiety students in the no design condition.
• IPL students stayed more often in class to work during breaks (IPL: 16% No Design: 3%).
• Furthermore, they did so during invention activities and not show-and-practice activities, suggesting that it is the activities that are motivating (IPL activities: 25%; show-and-practice activities: 7%)

Connections

• Tim Nokes's study

The Invention Lab

In addition to paper and pencil studies, we have created the Invention Lab.

The Invention Lab is an intelligent tutoring system for IPL. To give intelligent feedback, it uses two models:

• A meta-cognitive model of the invention process
• A cognitive model of the main concepts in the domain