Baker - Closing the Loop

From LearnLab
Revision as of 17:06, 5 December 2009 by Ryan (Talk | contribs) (Background & Significance: added prior predictors)

Jump to: navigation, search

Using educational data mining to design tutor lessons that students don’t choose to game: “Closing the loop”

Summary Table

PIs Ryan Baker
Other Contributers
Study Start Date Spring, 2010
Study End Date
LearnLab Site TBD
LearnLab Course Algebra
Number of Students TBD
Total Participant Hours TBD
DataShop TBD


This 12 month CMDM project proposes to “close the loop” on a data mining analysis previously conducted within the PSLC (Baker_Choices_in_LE_Space), showing that the previous analysis makes a contribution to improving student learning in in-vivo settings. In that previous study, a model of the differences between different tutor lessons (the Cognitive Tutor Lesson Variation Space, or the CTLVS1 -- full details on this model are given on the page Baker_Choices_in_LE_Space) was created, and used to study why some tutor lessons are gamed more than others in the Algebra tutor. The best model based on the CTLVS1 (developed via a combination of PCA and correlation mining) predicted over half of the variance in gaming, almost 6 times better than any previous model attempting to explain gaming through specific student individual differences.

In this study, we will choose a lesson from the Algebra tutor that is highly gamed, and modify it in accordance with the findings of that previous work, such that the modified lesson is predicted to lead to significantly less gaming.

Background & Significance

In recent years, there has been considerable interest in how students choose to interact with learning environments. At any given learning event, a student may choose from a variety of learning-oriented "deep" paths, including attempting to construct knowledge to solve a problem on one’s own (Brown and vanLehn, 1980), self-explaining (Chi et al, 1989; Siegler, 2002), and seeking help and thinking about it carefully (Aleven et al, 2003). Alternatively, the student may choose from a variety of non-learning oriented "shallow" strategies, such as Help Abuse (Aleven & Koedinger, 2001), and Systematic Guessing (Baker et al, 2004). This pair of strategies is referred to as Gaming the system (Baker et al, 2004). Gaming the system is an active and strategic type of shallow strategy known to occur in many types of learning environments (cf. Baker et al, 2004; Cheng and Vassileva, 2005; Rodrigo et al, 2007), including the Cognitive Tutors used in LearnLab courses (Baker et al, 2004).

Recent work has indicated that a variety of aspects of cognitive tutor lessons are predictive of greater quantities of gaming (Baker et al, 2009). In particular, gaming is predicted by (boldface indicates a particularly strong relationship)

  • The same number being used for multiple constructs
  • Reading hints does not positively influence performance on future opportunities to use skill
  • Proportion of hints in each hint sequence that refer to abstract principles
  • Not immediately apparent what icons in toolbar mean
  • Lack of text in problem statements not directly related to the problem-solving task, generally there to increase interest
  • Hints do not give directional feedback such as “try a larger number”
  • Hint requests that student perform some action
  • The location of the first problem step does not follow conventions (such as being the top-left cell of a worksheet) and is not directly indicated
  • The lesson is not an equation-solver unit


Computational Modeling and Data Mining

Gaming the system



Independent Variables

Dependent Variables

Planned Experiments


Further Information


Annotated Bibliography


Future Plans