REAP Study on Personalization of Readings by Topic (Fall 2006)
REAP Study on Personalization of Readings for Increased Interest
Intrinsic motivation and personal interest have been shown to lead to deeper processing, the use of learning strategies, and better learning outcomes (Lepper, 1988). Therefore, it seems reasonable to try to increase interest in a tutoring environment. Previous studies have shown both positive (Cordova & Lepper, 1997) and negative (Clark & Mayer, 2003) effects of attempts to increase interest. Clark and Mayer (2003) state that adding interest but irrelevant material may distract or disrupt the learner.
This study investigated the effects of personalizing practice readings by topic. The REAP tutor for ESL vocabulary can prefer practice readings that match personal interests as specified in a questionnaire. However, if this preference is included as a factor in reading selection, then other factors, such as the density of practice opportunities in a reading, are necessarily given less weight. Finding a reading for a student is like searching for an optimal point in a multidimensional space. In practice, the tutor never finds optimal readings, but must weigh different factors against each other (e.g., reading difficulty, length, density of practice opportunities, etc.). Adding personalization as another factor may significantly affect the tutor's ability to find readings of high quality according to other factors.
This study investigated the use of personalization of readings by topic as an attempt to reduce gaming and increase the likelihood that students would more deeply process the context around target words, rather than just access definitions for them. As such, this study primarily addresses the category of passive, implicit instruction in the following table of types of instruction and learning.
|Implicit (instance)||Interpreting meaning in context while reading|
Intrinsic Motivation: Motivation to learn for learning's own sake rather than some external goal.
Extrinsic Motivation: Motivation for learn in order to satisfy an external goal, such as completing a task or passing an assessment.
Personalization: The preference of practice materials to match the personal interests of the learner. In this context, this term is used as it is in information retrieval rather than as in Mayer's work (Clark & Mayer, 2003) where it means using casual, direct language.
Do the benefits of personalization of practice readings by topics of interest outweigh the costs in a tutoring system for ESL vocabulary practice?
Immediate post-test scores for practiced words.
Number of words practiced
Overall post-test scores (essentially a product of the previous two)
Long-term retention test scores similar to post-test but administered months later.
Transfer of knowledge: sentence production tasks for target words, correct use of words in writing assignments for other courses.
Personalization of readings by topics of interest. In the control condition, the tutor did not use potential personal interest as a factor in its selection of reading materials. In the treatment condition, the tutor did use interest as a factor. All other selection criteria were the same in both conditions. Time on task was also the same.
Since intrinsic motivation seems to be important in language learning, the benefits of personalization will outweigh the costs.
Students in the treatment condition with personalization performed better on average (M=35.5%, SD=14.9%) in terms of overall post-test scores compared to students in the control condition (M=27.1%, SD=17.2%). However, the improvement of average overall post-test scores in the treatment condition was only 8.4% (95% CI = -2.8%, 19.5%), which corresponds to a medium effect size of 0.51. This difference was not statistically significant (p=0.14). Therefore, the null hypothesis that personalization has no effect on overall post-test scores cannot be rejected.
Although personalization did have positive effects on the learning of words that were practiced with the REAP tutor, the personalization forced the tutor to choose readings that had fewer target words. The student’s progress through the curriculum was thus impeded by incorporating personalization into the algorithm for selecting tasks to give students. Overall scores suggest that personalization still improved overall learning as measured by post-test scores for all words tested. However, the overall effect was smaller and not statistically reliable. In the REAP tutor, attempts to increase interest required meaningful changes to the tasks given to students. For example, if a student was interested in science, then the tutor had to assign higher values to passages about science in its search for reading material containing the target vocabulary. This change necessarily meant that sometimes suboptimal tasks with respect to curriculum progress had to be given to students. Additionally, in choosing readings for specific topics, the tutor was selecting readings with different sets of non-target words, grammatical features, discourse, etc. Thus, learning opportunities other than the target vocabulary words were affected because deep changes to the task were required for personalization. By presenting the student with readings of interest, the REAP tutor was potentially introducing extraneous material, which would go against Clark and Mayer’s Coherence Principle (Clark & Mayer, 2003, pp. 111-129). It is plausible that students would be more interested in the content of the readings than the target vocabulary words in those readings, and the content would distract and disrupt their learning of vocabulary. In the case of acquiring vocabulary, however, the content of readings was likely not extraneous but important because it provided valuable contextual clues to word meanings. The goal of providing job-relevant practice to increase transfer and links with prior knowledge (Clark & Mayer, 2003, pp. 156-157) may also play an important role. The ultimate “job” or goal of the students practicing vocabulary with REAP is to read texts that they are interested in (e.g., material for future courses, employment, or personal enjoyment). Thus, by motivating students to more closely read texts and by giving them texts related to their personal interests, the REAP tutor is giving students job-relevant practice. Increasing motivation is a challenging task for the developers of tutoring systems. In the experiment described in this paper, the benefits of personalization appeared to outweigh the costs, but such may not always be the case. Previous research has indicated that intrinsic motivation has positive effects on learning. Therefore, personalization and other possible ways of increasing intrinsic motivation should be considered when designing a tutoring system or curriculum for students. However, attempts to increase interest by adding interesting but extraneous material may distract and disrupt students. Therefore, changes to increase motivation should be relevant to the immediate task as well as the ultimate goals of the individual student. However, even when material is altered in a relevant way, incorporating personalization and motivation as a factor for selecting tasks may negatively affect other curriculum goals. In this study, task-relevant personalization improved learning for target words student saw in reading passages, but it also decreased the number of target vocabulary words per reading. Overall learning results were improved slightly but not significantly. In short, ways of increasing the intrinsic motivation of students should be considered, but also weighed against their potential negative effects.
NOTE: Long-term retention test results are pending.
Submitted paper to appear here.
Clark, R. C. and Mayer, R. E. (2003). e-Learning and the Science of Instruction. Jossey-Bass/Pfeiffer.
Cordova, D. I. & Lepper, M. R. (1996). Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology. Vol. 88,l No. 4, 715-730.
Lepper, M. (1988). Motivational Considerations in the Study of Instruction. Cognition and Instruction. 5(4), 289-309.
Heilman, M., Juffs, A., & Eskenazi, M. (To Appear). Choosing Reading Passages for Vocabulary Learning by Topic to Increase Intrinsic Motivation. Proceedings of the 13th International Conferenced on Artificial Intelligence in Education. Marina del Rey, CA. (poster)