Difference between revisions of "Extending Reflective Dialogue Support (Katz & Connelly)"
|Line 70:||Line 70:|
=== Findings ===
=== Findings ===
Preliminary analyses marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of general "gaming the system" by some students, more detailed analyses the identification and omission of noisy data from our overall corpus
=== Further Information ===
=== Further Information ===
Revision as of 18:23, 22 December 2008
- 1 Extending Automated Dialogue Support for Robust Learning of Physics
Extending Automated Dialogue Support for Robust Learning of Physics
|PIs||Sandra Katz & John Connelly|
|Study Start Date||10/1/07|
|Study End Date||9/30/08|
|LearnLab Course||General Physics I|
|Number of Students||N = 75|
|Total Participant Hours||approx. 125 hrs|
Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to integrate quantitative and qualitative knowledge via post-practice reflection dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study (Katz, Connelly, & Treacy) by trying to better support robust learning via a third condition in which students work through longer dialogues designed to foster transfer by specifically tying together qualitative and quantitative knowledge in different contexts.
Software errors and students' circumvention of both the system and instructors’ scoring rubrics rendered our intended detailed analyses impossible. However, despite the various manifestations of “cheating” and otherwise gaming the system that were discovered, we found that dialogue exposure significantly boosted gains on a post-test and on a largely quantitative final exam two months after the end of our intervention, providing evidence of some robust learning and transfer to related problem solving contexts.
Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these knowledge components in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster transfer and robust learning?
Background and Significance
Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to integrate quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, integrative reflective dialogues that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.
To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the Andes physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufresne, & Mestre (1996): knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study (Katz, Connelly, & Treacy), this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more robust learning measures of transfer (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (what if scenarios), designed to support both qualitative and quantitative knowledge (most of our previous what if scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context and to new contexts, to help support near and far transfer; and (3) students were prompted to state the rules (knowledge components) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.
- Gains in qualitative and quantitative knowledge. Post-test score, and pre-test to post-test gain scores, on near and far transfer items.
- Long-term retention. Performance on final exam, taken several weeks after the intervention.
We had also intended to measure short-term retention via student performance on course exams that covered target topics (statics; translational dynamics, including circular motion; work-energy; power; and linear momentum, including impulse). However, critical omissions in the data provided to us by course instructors rendered such measurements impossible.
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of Knowledge Construction Dialogues (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study (Katz, Connelly, & Treacy), in which mostly conceptual dialogues followed problem solving on selected Andes problems.
The new long-KCD condition appended more quantitative practice and transfer content (via additional what if scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.
The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.
The following student variables were entered into a regression analysis, with post-test score as the dependent variable:
- Number of viable target problems completed (before the post-test, or before the final exam)
- Number of viable (non-gibberish) dialogues completed
- Grade point average (CQPR)
- Pre-test score
The main hypotheses tested were (a) that explicit, part-task training on the “what? how? and why?” components of planning, via strategically staged reflective dialogues, would be more effective and efficient than the traditional approach of letting students acquire these components implicitly, through lots of practice with solving problems (replicating last year's study); and (b) that extended dialogues with additional quantitative practice in applying these knowledge components in different problem-solving contexts would better foster transfer and robust learning.
Preliminary analyses showed marginal support for a replication of last year's finding that student performance on the post-test relative to the pre-test was significantly influenced by the number of dialogues they completed, as opposed to the number of target problems they completed. However, due to software glitches during data collection, low student participation (completing assigned homework and/or dialogues) in some course sections, and evidence of cheating and general "gaming the system" by some students, more detailed analyses were made possible only by the identification and omission of noisy data from our overall corpus. This lengthy and painstaking process required us to revisit KCD and Andes logs in a much lower level of detail, in an attempt to determine whether each students' list of assigned problems and dialogues were viable (i.e., legitimately completed).
In the end, the various problems plaguing our data rendered impossible any comparisons between our two treatment conditions. However, we were able to salvage enough data to compare student performance relative to degrees of viable problem and dialogue completion. The salvaged exposure measures were the number of viable KCDs completed (vs. KCDs that were "passed through" with gibberish responses) and the number of viable problems completed (vs. those on which students likely "cheated", as determined by answer-only "solutions" or minimal Andes inputs to attain a score of 50, the minimal criterion for credit used by one course instructor).
- Presentation to Advisory Board, January 2008
- Katz, S., Connelly, J., & Wilson, C. (2007). Out of the lab and into the classroom: An evaluation of reflective dialogue in Andes. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.), Artificial Intelligence in Education: Building Technology Rich Learning Contexts that Work (pp. 425-432). Amsterdam: IOS Press.
- Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. American Journal of Physics, 64 (12), 1495-1503.
- VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3).
- VanLehn, K., Lynch, C., Schmulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., & Wintersgill, M. (2005). The Andes physics tutoring system: Five years of evaluation. In G. McCalla, C.K. Looi, B. Bredeweg, & J. Breuker (Eds.), Artificial Intelligence in Education (pp. 678-685). Amsterdam, Netherlands: IOS Press.
This project shares features with the following research projects:
Use of Questions during learning
- Post-practice reflection (Katz & Connelly, 2005)
- Reflective Dialogues (Katz, Connelly & Treacy, 2006)
- Deep-level questions during example studying (Craig & Chi)
- FrenchCulture (Amy Ogan, Christopher Jones, Vincent Aleven)
- Collaborative Extensions to the Cognitive Tutor Algebra: Scripted Collaborative Problem Solving (Rummel, Diziol, McLaren, & Spada)
Self explanations during learning
- Deep-level questions during example studying (Craig & Chi)
- The Effects of Interaction on Robust Learning (Hausmann & Chi)
- A comparison of self-explanation to instructional explanation (Hausmann & Vanlehn)
Our future plans through September 2008:
- continue to try extracting viable data
- write journal article reporting updated findings