Difference between revisions of "Extending Reflective Dialogue Support (Katz & Connelly)"

From LearnLab
Jump to: navigation, search
Line 37: Line 37:
  
 
To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b).  We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance.  The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes.  In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufesne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996).  In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores.  However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007).  The extended dialogue condition we implemented differs from the other dialogue condition in three main ways:  (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit.  Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.
 
To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the [[Andes]] physics tutoring system (VanLehn et al., 2005a, 2005b).  We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance.  The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes.  In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufesne, & Mestre (1996): knowledge about ''what'' principle(s) to apply to a given problem, ''how'' to apply these principles (e.g., what equations to use), and ''why'' to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996).  In a prior LearnLab study [[Reflective Dialogues (Katz)|(Katz, Connelly & Treacy)]], this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores.  However, students did not outperform standard Andes users on more [[robust learning]] measures of [[transfer]] (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007).  The extended dialogue condition we implemented differs from the other dialogue condition in three main ways:  (1) reflective dialogues contained more problem variations (''what if'' scenarios), designed to support both qualitative and quantitative knowledge (most of our previous ''what if'' scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context '''and''' to new contexts, to help support near and far [[transfer]]; and (3) students were prompted to state the rules ([[knowledge components]]) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit.  Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.
 +
 +
=== Dependent Variables ===
 +
* ''Gains in qualitative and quantitative knowledge''.  Post-test score,and pre-test to post-test gain scores, on near and far [[transfer]] items.
 +
* ''Short-term retention''. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
 +
* ''[[Long-term retention]]''.  Performance on final exam, taken several weeks after the intervention.
 +
 +
=== Independent Variables ===
 +
Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of [[Knowledge Construction Dialogues]] (KCDs) differed.  The short-KCD condition approximated the treatment condition from last year's study [[Reflective Dialogues (Katz)|(Katz, Connelly & Treacy)]], in which mostly conceptual dialogues followed problem solving on selected [[Andes]] problems.  The new long-KCD condition appended more quantitative practice and [[transfer]] content (via additional ''what if'' scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task.  The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.
 +
 +
The following variables were entered into a regression analysis, with post-test score as the dependent variable:
 +
 +
* Number of problems completed before the post-test was administered
 +
* Number of dialogues that the student completed
 +
* Grade point average (CQPR)
 +
* College major
 +
* Pre-test score
  
  
 
[[Category:Protected]]
 
[[Category:Protected]]

Revision as of 16:40, 15 April 2008

Extending Automated Dialogue Support for Robust Learning of Physics

Sandra Katz

Summary Table

PIs Sandra Katz & John Connelly
Study Start Date 10/1/07
Study End Date 9/30/08
LearnLab Site USNA
LearnLab Course General Physics I
Number of Students N = 75
Total Participant Hours approx. 125 hrs
DataShop Yes

Abstract

Research on student understanding and problem-solving ability in first-year college physics courses shows that some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability, while other students show at least a glimmer of understanding of basic physics concepts and principles but are unable to use this knowledge to solve quantitative problems. The present research seeks to integrate quantitative and qualitative knowledge via post-practice reflection dialogues that guide students in learning and practicing the concepts and principles associated with a just-solved physics problem. It builds upon our 2006 LearnLab study (Katz, Connelly & Treacy) by trying to better support robust learning via a third condition in which students work through longer dialogues designed to foster transfer by specifically tying together qualitative and quantitative knowledge in different contexts.

Glossary

Research question

Does explicit training in the three main components of problem-solving knowledge (i.e., knowledge about what principles apply to a problem, how to apply these principles, and why to apply them), combined with quantitative practice in applying these knowledge components in different contexts, enhance students’ problem-solving ability more than additional problem solving and better foster transfer and robust learning?

Background and Significance

Research on student understanding and problem-solving ability in first-year college physics courses shows that instructors deal with a double-edged sword. Some students become adept at solving quantitative problems but do poorly on tests of conceptual knowledge and qualitative problem-solving ability. Other students display the reverse problem: they show at least a glimmer of understanding of basic physics concepts and principles, but are unable to use this knowledge to solve quantitative problems. Still other students master neither qualitative nor quantitative understanding of physics; very few master both. Thus, the instructional challenge motivating this project is to find effective pedagogical strategies to integrate quantitative and qualitative knowledge. Our scientific goal is to determine whether explicit and implicit learning can be effectively combined via post-practice dialogues that guide students in reflecting on the concepts and principles associated with a just-solved physics problem. The main hypothesis tested is that, in the context of tutored problem solving, integrative reflective dialogues that explicitly tie qualitative knowledge to quantitative knowledge can improve quantitative problem-solving ability and retention of qualitative knowledge better than problem-solving practice (implicit learning) alone.

To test this hypothesis, we conducted an experiment in the PSLC Physics LearnLab at the US Naval Academy in sections that use the Andes physics tutoring system (VanLehn et al., 2005a, 2005b). We compared students who were randomly assigned to one of three conditions on measures of qualitative and quantitative problem-solving performance. The two treatment conditions engaged in automated reflective dialogues after solving quantitative physics problems, while the control condition solved the same set of problems (plus a few additional problems to balance time on task) without any reflective dialogues, using the standard version of Andes. In one treatment condition, the reflective dialogues individually targeted the three main types of knowledge that experts employ during problem solving, according to Leonard, Dufesne, & Mestre (1996): knowledge about what principle(s) to apply to a given problem, how to apply these principles (e.g., what equations to use), and why to apply them—that is, what the applicability conditions are (Leonard, Dufresne, & Mestre, 1996). In a prior LearnLab study (Katz, Connelly & Treacy), this intervention significantly improved students’ qualitative understanding of basic mechanics, as measured by pre-test to post-test gain scores. However, students did not outperform standard Andes users on more robust learning measures of transfer (e.g., performance on quantitative course exams) and on a measure of retention of qualitative problem-solving ability (Katz, Connelly, & Wilson, 2007). The extended dialogue condition we implemented differs from the other dialogue condition in three main ways: (1) reflective dialogues contained more problem variations (what if scenarios), designed to support both qualitative and quantitative knowledge (most of our previous what if scenarios were qualitative only); (2) these “what if” scenarios were tied to the corresponding Andes problem-solving context and to new contexts, to help support near and far transfer; and (3) students were prompted to state the rules (knowledge components) applied to solving the problem variations, in order to promote a principle-based approach to learning, and they were given feedback that makes these rules explicit. Our goal is to determine whether reflective dialogues that make the links between qualitative and quantitative physics knowledge explicit are more effective than both our previous dialogues and an implicit learning condition that is based on problem-solving practice alone.

Dependent Variables

  • Gains in qualitative and quantitative knowledge. Post-test score,and pre-test to post-test gain scores, on near and far transfer items.
  • Short-term retention. Performance on course exams that cover target topics (e.g., work and energy, translational dynamics, rotation, momentum)
  • Long-term retention. Performance on final exam, taken several weeks after the intervention.

Independent Variables

Students within each course section were block-randomly assigned to one of three dialogue conditions in which the usage of Knowledge Construction Dialogues (KCDs) differed. The short-KCD condition approximated the treatment condition from last year's study (Katz, Connelly & Treacy), in which mostly conceptual dialogues followed problem solving on selected Andes problems. The new long-KCD condition appended more quantitative practice and transfer content (via additional what if scenarios) to the short KCDs; students in this condition were assigned five fewer Andes problems than those in the short-KCD condition to attempt to equate time on task. The control condition was identical to last year's; students saw no KCDs and were assigned five more Andes problems than the short-KCD students.

The following variables were entered into a regression analysis, with post-test score as the dependent variable:

  • Number of problems completed before the post-test was administered
  • Number of dialogues that the student completed
  • Grade point average (CQPR)
  • College major
  • Pre-test score