Difference between revisions of "Providing optimal support for robust learning of syntactic constructions in ESL"

From LearnLab
Jump to: navigation, search
(Experiment Methods)
(Experiment Methods)
Line 144: Line 144:
 
:The model of learning that will be used to interpret this data is already being used for [[optimized scheduling]] in other projects (e.g. [[Optimizing the practice schedule]]). This model is a version of the ACT-R declarative memory model which is a mathematical model consisting of a system of equations for describing expected performance as a function of a history of performance. To apply this model to our data will be a several step process.
 
:The model of learning that will be used to interpret this data is already being used for [[optimized scheduling]] in other projects (e.g. [[Optimizing the practice schedule]]). This model is a version of the ACT-R declarative memory model which is a mathematical model consisting of a system of equations for describing expected performance as a function of a history of performance. To apply this model to our data will be a several step process.
 
:*Describe a strucutral knowledge component model (e.g. 4 knowledge components, one for each factor)
 
:*Describe a strucutral knowledge component model (e.g. 4 knowledge components, one for each factor)
:*Use this strucutral model to determine how the history maps to performance for each item (e.g. performance for each item may be a compensatory function of the item features specifying th e4 facotrs and the historyd of practice with each factor)
+
:*Use this strucutral model to determine how the history maps to performance for each item (e.g. performance for each item may be a compensatory function of the item features specifying the 4 facotrs and the history of practice with each factor).
 
:*Given this model structure and mapping to history optimize parameters such as
 
:*Given this model structure and mapping to history optimize parameters such as
 
:**learning rate for implicit items
 
:**learning rate for implicit items

Revision as of 09:35, 29 March 2007

PIs Levin, Frishkoff, De Jong, Pavlik
Faculty Levin
Postdocs Frishkoff, De Jong, Pavlik
Others with > 160 hours n/a
Start date study 1 March 2007
End date study 1 April 2007
Learnlab ESL
Number of participants 30
Total Participant Hours 40
Datashop? Expected date 7/15

Abstract

The goal of this project is to examine how second-language learners acquire context-appropriate use of syntactic constructions*. In some cases, learning when to use a syntactic construction is straightforward. In other cases, use of a syntactic construction is seldom mastered, even by advanced students. The main challenge, in such cases, is to learn which contextual cues predict the occurrence of a particular form (for example, I am here for two years vs. I have been here for two years). That is, students must learn the “meanings” or functions of grammatical constructions, in order to use them in appropriate contexts.

This project focuses on acquisition of the dative alternation* ( give someone a book vs. give a book to someone) by ESL students. Recently, Bresnan and associates (Bresnan & Hay, 2006; Bresnan & Nikitina, 2003; Bresnan, Cueni, Nikitina, & Baayen, 2005) identified 14 features that combine to form a linear regression model of the dative alternation for native English speakers. Using this model as a blueprint, we propose to develop a student-centered, cognitive–linguistic model that will determine optimal scheduling of example texts to support acquisition of the dative alternation in ESL. We propose to use a constraint-based model ( the Bresnan model*), combined with a model of learning (the Pavlik model), to select training examples that we will present in a way that is hypothesized to result in optimal refinement, transfer, and retention, of proficiency with the dative alternation. This approach will comprise the following steps:

(A) The native-speaker model will constitute the target for ESL acquisition of the dative alternation.
(B) We will calculate distances between student performance and the native speaker model.
(C) We will select training items that maximize learning, i.e., that reduce the distance between the ESL student models and the native-speaker model.

Background

The stimuli for the experiment are based on Switchboard, which is a corpus of telephone conversations between native speakers of English. Switchboard was recorded by the Linguistic Data Consortium in the early 1990's for the purpose of speech research. The Bresnan team selected 2600 sentences from the Switchboard corpus that contain agent, theme, and recipient arguments. For each sentence, they annotated the values of fourteen features (is the action concrete or abstract, is the theme a pronoun, is the recipient a pronoun, etc.) A linear regression model trained on the features has 92% accuracy in predicting whether the native speaker chose the double object or prepositional option in each sentence. We want to know whether we can get non-native speakers to make the same choice that the native speaker made. However, we can't show the original dialogue to the non-native speaker because it contains disfluencies, culture-specific information, and advanced vocabulary. We are therefore adapting the Switchboard sentences for ESL students, but we must be sure not to change the values of any of the 14 features. Following is an example of a Switchboard segment and the adapted version for our experiment. The subjects make a forced choice between the double object and prepositional options.

Original

Well, the thing I think that annoys me the most is, I have, I have young children, a baby in the house and, and inevitably as soon as they're asleep, someone calls on the phone trying to sell me something.


Adapted

I have young children at home. As soon as they are asleep at night, someone calls on the phone, trying to sell

  • me something. (*NP-NP or "Double Object" Construction)
  • something to me. (*NP-PP or "Prepositional" Construction)

Significance

Novelty of the project: Almost all previous studies of the dative alternation in ESL have focused on the form of the dative alternation, rather than whether it is used appropriately in context. Our study will be the first to —

  • Teach the use (meaning or function) of the dative alternation;
  • Provide an implemented model of how the dative alternation is learned; and
  • Test whether models of native speaker use can serve as the basis for effective instruction augment the native speaker models with a model of how the dative alternation is learned

In the course of this study, the Pavlik and Anderson model will be applied to a new area: the interaction of multiple features in second language acquisition We will build a framework (theory, model, and tools) that can be re-used in subsequent studies of difficult-to-use constructions.

Relevance for Second-Language Learning Research: While the dative alternation itself represents a relatively small part of the English grammar, it is one instance of a broader (and relatively productive) class of constructions known as resultatives, which include non-prototypical uses of verbs like “sneeze” to express change-of-state — e.g., “He sneezed the letter across the table” (cf. Goldberg, 1995). Resultative constructions have received considerable attention among linguists (see Goldberg & Jackendoff, 2004 for a recent review), particularly in recent research on grammar acquisition (e.g., Goldberg & Casenhiser, 2004, 2005). Still more broadly, the dative alternation involves transitivity relations (also referred to as voice, or diathesis), which are represented cross-linguistically using several important syntactic devices (Givón, 1984). Voice markers are often associated with complex meanings and functions (cf. Frishkoff, 1997), and acquisition of grammatical voice in some languages occupies a large part of the curriculum. Therefore, although our selection of the dative may suggest a narrow focus, our studies should have wide-spread implications for acquisition of transitivity and diathesis relations in many (perhaps all) languages.

We view this project as seeking to establish “proof of concept,” which will justify work in other domains of grammar learning using this same approach. Our goal is to tune our procedures to our findings, particularly with respect to the degree to which training items affect student behavior, and the proportional effects of student performance vs. experience. After we tune our procedures for the dative alternation, we will extend our methods to address errors in the use of articles and tense-aspect constructions.

Contribution to the theory of robust learning This project relates to refinement and fluency in the learning process. Our hypothesis is that we can strengthen and refine feature representations to approximate native speaker competence. The instructional goals of this project focus on long-term retention and transfer. Following the Pavlik and Anderson model, training items are selected and spaced to optimize long-term gain. Transfer is expected from trained to untrained items, from comprehension to production, and from prototypical feature instances to less prototypical feature instances.

Glossary

Alternation Pair
A pair of sentences with the same verb, agent, recipient, and theme, where one sentence of the pair is a double object construction and the other is a prepositional dative construction:
  • Gretchen sent her a form.
  • Gretchen sent a form to her.
Argument
A participant in the action described by the verb. The sentence I gave a book to him has three arguments, I, book, and him.
Bresnan Model
A logistic regression model including fourteen features that predicts whether native English speakers will use the double object or prepositional variant of a dative sentence.
Dative Alternation
There are two ways to express sentences that contain agent, recipient, and theme arguments. In the double object construction, the recipient comes first, followed by the theme, with no prepositions. In the prepositional dative construction, the theme comes first, followed by the preposition to and then the recipient.
  • Gretchen sent her a form. (double object)
  • Gretchen sent a form to her. (prepositional dative)
Double object construction or NP NP construction
In the double object construction, the recipient argument comes first, followed by the theme argument with no prepositions.
  • Gretchen sent her a form.
  • This music gives me a headache.
  • The light gave her features a healthy glow.
NP NP Construction
See Double object construction
NP PP Construction
See Prepositional dative construction
Prepositional dative construction or NP PP construction
In the prepositional dative construction, the theme comes first, followed by the preposition to and then the recipient.
  • Gretchen sent a form to her.
  • The teacher told a story to the children.

The following sentence is also an instance of the prepositional dative construction, but it has undergone an alternation called Heavy NP Shift. The recipient retains its preposition and the theme moves to the end of the sentence because it is long (heavy).

  • Gretchen sent to her a long form containing many confusing questions.
Recipient
The argument that receives something in an abstract or concrete way. The recipient arguments are in italics in these examples:
  • Gretchen sent her a form.
  • Gretchen sent a form to her.
Syntactic Construction
A recognizable configuration of words and morphemes (prefixes and suffixes). Constructions may contain fixed expressions (What a ADJ NOUN!, What a nice dress!). They may also be normal parts of the syntax of the language such as using a noun phrase before a verb phrase to make a predication (The girl ran). Constructions have a form and a meaning or use. ESL textbooks may have lessons on when to use the present perfect ( have Verb-ed, I have eaten).
Theme
The argument that is moved, given, or communicated to the recipient in an abstract or concrete way. The theme arguments are in italics in these examples:
  • Gretchen sent her a form.
  • Gretchen sent a form to her.

Research Question(s)

Our central research question concerns the learning of complex form-meaning mappings, particularly those as complex as the dative shift, which involves fourteen meaning-related features (knowledge components) with different weights (cue strengths). We want to know whether a model of native speaker behavior can be used as a target for non-native speaker behavior and what instructional interventions (learning events) will bring non-native speakers closer to that target.

Hypotheses

We propose the following hypotheses:

(A) Learning will be more effective when training examples are selected to maximize the strength of model cues (high-contrast* versus low-contrast* examples)
(B) An algorithm that selects training items on the basis of student performance, as well as on the basis of training history, will lead to better performance than one that selects training items based on the history of training alone (optimized scheduling).
(C) Selection of examples based on student performance will be superior in:
  • transfer from trained to untrained items
  • transfer from comprehension to production
  • transfer from prototypical to less prototypical examples, provided they are presented at the right times, with the right frequencies.

Study 1 will calibrate the learning model. The effects of learning will be measured in terms of feature weights that characterize the student's responses. We will measure the size of the effect that is caused by exposure to specific types of examples. Study 1 also measures medium term forgetting between blocks of stimuli.

Study 2 will test whether trial selection by the learning model leads to native-like behavior. The learning model uses information about which trials have been presented as well as the students' performance. In the control condition, trials are selected based on the history of practice only, not on student performance (i.e., the performance sensitivity in the model will be turned off). In other words, in the control condition, the trial-selection is model-based, but not personalized, as in a classroom where all students see examples in the same order. It is expected that both conditions will result in learning, but that there will be higher gains in the experimental condition than in the control condition.

A follow up session will test two types of transfer: 1) from comprehension to production, and 2) from prototypical to non-prototypical features. The first type of transfer will be tested by including a posttest for production, in which students put sentence constituents into the preferred word order. The second type of transfer will be tested by including a number of test items with non-prototypical features. Some features are binary, such as pronominality ( the book is non-pronominal, whereas it is pronominal), while other features can be prototypical or non-prototypical. For example, in a context where only a house and a man are mentioned, the house is highly accessible in subsequent context, whereas, for instance, the clouds is not. However, in the same context, the kitchen and his wife have intermediate accessibility because they are implied by the mentioning of a house and a man.

Experiment Methods

Dative Model
The logistic regression model proposed by Bresnan and Nikitina (2003) includes 14 linguistic (syntactic, semantic, and discourse-pragmatic) variables that account for Native English speaker use of the dative alternation (model accuracy, ~92%). We applied Principal Components Analysis (PCA) to obtain a smaller set of variables that would be more amenable to experimental manipulation. The input to the PCA consisted of 2360 rows X 14 columns, where columns represent the 14 linguistic variables in the original Bresnan model, and rows are speech samples from the Switchboard corpus that include examples of the dative alternation (either an NP-NP or NP-PP construction for each sample). The data were transformed into a 14 x 14 correlation matrix, which was decomposed using PCA with varimax rotation. The resulting Pattern Factor Matrix showed a sensible clustering of variables. Givenness, Definiteness, and Pronominality of the Theme loaded on Factor 1 (variance accounted for ~23%). Givenness, Definiteness, and Pronominality of the Recipient loaded on Factor 2 (variance accounted for ~15%). Concretness (vs abstractness) of the Theme and verb semantics loaded on factor 3 (~ 9% variance). Relative length of the Theme and Recipient split across Factors 1 and 2 in the first analysis. The four variables that had the smallest contribution were dropped from the second analysis, resulting in a new 5-factor structure, where length loaded separately on Factor 4, and grammatical Person of the Recipient loaded uniquely on Factor 5. The first four factors were selected for manipulation in Studies 1 and 2.
Stimulus development
The Bresnan corpus consists of 2360 examples from the Switchboard corpus, with have been annotated for each of the 14 Bresnan model variables. For development of experiment stimuli, we selected 12 samples for each of 16 experiment conditions (see Independent Variables for details). The original 192 speech samples were modified (shortened, corrected, simplified in grammar and word choice), taking care not to affect linguistic variables, such as givenness and length.
Study Participants
Study participants were volunteers, recruited from Levels 3-5 (intermediate level) courses at the English Language Institute (ELI), University of Pittsburgh and native English speaking subjects, recruited from the Reading & Language Lab database. Native English speakers were included to test the accuracy of the reduced (4-Factor) model. Also, consistent with Hypothesis (A), we expected native-speaker task performance to be higher and less variable in the high-contrast vs. low-contrast condition.
Experiment Design & Protocol
Participants completed two sessions, scheduled one week apart. Subjects were paid for their participation ($15/hour plus an additional amount that was contingent on task performance, averaging ~$5-8). Session I. Prior to Session I, participants completed a Language History Questionnaire and read and signed a consent form. They then completed a sequence of 8 blocks (16 trials per block). On each trial, they were presented with a context* (one or two short sentences), which ended with a ditransitive verb, followed by two alternative completions (either an NP-NP or an NP-PP structure). Subjects selected the best completion by pressing the '1' or '2' key on the keyboard (response mapping randomized). A trial counter at the top of each screen tracked and displayed subject accuracy on each trial (number correct/number trials completed). ELI subjects took approximately 1.5-2 hours to complete Session I, and native English speakers took ~1-1.5 hours. Session II. In Session II (one week later), subjects completed another 4 blocks (64 trials). At the end of the task, they completed a 3-page questionnaire that was designed to test learning and retention of the grammar rules that were introduced in the first session. Subjects completed Session II in 1-2 hours.
Learning Model
The model of learning that will be used to interpret this data is already being used for optimized scheduling in other projects (e.g. Optimizing the practice schedule). This model is a version of the ACT-R declarative memory model which is a mathematical model consisting of a system of equations for describing expected performance as a function of a history of performance. To apply this model to our data will be a several step process.
  • Describe a strucutral knowledge component model (e.g. 4 knowledge components, one for each factor)
  • Use this strucutral model to determine how the history maps to performance for each item (e.g. performance for each item may be a compensatory function of the item features specifying the 4 facotrs and the history of practice with each factor).
  • Given this model structure and mapping to history optimize parameters such as
    • learning rate for implicit items
    • learning rate for explicit items
    • learning from instructions
  • Analyse the fitted model to determine how to optimize the long-term learning gain per second of practice.

Independent variables

For Study 1, the independent variables are factors, contrast, and time (session, half of session, and half of block).

  • Factors: Using principal components analysis, the fourteen features of the Bresnan model were reduced to four factors(see Methods for details).
    • Factor 1: Definiteness, pronominality, and discourse accessibility (givenness) of the theme.
    • Factor 2: Definiteness, pronominality, and discourse accessibility (givenness) of the recipient.
    • Factor 3: Ratio of lengths of the theme and recipient noun phrases.
    • Factor 4: Abstractness vs. concreteness of the action and of the theme argument.

Each factor has two values, which we can call plus and minus: e.g., action and the theme are abstract ( pay attention to someone) or concrete ( give something to someone).

  • Contrast: If the Bresnan model assigns a score close to zero, both the double object and prepositional variants of the sentence are generally acceptable ( send American Express a check vs. send a check to American Express). If the Bresnan model assigns a score farther from zero, one of the two options will be highly preferable ( give it to anyone who comes in vs. give anyone who comes in it).
  • Time - session, half of session, and half of block: The first experiment consists of two sessions, each session having eight blocks, and each block having two halves.
    • First half of first session: One block for each feature, with the plus value for the factor in one half of the block and the minus value of the factor in the other half.
    • Second half of first session: An additional block for each feature (medium term forgetting).
    • Second session: One more block for each feature (longer term forgetting.)


  • Instructions:
  • Feedback:


For Study 2, an additional independent variable is whether training examples are chosen based on student performance (the student's feature weights) or based on history only (which examples have been seen).

Dependent variables

  • Accuracy on a forced-choice (NP-NP vs. NP-PP) task
    • Learning (improvement in accuracy from early to later trials)
    • Long-term retention (improvement in accuracy from the beginning of Session 2, cf. with beginning of Session 1.
    • Transfer: comprehension to production; prototypical to nonprototypical;

Findings

  • Pilot study with native English speakers
In this study we tested the stimuli adapted from the Bresnan corpus with native English speakers. The goals was to confirm that there were no problems with specific stimuli and to verify that native English speaker preferences for NP or PP constructions was consistent with the Bresnan model categorization of these items. 19 subjects completed this test. A repeated measures comparison (factor by contrast) revealed significant differences as a fucntion of factor (F=7.5,p<.001) and contrast (F=110, p<.001). The very strong contrast effects in the data show that the stimuli selection and creation procedures resulted in stimuli that retain the differences the model predicts should occur.
  • Study 1
Study 1 is in progress. The design for study one uses two sessions seperated by a long-term interval of a week. During these sessions we will mix practice for the 4 dative factors and 2 contrast levels and 2 responses using 16 trials per block. Session 1 will be composed of 8 blocks and session 2 composed of 4 blocks.
    • Session 1
      • 1 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
      • 2 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
      • Set of instructional slides discussing the 4 factors
      • 3 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
      • 4 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
      • 5 Block of 16 trials with mixed factors, contrasts and response (yes no feedback and explicit isntruction feeedback)
      • 6 Block of 16 trials with mixed factors, contrasts and response (yes no feedback and explicit isntruction feeedback)
      • 7 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
      • 8 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
    • Session 2
      • 1 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
      • 2 Block of 16 trials with mixed factors, contrasts and response (yes no feeedback only)
      • 3 Block of 16 trials with mixed factors, contrasts and response (yes no feedback and explicit isntruction feeedback)
      • 4 Block of 16 trials with mixed factors, contrasts and response (yes no feedback and explicit isntruction feeedback)


  • Study 2
Study 2 will be completed after development of the Pavlik learning model, based on the results from Study 1.

Explanation & Discussion

This experiment seeks to increase fluency by adjusting cue strengths (weights of the fourteen features of the Bresnan model). It also seeks to achieve long-term retention. The Pavlik-Anderson model predicts the number and spacing of training items needed to change cue strength with long-term retention.

Annotated Bibliography

Bresnan, J. & Hay, J. (2006). Gradient grammar: An effect of animacy on the syntax of give in varieties of English. [draft downloaded from http://www-lfg.stanford.edu/bresnan/download.html]

Bresnan, J. & Nikitina, T. (2003). On the gradience of the dative alternation. [draft downloaded from http://www-lfg.stanford.edu/bresnan/download.html]

Bresnan, J., Cueni, A., Nikitina, T., & Baayen, R. H. (2005). Predicting the dative alternation. Paper presented at the KNAW Academy Colloquium: Cognitive Foundations of Interpretation, Amsterdam.

DeKeyser, R. M. (1994). How implicit can adult second language learning be? AILA Review, 11, 83–96.

Goldberg, A. E. (1995). A construction grammar approach to argument structure. Chicago: University of Chicago.

Goldberg, A. E. & Jackendoff, R. (2004, to appear). The English resultative as a family of constructions. Language.

Inagaki, S. (1997). Japanese and Chinese learner's acquisition of the narrow-range rules for the dative alternation in English. Language Learning, 47, 637-669.

Marefat, H. (2005). The impact of information structure as a discourse factor on the acquisition of the dative alternation by L2 learners. Studia Linguistica, 59(1), 66-82.

Morris, C. D., Bransford, J. D., & Franks, J. J. (1977). Levels of processing versus transfer appropriate processing. Journal of Verbal Learning and Verbal Behavior, 16, 519-533.

Pavlik Jr., P.I. & Anderson, J.R. (2005) Practice and Forgetting Effects on Vocabulary Memory: An Activation-Based Model of the Spacing Effect. Cognitive Science, 29, 559-586.