Frequently Asked Questions
General:
What is the LearnLab?
Learnlab is a facilty designed to dramatically increase the ease and speed with which learning researchers can create the rigorous, theory-based experiments that pave the way to an understanding of robust learning.
Run jointly by
PSLC's LearnLab is a national resource for learning research that includes:
- Authoring tools for online courses, experiments, and integrated computational learner models
- Support for running in vivo learning experiments
- Longitudinal microgenetic data from entire courses
- Data analysis tools, including software for learning curve analysis and semi-automated coding of verbal data.
What is in
vivo
experimentation?
Rigorous experimentation with laboratory-quality methods in real classroom settings.
Features of an in vivo learning experiment include: extending realism to an existing laboratory result, extending rigor to an existing classroom result, extending generalization to a result from a different domain, using rigorous experimental design features like random assignment, using technology to provide tight control on the micro-structure of instruction, making use of fine-grained longitudinal data by analyzing second-by-second interactions happening over weeks, and coding and analyze such data in terms of general cognitive theory and a specific theory of domain knowledge. It is precisely these features that PSLC allows researchers to more easily address as never before.
What is
robust
learning?
• Future
Learning: It
accelerates future learning. That is, when
related instruction is presented in the future, this knowledge allows
them to learn more quickly and effectively learn from it.
How is in vivo research conducted?
The
PSLC component relationships are displayed above.
Using in
vivo experimentation, principles of robust learning are
tested in studies
that occur in actual classrooms (LearnLab Courses) during an actual
course. This leads
to a better
understanding of learning theory which, in turn, leads to more in vivo
experimentation. PSLC
Enabling
Technology supports these activities by providing course development,
logging,
and analysis tools.
Research is conducted in seven core LearnLab courses: Algebra, Geometry, Physics, Chemistry, French, Chinese and English as a second language. These courses as well as some affiliate courses host PSLC’s in vivo experimentation.
How is
running a in
vivo study different from running a lab study?
To test a hypothesis requires a tightly controlled design where the only difference between experimental and control conditions is the principle being tested. For the test to be useful and trustworthy, it should meet as many of the features of an in vivo learning experiment as possible: extend realism to an existing laboratory result, extend rigor to an existing classroom result, extend generalization to a result from a different domain, use rigorous experimental design features like random assignment, use technology to provide tight control on the micro-structure of instruction, make use of fine-grained longitudinal data by analyzing second-by-second interactions happening over weeks, and code and analyze such data in terms of general cognitive theory and a specific theory of domain knowledge. It is precisely these features that PSLC allows researchers to more easily address as never before.
What do I do
if I
have problems with a LearnLab course chair or with a LearnLab teacher?
If you are having difficulties with a course chair and you feel that your issues cannot be resolved amicably, please see the PSLC Managing Director, Michael Bett. Michael will negotiate with the course chair and the researcher in order to find a workable and amicable solution.
If you are having difficulties with a course instructor that you feel cannot be resolved, please contact both the Course Leader and the PSLC Site Manager (or the PSLC Managing Director). All of these folks will work together to resolve the issues.
Using the PSLC Website
Who is the PSLC webmaster?
Ben Billings – bkb @ cs.cmu.edu. Contact Ben with any issues regarding the PSLC website.
What mailing lists are there and how so I subscribe? Mailing lists exist for each cluster, LearnLab course, graduate students, and executive committee as well a general announcement list. To subscribe to one or more of these mailing lists, contact Jo Bodnar via email (jobodnar @ cs.cmu.edu)
How do I get events added to the PSLC calendar? To have an event added to the calendar, contact Jo Bodnar (jobodnar @ cs.cmu.edu). If you will need to have regular access to the calendar to add or modify events, contact Michael Bett.
How do I submit a new project to the PSLC?
How do I submit a revision to an existing project?
Login to the PSLC website (LearnLab.org). Select My PSLC from the menu on the left; then select My Projects. A new window will open. Select the project you wish to revise from the list of your projects. Another new window will open. Select the button labeled 'Add Revision'.
Currently you must be the first
listed investigator to revise an existing plan.
What is “My PSLC”?
Publications & Conferences:
How do I acknowledge funding and support provided by PSLC?
In the acknowledgements section of your paper please state:Funding for this research was provided by the Pittsburgh Science of Learning Center DataShop which is funded by the National Science Foundation award No. SBE-0354420.
Where can I find a list of educational technology conferences?
See http://www.digitalsy.org.uk/html/SIGs/elearning/docs/Conferences.htm
Where can I find a list PSLC publications?
PSLC publications are listed under research on the PLSC webpage. See http://www.learnlab.org/research/papers.php
Is there a PSLC reprint/publication print series?
No, but most of the PSLC related papers can be found on our publications page. See http://www.learnlab.org/research/papers.php
DataShop
What
is the DataShop?
The DataShop is a data analysis service for
the PSLC
community. It is
both a central
repository for research data and a suite of tools to analyze that data.
How
do I access the DataShop?
You
can access the DataShop by going to: http://learnlab.web.cmu.edu/datashop.
If you are a
What
are the capabilities of the DataShop?
DataShop provides a number of analysis
reports. The current
reports are:
- Learning Curve by Knowledge Component
- Learning Curve by Student
- Step Rollup, a new tabular
report that arranges transaction data by step, and allows you to export
that data.
- Error Report by Problem
- Error Report by Knowledge Component
(aggregate data)
The current tools are:
- Sample Selector, a tool for creating and
modifying custom samples; and
- Data Export, a facility for exporting
anonymized data to a tab-delimited text file.
What can the
DataShop do
for me?
The
first step is data collection. We have developed an XML logging
standard and
two logging libraries (one in Flash ActionScript, the other in Java) to
write
this XML.
Then,
after importing the data into a database, the DataShop web application
can get
you started on your data analysis with the high frequency reports. As a researcher, you can
sample the data and
compare across samples, and export the data for further manipulation
and
analysis in other tools.
What
statistical support is available in
the DataShop?
As
of June, 2006 the statistical support directly available in the
DataShop is
limited to statistics on learning curves (see below).
However, you can export the data to a file
and use your favorite statistical software package.
The
DataShop integrates the Learning Factors Analysis algorithm (Koedinger,
Junker
1999) so that you can view a predicted learning curve.
With combinatorial search and a built-in
mathematical model measuring student proficiency, knowledge component
difficulty,
and knowledge component learning rates, this algorithm is able to
quantify the
student learning process for different knowledge components and predict
their
performance on each use of the knowledge component. The LFA algorithm
has been
shown to be able to help identify hidden difficulties of knowledge
components
that may hinder student learning (Cen, Koedinger, and Junker 2005).
What
data is available through the
DataShop?
The
list of data sets is growing all the time.
It’s best to go to the website
and take a look: http://learnlab.web.cmu.edu/datashop.
What
format is the data in? What kind of
data gets logged? In what form do I get the data?
The
DataShop stores its data in a relational database.
Currently, we store tutor log data with plans
of storing more types in the future (e.g., audio and video data,
writing
samples).
Your
data can be exported in a tab-delimited text file.
Can I use the DataShop data for my own
research purpose?
For
each data set, if you are the PI or have permission from the PI, you
may
examine the data sets and use them in your own research. This includes PSLC and
non-PSLC
researchers. To
request permission to
use a data set first create an account (see “How do I access
the DataShop?”
above) and then contact Alida Skogsholm at alida@cs.cmu.edu.
I
ran a LearnLab study. Who has
access to my data, and do I have any say in that?
The
principal investigator of a LearnLab study has full control over
his/her own
data. With a new
data set, we allow no
one but the PI to access the data.
Then,
as permitted by the PI, we will add other users.
How
do I get/create custom queries and/or
reports that I need to feed into SPSS?
Researchers
are encouraged to contact the DataShop team if a custom query or report
is
needed that is currently not available. We are continually trying to
understand
our users’ needs so that we can provide functionality that
reduces the amount
of time a researcher spends preparing data for analysis.
Data
can be exported from the DataShop by using the export button on either
the
Export or Learning Curve (Step Rollup) page.
What
is the timeframe between completing a
study and getting data from the DataShop? If I run a LearnLab
study, at
what point do I get access to the data?
The
principal investigator will have access to his/her own data as soon as
it is in
the DataShop.
The
timeframe varies from one day to a couple of weeks depending primarily
on the
source of the data. Tutors which log directly to the PSLC server are
moved into
the DataShop’s database daily.
For that
reason, we strongly encourage you to develop Example-Tracing tutors
using CTAT,
which can automatically log data to the PSLC server for you.
Tutors
which produce log data but do not log directly to the PSLC server, such
as
What
is the required lead time needed for
customized analyses?
The
availability of customized analyses varies with the request. The
DataShop
development team releases a new version of the web application three
times a
year. If your
request is for a new
report in the web application, then it could take 3-6 months to develop
and
release a new report. Commonly
requested
reports will be implemented first.
If
the request is for a custom report, then a lower fidelity solution
might be
more appropriate, which would reduce the lead time considerably. For instance, you can
export the data and use
other reporting and analysis tools, like a spreadsheet or statistical
analysis
tools. The DataShop
team or PSLC members
more generally may be able to help you to determine what is the best
option. (For
instance, many kinds of reports can be generated from Excel if you know
how to
use features like Pivot Tables and Auto Filter.)
We
highly encourage you to make requests so that we know what you need. You never know how hard
something is or how
long it will take to build until you ask.
What
restrictions are there on publishing
about the data?
As
long as proper IRB rules and guidelines have been followed, and you are
either
the PI or have permission from the PI of the data, then you may publish
the
data or an analysis of the data.
You
must acknowledge the source of the data in your publication. You should say something
like:
Data used in this research was provided by the Pittsburgh Science of Learning Center DataShop which is funded by the National Science Foundation award No. SBE-0354420.
Where can I get more
help?
DataShop
documentation is online at http://learnlab.web.cmu.edu/datashop/help
You
can also subscribe to the DataShop users email list at https://lists.andrew.cmu.edu/mailman/listinfo/datashop-users
or
email the DataShop team at
datashop-help@lists.andrew.cmu.edu
CTAT:
What is CTAT?
CTAT stands for the Cognitive Tutor Authoring Tools. These tools can be used to create computer-based tutors for use in LearnLab experiments and LearnLab courses in all 7 LearnLab subject matter areas.
CTAT supports two types of tutors. Example-Tracing Tutors can be built without programming. Cognitive Tutors require Artificial Intelligence programming to develop a cognitive model. Both types of tutors provide step-by-step assistance to students as they solve problems.
CTAT builds on the success of the Cognitive Tutors for high-school math, which were developed at CMU and have been proven to improve student learning, compared to regular classroom instruction. More information can be found about CTAT by going to http://ctat.pact.cs.cmu.edu/.
What is the role of CTAT
in LearnLab?
CTAT is a key enabling technology within the PSLC. Tutors built with CTAT are suitable vehicles for a particular type of in vivo experiment, in which tutors are used to deliver experimental treatments. The advantages of using computer tutors for that purpose are that one can administer experimental treatments in a consistent manner (e.g., no unwanted “random” variability) and ensure detailed logging of student activities in DataShop format. CTAT can also be used by LearnLab course developers to add tutoring capability to their courses.
How do I budget for help
from the CTAT team in my project
plan?
The CTAT project team offers consulting and technical services to the PSLC community, as part of their mission within the PSLC. It also tries hard to be driven by requests from real users when implementing new features. To find out more about how you can use CTAT consulting resources, or to suggest a new CTAT feature, contact either Vincent Aleven (aleven @ cs.cmu.edu <mailto:aleven @ cs.cmu.edu>) or Bruce McLaren (bmclaren @ cs.cmu.edu <mailto:bmclaren @ cs.cmu.edu>).
How
do I record logs of student activities with
CTAT-built tutors?
All CTAT tutors have easy to switch on / switch
off logging support. These tutors log a detailed record of all
student-tutor
interactions to the PSLC DataShop, a service offered by the PSLC for
recording
and analyzing data from LearnLab experiments. These facilities make it
possible
to do detailed analyses of students’ learning trajectories
and also to do
longitudinal analyses.
A comprehensive FAQ on logging to the PSLC DataShop exists on the DataShop web site: https://learnlab.web.cmu.edu/datashop/help_logging.jsp
In order to make it possible to provide logging from (Java or Flash) applications that are not built with CTAT, Java and Flash logging libraries are made available separately (see http://www.learnlab.org/technologies/datashop/downloads.php).
Can I use Flash with CTAT-built tutors?
CTAT supports tutors whose user interface is built in either Flash or Java. This goes for Example-Tracing Tutors and Cognitive Tutors.
Tutors with a Flash interface provide somewhat better web delivery of Example-Tracing Tutors than tutors with a Java-based interface. However, Cognitive Tutors built with a Flash interface do not yet run over the web. They do run locally. This restriction is likely to be lifted in the future, so please check back with us.
Can I use CTAT to build tutors that work on the web?
Yes. Flash-based Example-Tracing Tutors run on the web. Java-based Example-Tracing Tutors and Cognitive Tutors run on the web, using Java Webstart technology. In the future, we are likely to extend CTAT so that all Flash-based tutors run on the web (i.e., Example-Tracing Tutors and Cognitive Tutors).
How do I extend CTAT so that it is better suited for the way I plan to use tutors in one of the LearnLab courses?
The short answer is: if you need some capabilities that CTAT does not provide and have the technical means to implement them yourself, the CTAT team will likely (a) make the necessary source code available to you and (b) offer to consult with you on how to design your new feature and code modifications in such a way that they will likely be of benefit to other PSLC researchers.
One way to extend CTAT’s capabilities that may be attractive to development teams within the PSLC is to add to CTAT’s set of interface widgets. CTAT comes with a variety of “tutorable” Java and Flash widgets that allow you to build tutors without programming. Those who need new widgets can create them either in Flash or in Java. These new widgets must be made to observe the API for CTAT tutor widgets. Once a new widget is built, it can be used to develop tutors in the usual way. For example, Example-Tracing Tutors that include the new widget can be developed without programming.
The Flash-based tutor widgets can be customized relatively easily, by writing ActionScript without having to author complex event listeners and handlers; you only need to define the code that runs when student input in a widget is marked as correct or incorrect.
Contact either Vincent Aleven (aleven @ cs.cmu.edu) or Bruce McLaren (bmclaren @ cs.cmu.edu), to find out how to get help in extending/customizing CTAT.