Research Interests

Research

Generalizability in Item Response Modeling (GIRM)

An approach called Generalizability in Item Response Modeling (GIRM) is introduced in this paper. The GIRM approach essentially incorporates the sampling model of Generalizability Theory (GT) into the scaling model of Item Response Theory (IRT) by making distributional assumptions about the relevant measurement facets. By specifying a random effects measurement model, and taking advantage of the flexibility of Markov Chain Monte Carlo (MCMC) estimation methods, it becomes possible to estimate GT variance components concurrently with traditional IRT parameters. It is shown how GT and IRT can be linked together, in the context of a single facet measurement design with binary items, and a multi-faceted design with polytomous items. Using simulated data and the software WinBUGS, the GIRM approach is shown to produce results comparable to those from a standard GT analysis, but with certain advantages due to the incorporation of the IRT formulation.

This paper has been submitted for publication to the Journal of Educational Measurement.

Meta-Analysis: A Case Study

This paper discusses the usefulness of meta-analysis as a means of reviewing quantitative educational research. When a meta-analytic model for SAT coaching is used to predict the results from future studies, the amount of prediction error is quite large. Interpretations and the process of quantifying study characteristics are shown to be questionable. The match between the assumptions of the meta-analytic model and the data from SAT coaching studies is not good. Statistical inferences are therefore problematic. An alternative to the meta-analytic approach is suggested.

This paper has recently been published in the journal Evaluation Review.

Causal Inference and the Heckman Model

In the social sciences, evaluating the effectiveness of a program or intervention often leads researchers to draw causal inferences from observational research designs. Bias in estimated causal effects becomes an obvious problem in such settings. I present the Heckman Model as an approach sometimes applied to observational data for the purpose of estimating an unbiased causal effect. I show how the Heckman Model can be viewed as an extension of the linear regression model, and discuss in some detail the assumptions necessary before either approach can be used to make causal inferences. Linear regression and the Heckman Model make different assumptions about the relationship between two equations in an underlying behavioral model: a response schedule and a selection function. Under linear regression the two equations are assumed to be independent; under the Heckman Model, the two equations are allowed to be correlated. The Heckman Model is particularly sensitive to the choice of variables included in the selection function. This is demonstrated empirically in the context of estimating the effect of commercial coaching programs on the SAT performance of high school students. I estimate coaching effects for both sections of the SAT using data from the National Education Longitudinal Study of 1988 (NELS). Small changes in the selection function are shown to have a big impact on estimated coaching effects under the Heckman Model.

This paper was presented at the 2003 meeting of the National Council on Measurement in Education and was published in 2004 in the Journal of Educational and Behavioral Statistics.

Diagnostic Assessment with Ordered Multiple-Choice Items

This research is being conducted in collaboration with colleagues at the University of California, Berkeley (Mark Wilson & Cheryl Schwab) and Caltech (Alicia Alonzo). It stems from an NSF-funded project in collaboration with WestEd to analyze the impact of differing item formats on the testing of student understanding of science in large-scale settings. In this paper we describe the development, analysis and interpretation of a novel item format we call Ordered Multiple-Choice (OMC). A unique feature of OMC items is that they are linked to a model of student cognitive development for the construct being measured. Each of the possible answer choices in an OMC item is linked to developmental levels of student understanding, facilitating the diagnostic interpretation of student item responses. OMC items seek to provide greater diagnostic utility than typical multiple-choice items, while retaining their efficiency advantages. On the one hand, sets of OMC items provide information about the developmental understanding of students that is not available with traditional multiple-choice items; on the other hand, this information can be provided to schools, teachers and students quickly and reliably, unlike traditional open-ended test items.

This paper is in press with the journal Educational Assessment.

Selected Publications

Briggs, Derek C and Wilson, Mark. (2003) An Introduction to Multidimensional Measurement using Rasch Models. Journal of Applied Measurement, 4(1), 87-100.

Briggs, Derek C. Test Preparation Programs: Impact. Encyclopedia of Education. 2nd Edition (2002).

Stern, David and Briggs, Derek (2001) Does Paid Employment Help or Hinder Performance in Secondary School? Insights from US high school students. Journal of Education and Work, Vol. 14(3), 355-372.

Briggs, Derek C. (2000) The Effect of Admissions Test Preparation: Evidence from NELS-88. Chance. Vol. 14(1), 10-18.

Stern, David S and Briggs, Derek (2000) Changing Admissions Policies: Mounting Pressures, New Developments, More Questions. Change. January/February 2001, Vol. 33(1), 34-41.