Validity

Establishing validity confirms the ways that an assessment serves its shareholders. Therefore, when researching and debating any assessment model, it is imperative that we address the following questions through descriptive and inferential analysis of empirical evidence:

• What variables are incorporated in the model used to drive the assessment process? That is, how is the topic under examination defined, how will that topic be assessed, how will the results of the assessment be communicated, and what is the impact of the assessment of the community’s shareholders?
• In defining the topic to be assessed, is there a relationship among the performance variables? That is, if student performance is to be captured in a digital portfolio, what performances are expected to be captured within the portfolio, and what is the relationship among those performances?  
• If there is a relationship, what is the nature of that relationship? Does, for example, high performance on a variable of document design cause a high portfolio score, or is the ability to design documents associated with a high portfolio score?
• If there is a relationship between independent (predictor [e.g., the ability to design documents]) variables and dependent (outcome [e.g., the overall digital portfolio score) variable, do results using the model meaningfully capture the variable relationships?
• If variable relationship are identified and meaningfully captured by the model, can this model be generalized across assessment contexts?

Numerous statistical techniques can be used to support claims of validity, and we welcome these studies as evidence that our assessment models are serving their shareholders. That is, in program assessment, we value the achievement of consequential validity through empirical analysis. We seek outcomes assessment models that educate students and prepare them to become working professions.  We wish to learn more about the process of assessment-driven curriculum transformation.

In Context
By conducting an analysis of the independent variables (core competencies) and dependent variable (overall portfolio score) across multiple administrations, we were able to establish empirical validity of the NJIT model. This work continues at the present writing. An examination of measures of central tendency (range, mean, and standard deviation) suggests that the model can be used to assess student ability. (See Table: Measures of Central Tendency). A high internal consistency, or inter-relatedness of the independent core competency variables, was established using associative analysis. (See Table: Core Competency Correlations). Regression analysis consistently revealed a high coefficient of determination indicating that between 83% and 93% of the variability of the overall portfolio score can be explained by the eight core competency independent variables¹.

We invite CPTSC members to provide us with their own validity studies so that we may expand our community.

Suggested Resources on Validity

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. Standards for Educational and Psychological Testing.Washington, DC: APA. 1999.

Brennan, Robert L.  Eduational Measurement.  4th ed.  Westport, CT: Praeger P., 2006.

Educational Researcher. Spec. issue on Validity. 36.8 (2007).

Guba, Egon G. and Yvonna S. Lincoln. Fourth Generation Evaluation. Newbury Park, CA: Sage, 1989.

Messick, Samuel. “Validity.” Educational Measurement. 3rd ed.. Ed. Robert Linn. Washington, DC: American Council on Education and National Council on Measurement in Education, 1989. 13-103.