Accessibility of testing within a validity framework

• US 7,217,134 B2
• Filed: 02/09/2005
• Issued: 05/15/2007
• Est. Priority Date: 02/09/2004
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Hansen, Eric G., Mislevy, Robert J., Steinberg, Linda S.
• Primary Examiner(s): Cheng; Joe H.
• Application Number:
  US11/054,019
• Publication Number:
  US 20050175974A1
• Time in Patent Office: 825 Days
• Field of Search: 434/118, 434/112, 434/236, 434/322, 434/323, 434/350, 434/362, 434/365, 705/2
• US Class Current: 434/322
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

Methods for designing an assessment and providing accessibility features for a test taker profile are disclosed. A test taker profile, including multiple knowledge, skill and other attribute(s) (KSAs), is defined in which each KSA has a specific level. Each KSA is designated as either a focal KSA (an essential part of the targeted proficiency to be measured) or an ancillary KSA (not an essential part). For each focal KSA, a focal value (the value that the test taker must have to possess a high level in the targeted proficiency) is defined. For a particular test design alternative (which may include an accessibility feature), a requirement is determined for each KSA. An accessibility feature is selected which avoids indications of invalidity, including: a mismatch between the targeted proficiency and the expected proficiency; the test taker profile does not satisfy all ancillary requirements; and focal requirements are either excessive or insufficient.

Claims

1. A method of examining the validity of an assessment design for a test taker profile, the method comprising:

determining a set of knowledge, skills and other attributes (KSAs) for an assessment design;

for each KSA, defining the KSA to be one of a focal KSA and an ancillary KSA;

defining a focal value for each focal KSA;

defining a test taker profile, wherein the test taker profile includes a profile value for each KSA;

assigning a targeted proficiency value to the test taker profile based on the profile value and the focal value for each focal KSA;

determining, for each KSA, a requirement imposed by the assessment design;

assigning an effective proficiency value to the test taker profile based on the profile value and the requirement for each KSA;

if the targeted proficiency value matches the effective proficiency value;

determining whether an indication of invalidity exists,

if an indication of invalidity exists, assigning a problematic value to a validity variable, and

if an indication of invalidity does not exist, selecting the assessment design as an assessment design alternative for the test taker profile;

if the targeted proficiency value does not match the effective proficiency value, assigning a problematic value to the validity variable; and

if the validity variable has a problematic value, modifying one or more of the assessment design and the test taker profile.

2. The method of claim 1 wherein assigning a targeted proficiency value comprises:

if, for each focal KSA, the profile value is at least as high as the focal value, assigning a high targeted proficiency value to the test taker profile; and

if not, assigning a low targeted proficiency value to the test taker profile.

3. The method of claim 1 wherein assigning an effective proficiency value comprises:

if, for each KSA, the profile value is a least as high as the requirement, assigning a high effective proficiency value to the test taker profile; and

if not, assigning a low effective proficiency value to the test taker profile.

4. The method of claim 1 wherein determining whether an indication of invalidity exists comprises:

determining that an indication of invalidity exists if, for an ancillary KSA, the profile value is less than the requirement.

5. The method of claim 1 wherein determining whether an indication of invalidity exists comprises:

determining that an indication of invalidity exists if, for a focal KSA, the requirement exceeds the focal value.

6. The method of claim 1 wherein determining whether an indication of invalidity exists comprises:

determining that an indication of invalidity exists if, for a focal KSA, the requirement is less than the focal value.

7. The method of claim 1 wherein modifying one or more of the assessment design and the test taker profile comprises one or more of the following:

using, in the assessment design, a changed set of familiarization materials to increase the profile value for at least one ancillary KSA;

using, in the assessment design, a changed set of practice questions to increase the profile value for at least one KSA;

changing, in the assessment design, an ancillary KSA to be a focal KSA;

changing, in the assessment design, a focal KSA to be an ancillary KSA;

changing, in the assessment design, the focal value for a focal KSA;

using, in the assessment design, a changed accessibility feature to perform one or more of decreasing an ancillary requirement, decreasing a focal requirement, increasing a focal requirement;

using, in the assessment design, a changed set of eligibility rules for taking the assessment;

using, in the assessment design, a changed set of eligibility rules for using an accessibility feature;

using, in the assessment design, language having one or more of a lower requirement for an ancillary KSA, a higher requirement for a focal KSA, and a lower requirement for a focal KSA;

using, in the assessment design, questions having one or more of a lower requirement for an ancillary KSA, a higher requirement for a focal KSA, and a lower requirement for a focal KSA;

redefining the test taker profile by changing one or more of the set of KSAs for the test taker profile, a number of values of a KSA that is used in the test taker profile, a definition of values of a KSA that is used in the test taker profile, and a definition of the permissible KSA values;

redefining the test taker profile by realigning one or more values for a KSA to broaden the set of test takers represented at a value for a KSA;

redefining the test taker profile by realigning one or more values for a KSA to narrow the set of test takers represented at a value for a KSA;

using, in the assessment design, a changed set of provisions for score reports; and

using, in the assessment design, changed guidance materials for the interpretation of score reports.

8. The method of claim 1, further comprising:

if the targeted proficiency value does not match the effective proficiency value, determining whether an indication of invalidity exists.

9. The method of claim 8 wherein determining whether an indication of invalidity exists comprises:

determining that an indication of invalidity exists if, for an ancillary KSA, the profile value is less than the requirement.

10. The method of claim 8 wherein determining whether an indication of invalidity exists comprises:

determining that an indication of invalidity exists if, for a focal KSA, the requirement exceeds the focal value.

11. The method of claim 8 wherein determining whether an indication of invalidity exists comprises:

determining that an indication of invalidity exists if, for a focal KSA, the requirement is less than the focal value.

12. The method of claim 1 wherein the test taker profile comprises one or more profile values characteristic of a language learner.

13. The method of claim 1 wherein the test taker profile comprises one or more profile values characteristic of a test taker having a disability.

14. The method of claim 13 wherein the disability comprises one or more of deafness, blindness, dyslexia, low vision, and neuropathy.