Automated assessment development and associated methods

• US 20060099561A1
• Filed: 11/08/2005
• Published: 05/11/2006
• Est. Priority Date: 11/08/2004
• Status: Abandoned Application

• Current Assignee: Harcourt Assessment Incorporated (Pearson plc)
• Original Assignee: Harcourt Assessment Incorporated (Pearson plc)
• Inventors: Griph, Gerald W.
• Application Number:
  US11/269,120
• Publication Number:
  US 20060099561A1
• Time in Patent Office: Days
• US Class Current: 434/322
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

An automated system and method for creating a test form that conforms to predetermined content specifications (the blueprint) and has a desired statistical profile (target statistics). The algorithm divides the test-construction problem into two phases: a content-matching phase that is followed by a statistical-matching phase. In these phases, (1) a structural skeleton is sought that fits the blueprint; and (2) a search is made among the set of all possible tests that could be created from a pool of items for at least one test that matches the desired target statistics. The algorithm performs the statistical matching in the second phase without upsetting the content match that was attained in the first phase. This is possible because the content matching takes place using content specifications that match a number of different items or sets of items rather than specific items or sets of items.

Claims

1. A method of assembling an assessment having a predetermined target content structure and target statistics from a pool of passages, each passage having associated therewith a content profile and passage statistics, the method comprising the steps of:

a. selecting from the passage pool a predetermined number of content profiles to create a candidate form, the candidate form having a content structure based upon a sum of the content profiles;

b. comparing the candidate form content structure with the target content structure;

c. if a difference between the candidate form content structure and the target content structure is a currently lowest value, retaining the candidate form as a closest match;

d. repeating steps (a)-(c) until the retained closest match candidate matches the target content structure;

e. populating the retained closest match candidate form with a plurality of passages meeting the form content structure to create a potential assessment;

f. calculating test statistics for the potential assessment;

g. if a difference between the calculated test statistics and the target statistics is a currently lowest value, retaining the potential assessment as a candidate assessment; and

h. repeating steps (e)-(g) until all possible potential assessments have been created.

2. The method recited in claim 1, further comprising the step, prior to step (a), of creating at least one multiple-item passage in the pool from a plurality of items associated therewith.

3. The method recited in claim 2, further comprising the steps of:

providing a set of a first number of associated items;

determining a second number of items to be included in the multiple-item passage, the second number smaller than the first number; and

wherein the passage-creating step comprises creating all possible combinations of the first number of items to yield a multiple-item passage having the first number of items therein.

4. The method recited in claim 1, wherein the assessment further has a predetermined target information curve and each passage has associated therewith an information curve, and wherein step (a) comprises:

for each passage, calculating a difference between a peak of the target information curve and a peak of the passage information curve;

associating the distance with each passage content profile;

sorting the passage content profiles according to passage distances; and

selecting the content profiles preferentially according to smallest passage distances to create the candidate form.

5. The method recited in claim 1, wherein step (a) comprises:

for each different content profile, counting a number of passages that share a common content profile;

sorting the profiles according to number of shared passages; and

selecting the content profiles preferentially according to greatest number of shared passages to create the candidate form.

6. A method of assembling an assessment having a predetermined target content structure and target statistics from a pool of passages, each passage having associated therewith a content profile and passage statistics, the method comprising the steps of:

a. selecting from the passage pool a predetermined number of content profiles to create a candidate form, the candidate form having a content structure based upon a sum of the content profiles;

b. comparing the candidate form content structure with the target content structure;

c. if a difference between the candidate form content structure and the target content structure is a currently lowest value, retaining the candidate form as a closest match;

d. repeating steps (a)-(c) until the retained closest match candidate matches the target content structure;

e. randomly populating the retained closest match candidate form with a plurality of passages meeting the form content structure to create a potential assessment;

f. calculating test statistics for the potential assessment;

g. repeating steps (e) and (f) a plurality of times;

h. sorting the created potential assessments according to a distance between the respective calculated test statistics and the target statistics;

i. selecting a closest potential assessment from the sorting step;

j. searching among the potential assessments for a form complementary to the closest potential assessment;

k. forming a progeny assessment by randomly selecting a sequence of alternative passages from each of the closest potential assessment and the complementary assessment;

l. calculating test statistics for the progeny assessment;

m. repeating steps (k) and (l) a plurality of times; and

n. identifying a progeny assessment having a closest match to the target test statistics.

7. The method recited in claim 6, further comprising the step, prior to step (a), of creating at least one multiple-item passage in the pool from a plurality of items associated therewith.

8. The method recited in claim 7, further comprising the steps of:

providing a set of a first number of associated items;

determining a second number of items to be included in the multiple-item passage, the second number smaller than the first number; and

wherein the passage-creating step comprises creating all possible combinations of the first number of items to yield a multiple-item passage having the first number of items therein.

9. The method recited in claim 6, wherein the assessment further has a predetermined target information curve and each passage has associated therewith an information curve, and wherein step (a) comprises:

for each passage, calculating a difference between a peak of the target information curve and a peak of the passage information curve;

associating the distance with each passage content profile;

sorting the passage content profiles according to passage distances; and

selecting the content profiles preferentially according to smallest passage distances to create the candidate form.

10. The method recited in claim 6, wherein step (a) comprises:

for each different content profile, counting a number of passages that share a common content profile;

sorting the profiles according to number of shared passages; and

selecting the content profiles preferentially according to greatest number of shared passages to create the candidate form.

11. A computer-readable medium having stored thereon a software package for assembling an assessment having a predetermined target content structure and target statistics from a pool of passages, each passage having associated therewith a content profile and passage statistics, the software package comprising code segments adapted to:

a. select from the passage pool a predetermined number of content profiles to create a candidate form, the candidate form having a content structure based upon a sum of the content profiles;

b. compare the candidate form content structure with the target content structure;

c. if a difference between the candidate form content structure and the target content structure is a currently lowest value, retain the candidate form as a closest match;

d. repeat code segments (a)-(c) until the retained closest match candidate matches the target content structure;

e. populate the retained closest match candidate form with a plurality of passages meeting the form content structure to create a potential assessment;

f. calculate test statistics for the potential assessment;

g. if a difference between the calculated test statistics and the target statistics is a currently lowest value, retain the potential assessment as a candidate assessment; and

h. repeat code segments (e)-(g) until all possible potential assessments have been created.

12. The computer-readable medium recited in claim 11, wherein the software package further comprises a code segment adapted to, prior to code segment (a), create at least one multiple-item passage in the pool from a plurality of items associated therewith.

13. The computer-readable medium recited in claim 12, wherein the software package further comprises code segments adapted to:

provide a set of a first number of associated items;

determine a second number of items to be included in the multiple-item passage, the second number smaller than the first number; and

wherein the passage-creating code segment is adapted to create all possible combinations of the first number of items to yield a multiple-item passage having the first number of items therein.

14. The computer-readable medium recited in claim 11, wherein the assessment further has a predetermined target information curve and each passage has associated therewith an information curve, and wherein code segment (a) is adapted to:

for each passage, calculate a difference between a peak of the target information curve and a peak of the passage information curve;

associate the distance with each passage content profile;

sort the passage content profiles according to passage distances; and

select the content profiles preferentially according to smallest passage distances to create the candidate form.

15. The computer-readable medium recited in claim 11, wherein code segment (a) is adapted to:

for each different content profile, count a number of passages that share a common content profile;

sort the profiles according to number of shared passages; and

select the content profiles preferentially according to greatest number of shared passages to create the candidate form.

16. A computer-readable medium having stored thereon a software package for assembling an assessment having a predetermined target content structure and target statistics from a pool of passages, each passage having associated therewith a content profile and passage statistics, the software package having code segments adapted to:

a. select from the passage pool a predetermined number of content profiles to create a candidate form, the candidate form having a content structure based upon a sum of the content profiles;

b. compare the candidate form content structure with the target content structure;

c. if a difference between the candidate form content structure and the target content structure is a currently lowest value, retain the candidate form as a closest match;

d. repeat code segments (a)-(c) until the retained closest match candidate matches the target content structure;

e. randomly populate the retained closest match candidate form with a plurality of passages meeting the form content structure to create a potential assessment;

f. calculate test statistics for the potential assessment;

g. repeat code segments (e) and (f) a plurality of times;

h. sort the created potential assessments according to a distance between the respective calculated test statistics and the target statistics;

i. select a closest potential assessment from the sorting step;

j. search among the potential assessments for a form complementary to the closest potential assessment;

k. form a progeny assessment by randomly selecting a sequence of alternative passages from each of the closest potential assessment and the complementary assessment;

l. calculate test statistics for the progeny assessment;

m. repeat code segments (k) and (l) a plurality of times; and

n. identify a progeny assessment having a closest match to the target test statistics.

17. The computer-readable medium recited in claim 16, wherein the software package further comprises code segments adapted to, prior to code segment (a), create at least one multiple-item passage in the pool from a plurality of items associated therewith.

18. The computer-readable medium recited in claim 17, wherein the software package further comprises code segments adapted to:

provide a set of a first number of associated items;

determine a second number of items to be included in the multiple-item passage, the second number smaller than the first number; and

wherein the passage-creating code segment is adapted to create all possible combinations of the first number of items to yield a multiple-item passage having the first number of items therein.

19. The computer-readable medium recited in claim 16, wherein the assessment further has a predetermined target information curve and each passage has associated therewith an information curve, and wherein code segment (a) is adapted to:

for each passage, calculate a difference between a peak of the target information curve and a peak of the passage information curve;

associate the distance with each passage content profile;

sort the passage content profiles according to passage distances; and

select the content profiles preferentially according to smallest passage distances to create the candidate form.

20. The computer-readable medium recited in claim 16, wherein code segment (a) is adapted to:

for each different content profile, count a number of passages that share a common content profile;

sort the profiles according to number of shared passages; and

select the content profiles preferentially according to greatest number of shared passages to create the candidate form.