USE OF A RESOURCE ALLOCATION ENGINE IN PROCESSING STUDENT RESPONSES TO ASSESSMENT ITEMS

• US 20150154879A1
• Filed: 02/10/2015
• Published: 06/04/2015
• Est. Priority Date: 03/15/2007
• Status: Active Grant

• Current Assignee: McGraw Hill LLC
• Original Assignee: McGraw-Hill School Education, LLC
• Inventors: POOR, DAVID Deas Sinkler, FRY, RANDAL B.
• Granted Patent: US 9,792,828 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/06 of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers

Abstract

A resource allocation can be utilized to control resources to determine trustworthy item scores for a respondent'"'"'s response to an assessment item. Embodiments may utilize one or more resources to extract the student response to an item, one or more resources to establish an item score for that item based on the extracted response, and one or more resources to evaluate the extracted or score and further utilize such resources until a trustworthy item score is established. Confidence indicators may be used for both digital representations of student responses and the scores derived from such digital representations of student responses.

Claims

1. A computerized method for processing a plurality of digital representations of responses to assessment items to derive scores for the responses by one or more derived data extraction processing resources constructed and arranged to derive a score from a digital representation of a response, said method comprising:

A for each response, using a programmed computer to perform a first determination of whether the format of the current digital representation of the response is suitable for processing by at least one of the derived data extraction processing resources, or if the format of the digital representation of the response is not suitable for processing by at least one of the derived data extraction processing resources, using the programmed computer to;

(1) identify a data extraction processing resource to convert the format of the digital representation of the response to a different format; and

(2) submit the digital representation of the response to the data extraction processing resource identified to generate a current extraction of the digital representation of the response;

B for each response, using a programmed computer to compare the characteristics of the current digital representation of the response or the current extraction of the digital representation of the response to characteristics of digital representations that can be scored by each available derived data extraction processing resource to identify a derived data extraction processing resource that is appropriate for processing either the current digital representation of the response or the current extraction of the digital representation of the response, and to submit the current digital representation or the current extraction to the derived data extraction processing resource identified to generate a current derived extraction; and

C for each response, using a programmed computer to perform a second determination of whether there is an acceptable final score by submitting the current derived extraction to an evaluation resource configured to determine a final score for the response and, if an acceptable final score is returned by the evaluation resource, storing such final score for the response.

2. The method of claim 1, further comprising:

after step A, determining if no current extraction is returned by the extraction resource or if the current extraction returned by the data extraction processing resource is not reliable; and

repeating step A with a different data extraction processing resource if no current extraction is returned or if the current extraction returned is not reliable until a reliable current extraction is returned.

3. The method of claim 1, further comprising:

after step B, determining if no current derived extraction is returned by the derived data extraction processing resource or if the current derived extraction returned by the derived data extraction processing resource is not reliable; and

repeating step B with a different derived data extraction processing resource if no current derived extraction is returned or if the current derived extraction returned is not reliable until a reliable current derived extraction is returned.

4. The method of claim 1, further comprising:

after step C, determining if no derived extraction that is acceptable as a final score is returned by the derived data extraction processing resource; and

repeating step A with a different data extraction processing resource and step B with a different derived data extraction processing resource if no derived extraction that is acceptable as a final score is returned by the derived data extraction processing resource.

5. The method of claim 1, wherein the data extraction processing resource(s) comprise one or more of:

(a) OMR extraction processing engines(s);

(b) OCR extraction processing engine(s);

(c) ICR extraction processing engine(s);

(d) handwriting extraction processing engine(s);

(e) voice-to-text extraction processing engine(s);

(f) icon location and identification extraction processing engine(s);

(g) voice transcription processing;

(h) key from image processing;

(i) coding from observing video; and

(j) scanning.

6. The system of claim 1, wherein the derived data extraction processing resource(s) comprise one or more of:

(a) OMR set conversion to character or number engine(s);

(b) human review/resolution from captured image;

(c) image enhancement engine(s);

(d) symantic analysis;

(e) letter/word matching or pattern matching;

(f) icon selection and location score assignment engine;

(g) image analysis mark location or pattern matching;

(h) human reader/evaluator scoring;

(i) scoring from recorded audio;

(j) scoring from recorded video; and

(k) audio video pattern matching engine.

7. The method of claim 1, wherein step B comprises extracted by an extraction processing resource by one or more of:

(a) comparing the characteristics of the current digital representation of the response or the current extraction of the digital representation of the response against predetermined evaluation criteria;

(b) calculating dynamic criteria and then comparing the characteristics of the current digital representation of the response or the current extraction of the digital representation of the response against the dynamic criteria; and

(c) comparing the characteristics of the current digital representation of the response or the current extraction of the digital representation of the response with one or more other sets of extracted data.

8. The method of claim 1, in which the digital representation of the response comprises at least one of:

an image of a section of an assessment document on which the respondent has made a mark to represent the response,

an image of a section of an assessment document on which the respondent has applied a sticker for a constrained constructed response item,

a numeric value representing the darkness of a mark at a specified OMR position on the assessment document,

an alphanumeric value representing the respondent selection of one of a set of specified OMR positions on the assessment document,

an image of a page within the assessment document,

an alphanumeric representation of a keystroke,

a collection of characters representing a series of keystrokes representing a word, a sentence, or an essay,

a digital representation of stylus movements,

an audio file, and

a video file.

9. The method of claim 1, wherein step A further comprises generating a confidence indicator for the current extraction of the digital representation of the response.

10. The method of claim 1, wherein step A further comprises generating a confidence indicator for the current derived extraction.