Use of a resource allocation engine in processing student responses to assessment items

• US 9,792,828 B2
• Filed: 02/10/2015
• Issued: 10/17/2017
• Est. Priority Date: 03/15/2007
• Status: Active Grant

• Current Assignee: McGraw Hill LLC
• Original Assignee: McGraw-Hill School Education, LLC
• Inventors: Poor, David Deas Sinkler, Fry, Randal B.
• Primary Examiner(s): HONG, THOMAS J
• Application Number:
  US14/618,048
• Publication Number:
  US 20150154879A1
• Time in Patent Office: 980 Days
• Field of Search: 434353-361, 382216
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: G09B 7/06 of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers

Abstract

A resource allocation can be utilized to control resources to determine trustworthy item scores for a respondent'"'"'s response to an assessment item. Embodiments may utilize one or more resources to extract the student response to an item, one or more resources to establish an item score for that item based on the extracted response, and one or more resources to evaluate the extracted or score and further utilize such resources until a trustworthy item score is established. Confidence indicators may be used for both digital representations of student responses and the scores derived from such digital representations of student responses.

Claims

1. A computerized method for processing a plurality of digital representations of responses to assessment items stored in a database to derive scores for the responses by at least one derived data extraction processing resources constructed and arranged to derive a score from a digital representation of a response, said method being performed on a resource allocation engine implemented on a computer and coupled to the database and further coupled to a plurality of said derived data extraction processing resources and to a plurality of data extraction processing resources, wherein said plurality of data extraction processing resources comprise two or more of:

(a) OMR extraction processing engines(s);

(b) OCR extraction processing engine(s);

(c) ICR extraction processing engine(s);

(d) handwriting extraction processing engine(s);

(e) voice-to-text extraction processing engine(s);

(f) icon location and identification extraction processing engine(s);

(g) voice transcription processing;

(h) key from image processing;

(i) coding from observing video; and

(j) scanning, and

said plurality of derived data extraction processing resources comprise two or more of;

(a) OMR set conversion to character or number engine(s);

(b) image enhancement engine(s);

(c) symantic analysis;

(d) letter/word matching or pattern matching;

(e) icon selection and location score assignment engine;

(f) image analysis mark location or pattern matching;

(g) scoring from recorded audio;

(h) scoring from recorded video; and

(i) audio video pattern matching engine,

said method comprising, for each response of the plurality of digital representations of responses to assessment items;

(Step

     1) receiving, by the resource allocation engine, a first digital representation of said response;

(Step

     2) storing, by the resource allocation engine, the first digital representation of said response to a set of digital representations in a memory of the computer;

(Step

     3) determining, by the resource allocation engine, whether said of the plurality of derived data extraction processing resources is suitable to extract a score based on said set of digital representations;

(Step 3a) if it is determined that the one of said plurality of derived data extraction processing resources is suitable to extract a score based on said set of digital representations, proceeding to (Step

     4);

(Step 3b) if it is determined that the one of said plurality of derived data extraction processing resources is not suitable to extract a score based on said set of digital representations, performing the following sub-steps (i) to (iv);

(i) identifying, by the resource allocation engine, a digital representation from said set of digital representations and a data extraction processing resource from said plurality of data extraction processing resources to convert said identified digital representation to a different format;

(ii) using, by the resource allocation engine, said identified data extraction processing resource to convert said identified digital representation to a second digital representation of said response different from the first digital representation of said response;

(iii) storing, by the resource allocation engine, said second digital representation of said response to said set of digital representations in the memory; and

(iv) proceeding to (Step

     3);

(Step

     4) selecting, by the resource allocation engine, one of said plurality of derived data extraction resources by comparing the characteristics of at least one digital representation from said set of digital representations to the characteristics of digital representations scoreable by each of said plurality of derived data extraction resources;

(Step

     5) generating, by the resource allocation engine, a first score for said response by submitting the at least one digital representation from said set of digital representations to said selected derived data extraction resource;

(Step

     6) storing, by the resource allocation engine, the first score to a set of scores in the memory;

(Step

     7) determining, by the resource allocation engine, whether a final score can be determined from said set of scores;

(Step 7a) if it is determined that the final score can be determined from said set of scores, proceeding to (Step

     8);

(Step 7a) if it is determined that the final score cannot be determined from said set of scores, proceeding to (Step

     3); and

(Step

     8) determining the final score from said set of scores, wherein

selecting one of said plurality of derived data extraction resources in (step

     4) further comprises one or more of;

(a) comparing the characteristics of at least one digital representation from said set of digital representations to predetermined evaluation criteria;

(b) calculating dynamic criteria and then comparing the characteristics of at least one digital representation from said set of digital representations to the dynamic criteria; and

(c) comparing the characteristics of at least one digital representation from said set of digital representations to one or more other sets of extracted data, wherein the one or more other sets of extracted data include one of a previous extraction from (Step 3b) and a previous score from (Step

     5).

2. The method of claim 1, in which a digital representation from said plurality of digital representations of responses comprises at least one of:

an image of a section of an assessment document including on which a respondent has made a mark made to represent the response,

an image of a section of the assessment document including on which the respondent has applied a sticker applied for a constrained constructed response item,

a numeric value representing the darkness of a mark at a specified OMR position on the assessment document,

an alphanumeric value representing the respondent a selection of one of a set of specified OMR positions on the assessment document,

an image of a page within the assessment document,

an alphanumeric representation of a keystroke,

a collection of characters representing a series of keystrokes representing a word, a sentence, or an essay,

a digital representation of stylus movements,

an audio file, and

a video file.