Constructed response scoring mechanism
1. A method for scoring user responses to a test question, comprising:
- receiving a user response wherein the response is a collection of objects with one or more attributes;
binding one or more variables identified in a binding tree based on the collection of objects;
evaluating at least one named assertion associated with an element;
assigning a value to each node of the binding tree based on the named assertion;
storing the named assertion; and
scoring the binding tree based on each value of the nodes, wherein the scoring determines the range of scores.
A system, method, and related techniques are disclosed for scoring user responses to constructed response test items. The system includes a scoring engine for receiving a user response to a test question and evaluating the response against a scoring rubric. The scoring rubric may include a binding stage, an assertion stage, and a scoring stage. Furthermore, the system includes a database for referencing elements used by the scoring engine which may comprise objects, object sets, attributes of objects, and transformations of any elements.
|CONSOLIDATED ON-LINE ASSESSMENT SYSTEM|
Patent #US 20070105084A1
Current AssigneeEducational Testing Service
Sponsoring EntityEducational Testing Service
|System for obtaining and integrating essay scoring from multiple sources|
Patent #US 20070218450A1
Current AssigneeVANTAGE TECHNOLOGIES KNOWLEDGE ASSESSMENT L.L.C.
Sponsoring EntityVANTAGE TECHNOLOGIES KNOWLEDGE ASSESSMENT L.L.C.
|System and method for interactive scoring of standardized test responses|
Patent #US 6,120,299 A
Current AssigneeEducational Testing Service
Sponsoring EntityEducational Testing Service
- 1. A method for scoring user responses to a test question, comprising:
receiving a user response wherein the response is a collection of objects with one or more attributes; binding one or more variables identified in a binding tree based on the collection of objects; evaluating at least one named assertion associated with an element; assigning a value to each node of the binding tree based on the named assertion; storing the named assertion; and scoring the binding tree based on each value of the nodes, wherein the scoring determines the range of scores.
- View Dependent Claims (2, 3)
This application is a Divisional of U.S. application Ser. No. 12/320,631, filed Jan. 30, 2009, which claims the benefit of U.S. Provisional Application No. 61/193,252, filed Nov. 12, 2008 and titled “Constructed Response Scoring Mechanism,” each of which application is incorporated by reference in its entirety for all purposes.
Learning often happens incrementally. At first students may be able to recall, recognize or name concepts. As mastery increases, they may be able describe concepts, the properties of concepts, or relationships among concepts. Eventually, students may be able to apply concepts to novel situations, use learned material to generate new insights, or synthesize learned material. This learning sequence is often referred to as “depth of knowledge,” and refers to the depth with which students understand the material that they are taught. The specific stages and levels of depth vary across taxonomies, but the general idea is that knowledge becomes deeper and more internalized with additional mastery, and that in turn allows more robust application of the knowledge.
When assessing student mastery, it is often desirable to evaluate their depth of knowledge. From the perspective of test developers it can be quite difficult to develop selected response items (test questions) that measure deeper levels of knowledge. A selected response item is a test question, such as a multiple choice question, in which the correct answer is selected from a collection of choices.
Many testing programs use constructed response items to measure content at deeper levels of knowledge. A constructed response item is an item that does not offer the examinee answer options from which to choose, but rather the examinee must construct a response.
In a typical system, each student'"'"'s response is evaluated against a scoring rubric, which describes the characteristics of a response that should receive full credit. When partial credit is to be awarded, the characteristics of responses that receive some portion of the total overall score are also enumerated. For example, an item might award three points for full credit, and individually enumerate characteristics of imperfect responses that would warrant the award of two points, one point, and zero points.
The scoring rubric usually goes through a refinement process called rangefinding. In this process, samples of student responses (usually from a field test) are evaluated by a committee of subject matter experts with the goal of selecting sample responses exemplifying each score point to be awarded. It is not uncommon for the scoring rubrics to be refined during this process.
Using the refined rubric, human scorers apply the scoring criteria to score each examinee'"'"'s response to the item. Typically, this process is monitored and managed, giving each scorer a number of pre-scored papers to evaluate whether they continue to apply the rubric correctly, and having a proportion of scored papers independently scored by a second scorer to monitor the reliability with which scorers apply the rubric.
The current process has several limitations. First, it is very expensive to score constructed response items by hand, requiring that each response be read by one or more qualified scorers. Furthermore, the process by which scoring rubrics are refined does not offer an opportunity for large-scale evaluation of the consequences of the refinements, risking potential unintended consequences. Additionally, the process necessarily takes time, limiting the usefulness of constructed response items in online tests. For example, adaptive online tests use the scores on items administered early in the test to select the best items to administer later. Due to current limitations, human scoring prevents using constructed response items to support adaptive testing.
Presented is an invention directed towards systems and methods that improve the current process for administering and scoring constructed response items. These systems and methods, used separately or together, allow for the immediate score of constructed response items. The invention has the practical benefits of reducing costs of administering constructed response items, providing more reliable scoring constructed response items, broadly validating constructed response scoring rubrics, and allowing for the integration of constructed response items into computerized adaptive tests delivered online or at testing centers.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Illustrative embodiments of the present invention are described in detail below with reference to the attached drawing figures, which are incorporated by reference herein and wherein:
As one skilled in the art will appreciate, embodiments of the present invention may be embodied as, among other things: a method, system, or computer-program product. Accordingly, the embodiments may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware. In one embodiment, the present invention takes the form of a computer-program product that includes computer-useable instructions embodied on one or more computer-readable media.
Computer-readable media include both volatile and nonvolatile media, removable and nonremovable media, and contemplates media readable by a database, a switch, and various other network devices. Network switches, routers, and related components are conventional in nature, as are means of communicating with the same. By way of example, and not limitation, computer-readable media comprise computer-storage media and communications media.
Computer-storage media, or machine-readable media, include media implemented in any method or technology for storing information. Examples of stored information include computer-useable instructions, data structures, program modules, and other data representations. Computer-storage media include, but are not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile discs (DVD), holographic media or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage, and other magnetic storage devices. These memory components can store data momentarily, temporarily, or permanently.
Communications media typically store computer-useable instructions—including data structures and program modules—in a modulated data signal. The term “modulated data signal” refers to a propagated signal that has one or more of its characteristics set or changed to encode information in the signal. An exemplary modulated data signal includes a carrier wave or other transport mechanism. Communications media include any information-delivery media. By way of example but not limitation, communications media include wired media, such as a wired network or direct-wired connection, and wireless media such as acoustic, infrared, radio, microwave, spread-spectrum, and other wireless media technologies. Combinations of the above are included within the scope of computer-readable media.
Client 102 may be or can include a desktop computer, a laptop computer or other mobile computing device, a network-enabled cellular telephone (with or without media capturing/playback capabilities), a server, a wireless email client, or other client, machine or device, or any combination of the above, to perform various tasks including Web browsing, search, electronic mail (email) and other tasks, applications and functions. Client 102 may additionally be any portable media device such as digital still camera devices, digital video cameras (with or without still image capture functionality), media players such as personal music players and personal video players, and any other portable media device, or any combination of the above.
Scoring manager 104 is utilized for administering and scoring constructed response items for a user. The scoring manager 104 may also be utilized to generate individual expert systems to represent the scoring knowledge for a single constructed response item. The scoring manager 104 additionally may be configured to refine each expert system and test it against a broad range of student responses. In an embodiment, scoring manager 104 is a server external to client 102. In another embodiment, scoring manger 104 may be an application that resides and is executable within client 102.
As shown, scoring engine 106 and primitive library 108 are components that reside within scoring manager 104. In other embodiments, one or more of the scoring engine 106 and primitive library 108 may be external to the scoring manager 104. The scoring engine 106 is a component that receives a user'"'"'s response to a question and evaluates the response against a scoring rubric. The scoring engine 106 may include or have access to the library of primitives 108. In an embodiment, the primitive library 108 may include the calculation of distances, slopes, comparisons of strings and numbers, and other basic operations. In order to make the scoring engine 106 general so that it can support a very large range of items, the primitive library 108 may be low-level and higher order predicates may be created from the primitive library 108. In other embodiments, complex predicates may be added to the primitive library. In an embodiment, in using the primitive library 108, the language for representing a scoring rubric may enable the library functions to reference elements including, but not limited to, object sets, objects, attributes of objects, as well as transformations of any of these elements.
The UI 200 may be used to ensure that user responses are collected with a consistent mechanism that creates and transmits a data structure to a scoring engine. A user response may comprise a set of objects, each of which may have one or more attributes. For example, the UI can produce a collection of objects that may include points, line segments connecting points, geometric objects comprised of connected line segments, and user-defined atomic objects, such as the weights 202 on the left palette in
In another embodiment, the UI 200 can be configured to capture natural language where the object set may include elements of a semantic network derived from a parse of the text provided by the user. Alternatively, the UI 200 can be configured to capture input from an equation editor representing sequences of symbols as the initial set of objects. Moreover, in other embodiments, an application to test proficiency with a computer program may capture menu commands, keyboard input, or mouse events as the set of objects. However, this list is intended to be exemplary rather than exhaustive.
In an embodiment, a scoring rubric may be defined in three sequential stages: a binding stage, in which references to elements are established; an assertion stage, in which assertions about elements are evaluated and stored; and a scoring stage, in which a score is assigned based on the values of the results of the assertions. XML-based language may be used for implementing these stages for the UI responses.
An assertion is a predicate that is either true or false. The assertion further is an atomic unit from which scoring rubrics can be built. Each assertion can be named for later reference in the scoring stage.
In the scoring stage, named assertions are collected in a set of And-Or trees, one tree for each numeric score point. An exemplary snippet from a scoring specification for a three-point item appears in
The representation of annotated And-Or trees is well known in the computer science art. In an embodiment, the internal representation used is a set of nodes, in which each node has a list of children, each of which can be an And node, an Or node, or an assertion node. The resulting internal representation of the binding, assertion, and scoring trees comprises an Answer Set that includes an expert system embodying the knowledge of the scoring rubric for a particular item. The scoring rubric may be written directly in the specification language or authoring tools may be developed to help test developers specify the rubrics. In some embodiments, tools may be domain specific.
At operation 602, a user response is captured as a collection of objects with attributes. In an embodiment, the response is captured through a UI such as UI 200 (
The disclosed invention also presents an enhanced method of “rangefinding” which refines expert systems and tests them against a broad range of student responses. Rangefinding is a committee process in which subject-matter experts agree on appropriate scores for sample examinee responses. During rangefinding, a small sample of items, often in the range of 25-100, are reviewed by committees to test the application of the scoring rubrics. During this process, refinements are made to the rubric, and sample papers are selected to train scoring staff on the accurate scoring of responses to the item.
However, improvements are needed for enhancing the rangefinding process. The invention provides such improvements. For example, through the invention, decisions of the rangefinding committee can be expressed formally as assertions in the language used to define the scoring rubrics. Formalizing the committee results as a series of explicit rules improves the accuracy of scoring, and would likely lead to more reliable scoring even when scoring is done by human scorers. Furthermore, committee decisions can be systematically tested against the full set of field-test data to locate unintended consequences of the proposed new rules.
At operation 706, items and corresponding scores are provided to the rangefinding committee. In an embodiment, the rangefinding committee is trained in the formal specifications of the scoring rubric. In instances where the committee reaches a consensus that a score is incorrect, at operation 708, one or more rules or principles are identified that differentiates the correct score from the incorrect scores. At operation 710, a modification to the scoring rubric, corresponding to the indentified rules, is provided.
At operation 712, the identified rules for modifying the scoring rubric are applied to field test responses in order to identify any unintended consequences of the new rules. In an embodiment, this may be done by identifying scores that changed under the new rules and evaluating those changes. At operation 714, a consensus on whether to fully implement the new rules is achieved based on the modification to the formal scoring rubric. In an embodiment, the consensus is achieved after the committee reviews a new sample of responses for which the revision resulted in a change of scores and determines that the changes are limited to those intended.
While particular embodiments of the invention have been illustrated and described in detail herein, it should be understood that various changes and modifications might be made to the invention without departing from the scope and intent of the invention. The embodiments described herein are intended in all respects to be illustrative rather than restrictive. Alternate embodiments will become apparent to those skilled in the art to which the present invention pertains without departing from its scope.
From the foregoing it will be seen that this invention is one well adapted to attain all the ends and objects set forth above, together with other advantages, which are obvious and inherent to the system and method. It will be understood that certain features and sub-combinations are of utility and may be employed without reference to other features and sub-combinations. This is contemplated and within the scope of the appended claims.