ASSESSMENT-DRIVEN COGNITION SYSTEM

• US 20100068687A1
• Filed: 03/18/2009
• Published: 03/18/2010
• Est. Priority Date: 03/18/2008
• Status: Active Grant

• Current Assignee: Jones/Ncti Inc.
• Original Assignee: Jones International
• Inventors: Bertelsen, Jim
• Granted Patent: US 8,385,812 B2
• Time in Patent Office: Days
• US Class Current: 434/322
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A computer-based assessment driven cognition system includes, in embodiments, a generator configured to present questions, where each question presented is associated with a stacking weight and has at least one correct response, and a question property evaluator configured to, for each question presented, determine a likelihood of randomly selecting the at least one correct response. The system also includes a question response receiver configured to receive at least one response to each question presented. Additionally, the system includes a question response evaluator configured to determine, for each question presented, when the at least one response is the at least one associated correct response, a likelihood that the at least one response is a false positive. The question response evaluator is further configured to adjust the stacking weight of each question presented depending at least upon the likelihood that the at least one response to the question is a false positive.

Claims

1. A computer-based system for assessment driven learning, the system comprising:

a generator configured to present questions received from a processor to a question presentation device utilizing a question module and a question database having a plurality of questions,

wherein the plurality of questions are associated with a stacking weight,

wherein the plurality of questions are ordered according to the stacking weight of each question, and wherein each question presented includes at least one answer option and at least one associated correct response;

a question property evaluator configured to utilize the question module to determine a likelihood of randomly selecting the at least one associated correct response,

wherein the question property evaluator considers, for each question presented, at least the number of answer options and a number of answer options a user is instructed to select;

a question response receiver configured to receive, from a device associated with the user, at least one response to each question presented utilizing a communications module; and

a question response evaluator configured to, utilizing an evaluation module and a user response history database for each question presented;

determine, when the at least one response is the at least one associated correct response, a likelihood that the at least one response is a false positive using the question property evaluator, a response history of the user, and a response history of one or more other users, and

adjust the stacking weight of each question presented within the question database depending at least upon the likelihood that the at least one response to the question is a false positive.

2. The computer-based system according to claim 1, wherein for each question presented,

the question property evaluator is further configured to, utilizing the question module and question database, determine the likelihood of randomly selecting the at least one associated correct response for each question presented using

an inverse of a number of combinations of answer options the user is instructed to select from a set including the number of answer options when the number of answer options the user is instructed to select is less than the number of answer options, and

an inverse of two raised to a power of the number of answer options when the number of answer options the user is instructed to select is greater than or equal to the number of answer options, and

the question response evaluator is further configured to, utilizing the evaluation module and the user response history database, upon determination that the at least one response is the at least one associated correct response, increase the likelihood that the at least one response is a false positive upon an increase in the likelihood of randomly selecting the at least one correct response.

3. The computer-based system according to claim 1, wherein the question response evaluator is further configured to, utilizing the evaluation module and the user response history database for each question presented,

increase the likelihood that the at least one response is a false positive upon an indication from the user response history that the at least one response is not the at least one associated correct response consecutively one or more times, and

decrease the likelihood that the at least one response is a false positive upon an indication from the user response history that the at least one response is the at least one associated correct response consecutively one or more times.

4. The computer-based system according to claim 1, wherein for each question presented,

the question property evaluator is further configured to, utilizing the question module and question database,

determine an amount of time the user takes to submit the at least one associated correct response, and

determine, from the response history of the one or more other users the amount of time the user takes to submit the at least one associated correct response, an average response time of the one or more other users and the user to submit the at least one associated correct response, and

the question response evaluator is further configured to, utilizing the evaluation module and question database, increase the likelihood that the at least one response is a false positive upon determination that the amount of time the user takes to submit the at least one associated correct response is at least one standard deviation below the average response time of the one or more other users and the user to answer to submit the at least one associated correct response.

5. The computer-based system according to claim 4, wherein for each question presented, upon determination that the at least one response is not the at least one associated correct response and the user response history indicates that the user previously submitted the at least one associated correct response,

the question property evaluator is further configured to, utilizing the question module and the user response history database, remove an amount of time the user took to previously submit the at least one associated correct response from the average response time of the one or more other users and the user to submit the at least one associated correct response.

6. The computer-based system according to claim 1, wherein upon determination that the user is permitted to report a level of confidence for each question presented, the question response evaluator is further configured to, utilizing the evaluation module, set the likelihood that the at least one response is a false positive to a value representative of a definite false positive when the reported level of confidence is less than a certainty threshold.

7. The computer-based system according to claim 1, wherein upon determination that the user is permitted to report a level of confidence in the at least one response for each question presented, the question response evaluator is further configured to, utilizing the evaluation module and user response history database,

determine a first certainty weight that decreases the stacking weight upon determination that the reported level of confidence is higher than a certainty threshold and upon an indication from the user response history that the at least one response is the at least one associated correct response consecutively one or more times,

determine a second certainty weight that increases the stacking weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the user response history that the at least one response is the at least one associated correct response consecutively one or more times,

determine a third certainty weight that increases the stacking weight by an amount higher than the second certainty weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the user response history that the at least one response is not the at least one associated correct response consecutively one or more times, and

determine a fourth certainty weight that increases the stacking weight by an amount higher than the third certainty weight upon determination that the reported level of confidence is higher than the certainty threshold and upon an indication from the user response history that the at least one response is not the at least one associated correct response consecutively one or more times.

8. The computer-based system according to claim 1, wherein for each question presented,

the question property evaluator is further configured to, utilizing the question module and question database, determine a likelihood of selecting one or more incorrect options,

wherein the question property evaluator considers at least a percentage of a number of times that the one or more incorrect options are expected to be selected and a percentage of a number of times that the one or more incorrect options are selected, and

the question response evaluator is further configured to, utilizing the evaluation module, increase the likelihood that the at least one response is a false positive upon a decrease in the likelihood of selecting the one or more incorrect options.

9. A computer-implemented method for assessment driven learning, the method comprising:

assigning a question stack to a user, wherein the question stack includes a plurality of questions;

retrieving the question stack from a question database;

retrieving, by a question property evaluator from the question database, one or more question properties associated with each question presented on a question presentation device,

wherein each question presented includes at least one answer option and at least on associated correct response,

wherein the one or more question properties indicate at least the number of answer options and a number of answer options the user is instructed to select;

receiving, utilizing a communications module, at least one response to each question presented from a device associated with the user;

estimating, by the question property evaluator utilizing a question module for each question presented, a likelihood of randomly selecting the at least one correct response using at least the number of answer options and the number of answer options the user is instructed to select; and

adjusting, by the question response evaluator utilizing an evaluation module and a user response history database, an associated stacking weight of each question presented depending at least upon a likelihood that the at least one response to the question is a false positive,

wherein the plurality of questions are ordered according to the associated stacking weight of each question presented,

wherein, for each question presented, when the at least one response to the question is the at least one associated correct response, the question property evaluator estimates the likelihood that the at least one response is a false positive using the question property evaluator, a response history of the user, and a response history of one or more other users.

10. The computer-implemented method according to claim 9, further comprising:

estimating, by the question property evaluator for each question presented, the likelihood of randomly selecting the at least one associated correct response using

an inverse of a number of combinations of answer options the user is instructed to select from a set of the number of answer options when the number of answer options the user is instructed to select is less than the number of answer options, and

an inverse of two raised to a power of the number of answer options when the number of answer options the user is instructed to select is greater than or equal to the number of answer options; and

establishing, by the question response evaluator for each question presented upon determination that the at least one response is the at least one correct response, a direct proportional relationship between the likelihood that the at least one response is a false positive and the likelihood of randomly selecting the at least one correct response,

wherein, the question response evaluator increases the likelihood that the at least one response is a false positive when the likelihood of randomly selecting the at least one correct response increases.

11. The computer-implemented method of claim 9, further comprising:

retrieving, for each question presented, the user response history from the user response history database;

comparing, by the question response evaluator for each question presented, the at least one response to the retrieved user response history database,

wherein the question property evaluator increases the likelihood that the at least one response is a false positive upon an indication from the retrieved user response history that the at least one response is not the at least one associated correct response consecutively one or more times, and

wherein the question property evaluator decreases the likelihood that the at least one response is a false positive upon an indication from the user response history that the at least one response is the at least one associated correct response consecutively one or more times.

12. The computer-implemented method of claim 9, further comprising:

retrieving, by the question property evaluator from the question database for each question presented, an amount of time the user takes to submit the at least one associated correct response and an average response time of the one or more other users to submit the at least one associated correct response;

adjusting, by the question property evaluator for each question presented, the average response time of the one or more other users to submit the at least one associated correct response to include that amount of time the user takes to submit the at least one associated correct response; and

increasing, by the question response property evaluator for each question presented, the likelihood that the at least one response is a false positive upon determination that the amount of time the user takes to submit the at least one associated correct response is at least one standard deviation below the adjusted average response time to submit the at least one associated correct response.

13. The computer-implemented method of claim 12, further comprising:

retrieving, by the question property evaluator for each question presented upon determination that the at least one response is not the at least one associated response, the user response history from the user response history database; and

adjusting, by the question property evaluator utilizing the question module, the average response time of the one or more other users to submit the at least one associated correct response upon indication from the retrieved user response history that the user previously submitted the at least one associated correct response,

wherein the question property evaluator removes an amount of time the user took to previously submit the at least one associated correct response from the average response time of the one or more other users to submit the at least one associated correct response.

14. The computer-implemented method of claim 9, further comprising:

receiving, from the device associated with the user, a reported level of confidence for each question presented,

adjusting, by the question response evaluator for each question presented upon determination that the received reported level of confidence is less than a certainty threshold, the likelihood that the at least one response to the question is a false positive to a value representative of a definite false positive.

15. The computer-implemented method according to claim 9, further comprising:

receiving, from the device associated with the user, a reported level of confidence for each question presented;

retrieving, by the question response evaluator, the user response history from the user response history database; and

adjusting, by the question response evaluator for each question presented, the stacking weight using the received reported level of confidence, wherein

the question response evaluator decreases the stacking weight by a first certainty weight upon determination that the received reported level of confidence is higher than a certainty threshold and upon an indication from the retrieved user response history that the at least one response is the at least one associated correct response one or more times,

the question response evaluator increases the stacking weight by a second certainty weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the retrieved user response history that the at least one response is the at least one associated correct response consecutively one or more times,

the question response evaluator increases the stacking weight by a third certainty weight that is an amount higher than the second certainty weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the retrieved user response history that the at least one response is not the at least one associated correct response consecutively one or more times, and

the question response evaluator increases the stacking weight by an amount higher than the third certainty weight upon determination that the reported level of confidence is higher than the certainty threshold and upon an indication from the retrieved user response history that the at least one response is not the at least one associated correct response consecutively one or more times.

16. The computer implemented method according to claim 9, further comprising:

estimating, by the question property evaluator for each question presented, a likelihood of selecting one or more incorrect options,

wherein the question property evaluator considers at least a percentage of a number of times that the one or more incorrect options are expected to be selected and a percentage of a number of times that the one or more incorrect options are selected; and

establishing an inverse relationship between the likelihood that the at least one response is a false positive and the likelihood of selecting the one or more incorrect options,

wherein the question property evaluator increases the likelihood that the at least one response is false positive when the likelihood of selecting the one or more incorrect options decreases.

17. A computer-readable medium having instructions stored thereon, the instructions configured to cause the one or more processors to instruct:

a generator to present questions received from a processor to a question presentation device utilizing a question module and a question database having a plurality of questions,

wherein the plurality of questions are associated with a stacking weight,

wherein the plurality of questions are ordered according to the stacking weight of each question, and wherein each question presented includes at least one answer option and at least one associated correct response;

a question property evaluator to utilize the question module to determine a likelihood of randomly selecting the at least one associated correct response,

wherein the question property evaluator considers, for each question presented, at least the number of answer options and a number of answer options a user is instructed to select;

a question response receiver to receive, from a device associated with the user, at least one response to each question presented utilizing a communications module; and

a question response evaluator to, utilizing an evaluation module and a user response history database for each question presented;

determine, when the at least one response is the at least one associated correct response, a likelihood that the at least one response is a false positive using the question property evaluator, a response history of the user, and a response history of one or more other users, and

adjust the stacking weight of each question presented within the question database depending at least upon the likelihood that the at least one response to the question is a false positive.

18. The computer-readable medium according to claim 17, wherein the instructions are further configured to cause the one or more processors to instruct:

the question property evaluator to, utilizing the question module and question database for each question presented, determine the likelihood of randomly selecting the at least one associated correct response for each question presented using

an inverse of a number of combinations of answer options the user is instructed to select from a set including the number of answer options when the number of answer options the user is instructed to select is less than the number of answer options, and

an inverse of two raised to a power of the number of answer options when the number of answer options the user is instructed to select is greater than or equal to the number of answer options, and

the question response evaluator to, utilizing the evaluation module and the user response history database for each question presented, upon determination that the at least one response is the at least one associated correct response, increase the likelihood that the at least one response is a false positive upon an increase in the likelihood of randomly selecting the at least one correct response.

19. The computer-readable medium according to claim 17, wherein the instructions are further configured to cause the one or more processors to instruct:

the question response evaluator to, utilizing the evaluation module and the user response history database for each question presented,

increase the likelihood that the at least one response is a false positive upon an indication from the user response history that the at least one response is not the at least one associated correct response consecutively one or more times, and

decrease the likelihood that the at least one response is a false positive upon an indication from the user response history that the at least one response is the at least one associated correct response consecutively one or more times.

20. The computer-readable medium according to claim 17, wherein the instructions are further configured to cause the one or more processors to instruct:

the question property evaluator to, utilizing the question module and question database for each question presented,

determine an amount of time the user takes to submit the at least one associated correct response, and

determine, from the response history of the one or more other users the amount of time the user takes to submit the at least one associated correct response, an average response time of the one or more other users and the user to submit the at least one associated correct response, and

the question response evaluator to, utilizing the evaluation module and question database for each question presented, increase the likelihood that the at least one response is a false positive upon determination that the amount of time the user takes to submit the at least one associated correct response is at least one standard deviation below the average response time of the one or more other users and the user to answer to submit the at least one associated correct response.

21. The computer-readable medium according to claim 20, wherein the instructions are further configured to cause the one or more processors to instruct:

the question property evaluator to, utilizing the question module and the user response history database for each question presented, remove an amount of time the user took to previously submit the at least one associated correct response from the average response time of the one or more other users and the user to submit the at least one associated correct response upon determination that the at least one response is not the at least one associated correct response and the user response history indicates that the user previously submitted the at least one associated correct response.

22. The computer-readable medium according to claim 17, wherein the instructions are further configured to cause the one or more processors to instruct:

the question response evaluator to, utilizing the evaluation module upon determination that the user is permitted to report a level of confidence for each question presented, set the likelihood that the at least one response is a false positive to a value representative of a definite false positive when the reported level of confidence is less than a certainty threshold.

23. The computer-readable medium according to claim 17, wherein the instructions are further configured to cause the one or more processors to instruct:

the question response evaluator to, utilizing the evaluation module and user response history database upon determination that the user is permitted to report a level of confidence for each question presented,

determine a first certainty weight that decreases the stacking weight upon determination that the reported level of confidence is higher than a certainty threshold and upon an indication from the user response history that the at least one response is the at least on associated correct response consecutively one or more times,

determine a second certainty weight that increases the stacking weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the user response history that the at least one response is the at least on associated correct response consecutively one or more times,

determine a third certainty weight that increases the stacking weight by an amount higher than the second certainty weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the user response history that the at least one response is not the at least on associated correct response consecutively one or more times, and

determine a fourth certainty weight that increases the stacking weight by an amount higher than the third certainty weight upon determination that the reported level of confidence is higher than the certainty threshold and upon an indication from the user response history that the at least one response is not the at least on associated correct response consecutively one or more times.

24. The computer-readable medium according to claim 17, wherein the instructions are further configured to cause the one or more processors to instruct:

the question property evaluator to, utilizing the question module and question database for each question presented, determine a likelihood of selecting one or more incorrect options,

wherein the question property evaluator considers at least a percentage of a number of times that the one or more incorrect options are expected to be selected and a percentage of a number of times that the one or more incorrect options are selected, and

the question response evaluator to, utilizing the evaluation module for each question presented, increase the likelihood that the at least one response is a false positive upon a decrease in the likelihood of selecting the one or more incorrect options.

25. A system for assessment driven learning, the system comprising:

means for displaying questions received from a processor on a question presentation device utilizing a question database having a plurality of questions,

wherein the plurality of question are associated with a stacking weight,

wherein the plurality of questions are ordered according to the stacking weight of each question, and wherein each question presented includes at least two answer options and at least on associated correct response;

means for determining, for each question presented, a likelihood of randomly selecting the at least one associated correct response using at least the number of answer options and a number of answer options a user is instructed to select;

means for receiving, from a device associated with the user, at least one response to each question presented;

means for determining, for each question presented, when the at least one response is the at least on associated correct response, a likelihood that the at least one response to the question is a false positive using the means for determining the likelihood of randomly selecting the at least one associated correct response, a response history of the user, and a response history database indicating a response history of one or more other users; and

means for adjusting the stacking weight of the question depending at least upon the likelihood that the at least one response to the question is a false positive.

26. The system according to claim 25, further comprising:

means for determining, for each question presented, the likelihood of randomly selecting the at least one associated correct response using

an inverse of a number of combinations of answer options the user is instructed to select from a set of the number of answer options when the number of answer options the user is instructed to select is less than the number of answer options, and

an inverse of two raised to a power of the number of answer options when the number of answer options the user is instructed to select is greater than or equal to the number of answer options; and

upon determination that the at least one response to the question is the at least one correct response,

means for increasing, for each question presented, the likelihood that the at least one response to the question is a false positive upon an increase in the likelihood of randomly selecting the at least one correct response.

27. The system according to claim 25, further comprising:

means for increasing, for each question presented, the likelihood that the at least one response to the question is a false positive upon an indication from the user response history that the at least one response is not the at least one associated correct response consecutively one or more times; and

means for decreasing, for each question presented, the likelihood that the at least one response to the question is a false positive upon an indication from the user response history that the at least one response is the at least one associated correct response consecutively one or more times.

28. The system of claim 25, further comprising:

means for determining, for each question presented, an amount of time the user takes to submit the at least one associated correct response;

means for determining, for each question presented, an average response time of the one or more other users and the user to submit the at least one associated correct response using the response history of the one or more other users and the amount of time the user takes to submit the at least one associated correct response; and

means for increasing, the likelihood that the at least one response is a false positive upon determination that the amount of time the user takes to submit the at least one associated correct response is at least one standard deviation below the average response time of the one or more other users and the user to submit the at least one associated correct response.

29. The system according to claim 25, further comprising:

means for determining, upon determination that the user is permitted to report a level of confidence for each question presented,

a first certainty weight that decreases the stacking weight upon determination that the reported level of confidence is higher than a certainty threshold and upon an indication from the user response history that the at least one response is the at least one associated correct response one or more times,

a second certainty weight that increases the stacking weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the user response history that the at least one response is the at least one associated correct response consecutively one or more times,

a third certainty weight that increases the stacking weight by an amount higher than the second certainty weight upon determination that the reported level of confidence is lower than the certainty threshold and upon an indication from the user response history that the at least one response is not the at least one associated response consecutively one or more times, and

a fourth certainty weight that increases the stacking weight by an amount higher than the third certainty weight upon determination that the reported level of confidence is higher than the certainty threshold and upon an indication from the user response history that the at least one response is not the at least one associated correct response consecutively one or more times.

30. The system according to claim 25, further comprising:

means for determining, for each question presented, a likelihood of selecting one or more incorrect options using at least a percentage of a number of times that the one or more incorrect options are expected to be selected and at least a percentage of a number of times that the one or more incorrect options are selected; and

means for increasing, for each question presented, the likelihood that the at least one response to the question is a false positive upon a decrease in the likelihood of selecting the one or more incorrect options.