System and method for assessment design

• US 20050221266A1
• Filed: 04/04/2005
• Published: 10/06/2005
• Est. Priority Date: 04/02/2004
• Status: Abandoned Application

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Wilson, Mark, Mislevy, Robert J., Kennedy, Cathleen A., Haertel, Geneva D., Hamel, Lawrence A.
• Application Number:
  US11/098,059
• Publication Number:
  US 20050221266A1
• Time in Patent Office: Days
• US Class Current: 434/322
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A method and apparatus are provided for designing educational assessments. In one embodiment, a method for guiding a user in the design of an assessment includes receiving, from the user, one or more goals relating to the assessment and translating those goals into a task specification for an assessment task in accordance with one or more user-defined variables.

Claims

1. A method for guiding a user in the design of an educational assessment task specification, the method comprising:

receiving, from said user, one or more goals relating to the assessment; and

translating said one or more goals into an operational assessment task specification in accordance with one or more user-defined variables.

2. The method of claim 1, wherein said receiving comprises:

receiving, from said user, at least one selected generating object, where said at least one generating object provides a narrative groundwork for said translating.

3. The method of claim 2, wherein said at least one generating object is at least one of:

a design pattern, a student model, a task model variable, a rubric, a wizard and an educational standard.

4. The method of claim 3, wherein said at least one generating object is at least one design pattern specifying:

at least one aspect of knowledge, skill or ability to be assessed, at least one potential observation for providing evidence of said knowledge, skill or ability and at least one characteristic of a task for evoking said evidence.

5. The method of claim 4, wherein said at least one design pattern further specifies at least one of:

a summary of said at least one design pattern, a rationale for said at least one design pattern, at least one additional aspect of knowledge, skill or ability to be assessed, at least one work product for yielding said evidence, at least one rubric for scoring said at least one work product, at least one variable feature for shifting a focus or difficulty level of said at least one design pattern, at least one link to another design pattern, at least one link to an educational standard, at least one unifying concept relating to said at least one design pattern, at least one link to a task template implementing said at least one design pattern, at least one link to an exemplar assessment task that is an instance of said at least one design pattern and at least one resource relevant to said at least one design pattern.

6. The method of claim 3, wherein said at least one generating object is at least one student model specifying at least one of:

a distribution summary, a distribution type, a covariance matrix, a means matrix, an online resource related to said at least one student model and a reference related to said at least one student model.

7. The method of claim 3, wherein said at least one generating object is at least one task model variable specifying at least one of:

a task model variable type, a task model variable category, an online resource related to said at least one task model variable and a reference related to said at least one task model variable.

8. The method of claim 3, wherein said at least one generating object is related to at least one ancillary object comprising at least one of:

a student model, a student model variable, a task model variable, a measurement model, an evaluation procedure, an evaluation phase, a materials and presentation setting, an activity, an exemplar, an educational standard, a continuous zone, a work product, an observable variable, a design pattern and a task template.

9. The method of claim 8, wherein said at least one ancillary object is at least one task template specifying at least one of:

a task template type, a student model summary, a measurement model summary, an evaluation procedure summary, a work product summary, a task model variable summary, a task model variable setting, a materials and presentation requirement, an activity summary, a student tool, an online resource related to said at least on task template and a reference related to said at least one task template.

10. The method of claim 8, wherein said at least one ancillary object is at least one student model variable specifying at least one of:

a student model variable type, a minimum student model variable value, a maximum student model variable value, a finite category, an online resource related to said at least one student model variable and a reference related to said at least one student model variable.

11. The method of claim 8, wherein said at least one ancillary object is at least one activity specifying at least one of:

presentation logic, an online resource related to said at least one activity and a reference related to said at least one activity.

12. The method of claim 8, wherein said at least one ancillary object is at least one evaluation procedure specifying at least one of:

an online resource related to said at least one evaluation procedure and a reference related to said at least one evaluation procedure.

13. The method of claim 8, wherein said at least one ancillary object is at least one evaluation phase specifying at least one of:

evaluation action data, an evaluation action, an online resource related to said at least one evaluation phase and a reference related to said at least one evaluation phase.

14. The method of claim 8, wherein said at least one ancillary object is at least one observable variable specifying at least one of:

an observable variable category, an online resource related to said at least one observable variable and a reference related to said at least one observable variable.

15. The method of claim 8, wherein said at least one ancillary object is at least one measurement model specifying at least one of:

a measurement model type, a scoring matrix, a design matrix, a calibration parameter, an online resource related to said at least one measurement model and a reference related to said at least one measurement model.

16. The method of claim 8, wherein said at least one ancillary object is at least one work product specifying at least one of:

a work product type, an example, an online resource related to said at least one work product and a reference related to said at least one work product.

17. The method of claim 8, wherein said at least one ancillary object is at least one materials and presentation setting specifying at least one of:

a material type, a role of at least one stimulus related to said at least one materials and presentation, an online resource related to said at least one materials and presentation and a reference related to said at least one materials and presentation.

18. The method of claim 8, wherein at least one of said at least one generating object and said at least one ancillary object is extensible.

19. The method of claim 2, wherein said translating comprises:

instantiating said at least one generating object in a task template.

20. The method of claim 19, wherein said task template comprises:

at least one student model defining at least one aspect of knowledge, skill or ability to be assessed; and

at least one activity representing a phase of a task for providing evidence of said at least one aspect of knowledge, skill or ability.

21. The method of claim 20, wherein said at least one student model is configured for an overall measure of proficiency in said at least one aspect of knowledge, skill or ability.

22. The method of claim 20, wherein said at least one student model is configured for:

a first measure of proficiency related to content associated with said at least one aspect of knowledge, skill or ability; and

a second measure of proficiency related to inquiry associated with said at least one aspect of knowledge, skill or ability.

23. The method of claim 20, wherein said at least one student model is configured for a plurality of measures of proficiency, each of said plurality of measures relating to a different subset of said at least one aspect of knowledge, skill or ability.

24. The method of claim 20, wherein said at least one activity comprises:

at least one evidence model defining at least one behavior for revealing said at least one aspect of knowledge, skill or ability; and

at least one task model defining at least one task for eliciting said at least one behavior.

25. The method of claim 24, wherein said at least one evidence model comprises:

at least one evaluation procedure for evaluating features of said at least one behavior; and

at least one measurement model for relating said features of said at least one behavior to said at least one aspect of knowledge, skill or ability.

26. The method of claim 25, wherein said at least one measurement model is a multidimensional random coefficients multinomial logit model.

27. A computer readable medium containing an executable program for guiding a user in the design of an educational assessment task specification, where the program performs the steps of:

receiving, from said user, one or more goals relating to the assessment; and

translating said one or more goals into an operational assessment task specification in accordance with one or more user-defined variables.

28. The computer readable medium of claim 27, wherein said receiving comprises:

receiving, from said user, at least one selected generating object, where said at least one generating object provides a narrative groundwork for said translating.

29. The computer readable medium of claim 28, wherein said at least one generating object is at least one of:

a design pattern, a student model, a task model variable, a rubric, a wizard and an educational standard.

30. The computer readable medium of claim 29, wherein said at least one generating object is at least one design pattern specifying:

at least one aspect of knowledge, skill or ability to be assessed, at least one potential observation for providing evidence of said knowledge, skill or ability and at least one characteristic of a task for evoking said evidence.

31. The computer readable medium of claim 30, wherein said at least one design pattern further specifies at least one of:

a summary of said at least one design pattern, a rationale for said at least one design pattern, at least one additional aspect of knowledge, skill or ability to be assessed, at least one work product for yielding said evidence, at least one rubric for scoring said at least one work product, at least one variable feature for shifting a focus or difficulty level of said at least one design pattern, at least one link to another design pattern, at least one link to an educational standard, at least one unifying concept relating to said at least one design pattern, at least one link to a task template implementing said at least one design pattern, at least one link to an exemplar assessment task that is an instance of said at least one design pattern and at least one resource relevant to said at least one design pattern.

32. The computer readable medium of claim 29, wherein said at least one generating object is at least one student model specifying at least one of:

a distribution summary, a distribution type, a covariance matrix, a means matrix, an online resource related to said at least one student model and a reference related to said at least one student model.

33. The computer readable medium of claim 29, wherein said at least one generating object is at least one task model variable specifying at least one of:

a task model variable type, a task model variable category, an online resource related to said at least one task model variable and a reference related to said at least one task model variable.

34. The computer readable medium of claim 29, wherein said at least one generating object is related to at least one ancillary object comprising at least one of:

a student model, a student model variable, a task model variable, a measurement model, an evaluation procedure, an evaluation phase, a materials and presentation setting, an activity, an exemplar, an educational standard, a continuous zone, a work product, an observable variable, a design pattern and a task template.

35. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one task template specifying at least one of:

a task template type, a student model summary, a measurement model summary, an evaluation procedure summary, a work product summary, a task model variable summary, a task model variable setting, a materials and presentation requirement, an activity summary, a student tool, an online resource related to said at least on task template and a reference related to said at least one task template.

36. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one student model variable specifying at least one of:

a student model variable type, a minimum student model variable value, a maximum student model variable value, a finite category, an online resource related to said at least one student model variable and a reference related to said at least one student model variable.

37. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one activity specifying at least one of:

presentation logic, an online resource related to said at least one activity and a reference related to said at least one activity.

38. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one evaluation procedure specifying at least one of:

an online resource related to said at least one evaluation procedure and a reference related to said at least one evaluation procedure.

39. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one evaluation phase specifying at least one of:

evaluation action data, an evaluation action, an online resource related to said at least one evaluation phase and a reference related to said at least one evaluation phase.

40. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one observable variable specifying at least one of:

an observable variable category, an online resource related to said at least one observable variable and a reference related to said at least one observable variable.

41. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one measurement model specifying at least one of:

a measurement model type, a scoring matrix, a design matrix, a calibration parameter, an online resource related to said at least one measurement model and a reference related to said at least one measurement model.

42. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one work product specifying at least one of:

a work product type, an example, an online resource related to said at least one work product and a reference related to said at least one work product.

43. The computer readable medium of claim 34, wherein said at least one ancillary object is at least one materials and presentation setting specifying at least one of:

a material type, a role of at least one stimulus related to said at least one materials and presentation, an online resource related to said at least one materials and presentation and a reference related to said at least one materials and presentation.

44. The computer readable medium of claim 34, wherein at least one of said at least one generating object and said at least one ancillary object is extensible.

45. The computer readable medium of claim 28, wherein said translating comprises:

instantiating said at least one generating object in a task template.

46. The computer readable medium of claim 45, wherein said task template comprises:

at least one student model defining at least one aspect of knowledge, skill or ability to be assessed; and

at least one activity representing a phase of a task for providing evidence of said at least one aspect of knowledge, skill or ability.

47. The computer readable medium of claim 46, wherein said at least one student model is configured for an overall measure of proficiency in said at least one aspect of knowledge, skill or ability.

48. The computer readable medium of claim 46, wherein said at least one student model is configured for:

a first measure of proficiency related to content associated with said at least one aspect of knowledge, skill or ability; and

a second measure of proficiency related to inquiry associated with said at least one aspect of knowledge, skill or ability.

49. The computer readable medium of claim 46, wherein said at least one student model is configured for a plurality of measures of proficiency, each of said plurality of measures relating to a different subset of said at least one aspect of knowledge, skill or ability.

50. The computer readable medium of claim 46, wherein said at least one activity comprises:

at least one evidence model defining at least one behavior for revealing said at least one aspect of knowledge, skill or ability; and

at least one task model defining at least one task for eliciting said at least one behavior.

51. The computer readable medium of claim 50, wherein said at least one evidence model comprises:

at least one evaluation procedure for evaluating features of said at least one behavior; and

at least one measurement model for relating said features of said at least one behavior to said at least one aspect of knowledge, skill or ability.

52. The computer readable medium of claim 51, wherein said at least one measurement model is a multidimensional random coefficients multinomial logit model.

53. System for guiding a user in the design of an educational assessment task specification, the system comprising:

means for receiving, from said user, one or more goals relating to the assessment; and

means for translating said one or more goals into an operational assessment task specification in accordance with one or more user-defined variables.