Methods for automated essay analysis

• US 20050042592A1
• Filed: 09/22/2004
• Published: 02/24/2005
• Est. Priority Date: 01/23/2001
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Marcu, Daniel, Burstein, Jill, Andreyev, Vyacheslav, Leacock, Claudia, Chodorow, Martin Sanford
• Granted Patent: US 7,729,655 B2
• Time in Patent Office: Days
• US Class Current: 434/335
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

An essay is analyzed automatically by accepting the essay and determining whether each of a predetermined set of features is present or absent in each sentence of the essay. For each sentence in the essay a probability that the sentence is a member of a certain discourse element category is calculated. The probability is based on the determinations of whether each feature in the set of features is present or absent. Furthermore, based on the calculated probabilities, a sentence is chosen as the choice for the discourse element category.

Claims

1-23. -23. (cancelled)

24. A method for providing individualized essay writing instruction, the method comprising:

receiving an essay in an electronic format;

automatically determining a first value for each sentence in the essay that reflects the probability that the sentence is a member of a discourse element category, wherein the probability is based on the presence of each of a predetermined set of features in the sentence;

utilizing the first value to determine whether each sentence in the essay should be assigned to a discourse element category; and

identifying any discourse elements in the essay.

25. The method of claim 24, further comprising:

if the first values do not indicate the presence of a discourse element in the essay, indicating that the essay lacks sufficient clarity.

26. The method of claim 24 wherein the receiving step comprises:

receiving the essay into a computer via a keyboard.

27. The method of claim 24 wherein the discourse element category is thesis statement.

28. The method of claim 24 wherein the receiving step comprises:

scanning a paper form of the essay; and

performing optical character recognition on the scanned paper essay.

29. The method of claim 24 wherein the predetermined set of features comprises a feature based on a position within the essay.

30. The method of claim 24 wherein the predetermined set of features comprises a feature based on a presence or an absence of one or more selected words.

31. The method of claim 30 wherein the one or more selected words comprise one or more words empirically associated with a thesis statement.

32. The method of claim 30 wherein the one or more selected words comprise one or more words of belief.

33. The method of claim 30 wherein the predetermined set of features comprises a feature based on rhetorical relation.

34. The method of claim 30 wherein the first value is determined by utilizing a multivariate Bernoulli model.

35. The method of claim 34 wherein the first value comprises:

log[P(A_i|T)]/P(A_i)] if A_i present, and
log[P(T)]+Σ

log[P({overscore (A)}_i|T)]/P({overscore (A)}_i)] if A_i not present,

wherein

P(A_i|T) is a conditional probability that a sentence has a feature A_i given that the sentence is in a class T;

P({overscore (A)}_i|T) is a conditional probability that a sentence does not have a feature A_i given that the sentence is in a class T;

P(T) is a prior probability that a sentence is in a class T;

P(A_i) is a prior probability that a sentence contains a feature A_i; and

P({overscore (A)}_i) is a prior probability that a sentence does not contain a feature A_i.

36. The method of claim 35 wherein assigning comprises

assigning the sentence for which the first value is largest to the discourse category element.

37. The method of claim 24 wherein the probability is calculated utilizing a LaPlace estimator.

38. A system for identifying a discourse element of an essay, the system comprising:

an input device, wherein the input device receives an essay;

a first processing device for determining the presence of each of a predetermined set of features in each sentence of the essay;

a second processing device for calculating a probability that each sentence in the essay should be assigned to a discourse element category;

a third processing device for selecting one or more sentences in the essay that should be assigned to the discourse element category; and

a display device for displaying each of the one or more sentences that should be assigned to the discourse element category.

39. The system of claim 38 wherein the input device comprises a keyboard for entering the essay.

40. The system of claim 38 wherein the input device comprises a scanner for converting an essay in a non-electronic format into an essay in an electronic format.

41. The system of claim 38 wherein the first processing device utilizes a rhetorical structure parser.

42. The system of claim 38 wherein the second processing device utilizes a multivariate Bernoulli model.

43. The system of claim 38 wherein the second processing device utilizes a LaPlace estimator.

44. A method for creating a mathematical model for use in identifying discourse elements, the method comprising:

receiving a plurality of first essays relating to a particular subject, wherein each first essay is in an electronic format;

receiving annotations for each first essay, wherein each annotation identifies at least one discourse element;

identifying features, wherein each feature is exhibited by at least one identified discourse element;

computing empirical frequencies, wherein each empirical frequency relates to the presence of a feature with respect to the identified discourse elements across the plurality of first essays;

associating each empirical frequency with the related identified discourse element; and

utilizing the empirical frequencies to select discourse elements in at least one second essay.

45. The method of claim 44 wherein the annotations are prepared by human annotators.

46. The method of claim 44 wherein the annotations are prepared by electronic annotators.

47. The method of claim 44 wherein the features comprise positional features.

48. The method of claim 44 wherein the features comprise word choice features.

49. The method of claim 44 wherein the features comprise rhetorical structure theory features.

50. A method for providing feedback an essay, the method comprising:

receiving an essay prepared by a writer, wherein the essay is received in an electronic format;

automatically determining a first value for each sentence in the essay that reflects the probability that each sentence in the essay is a member of a discourse element category, wherein the probability is based on the presence of each of a predetermined set of features in each sentence of the essay;

utilizing the first value to determine whether each sentence in the essay should be assigned a discourse element category; and

providing feedback to the writer related to any discourse elements identified in the essay.

51. The method of claim 50, further comprising:

if the first values do not indicate the presence of a discourse element in the essay, indicating that the essay lacks sufficient clarity.

52. The method of claim 50 wherein the discourse element category is thesis statement.

53. The method of claim 50 wherein the predetermined set of features comprises a feature based on a position within the essay.

54. The method of claim 50 wherein the predetermined set of features comprises a feature based on a presence or an absence of one or more selected words.

55. The method of claim 54 wherein the one or more selected words comprise one or more words empirically associated with a thesis statement.

56. The method of claim 54 wherein the one or more selected words comprise one or more words of belief.

57. The method of claim 54 wherein the predetermined set of features comprises a feature based on rhetorical relation.

58. The method of claim 54 wherein the first value is determined by utilizing a multivariate Bernoulli model.

59. The method of claim 50 wherein the probability is calculated utilizing a LaPlace estimator.