System and method for identifying organizational elements in argumentative or persuasive discourse

• US 9,208,139 B2
• Filed: 01/07/2013
• Issued: 12/08/2015
• Est. Priority Date: 01/05/2012
• Status: Active Grant

• Current Assignee: Educational Testing Service
• Original Assignee: Educational Testing Service
• Inventors: Madnani, Nitin, Heilman, Michael, Tetreault, Joel, Chodorow, Martin
• Primary Examiner(s): Mills, Frank D
• Application Number:
  US13/735,493
• Publication Number:
  US 20130179766A1
• Time in Patent Office: 1,065 Days
• Field of Search: 715/230
• US Class Current: 1/1
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation

Abstract

In accordance with the teachings described herein, systems and methods are provided for identifying organizational elements in argumentative or persuasive discourse. A text that has been annotated is received. The annotated text includes argumentative or persuasive discourse that includes claims and evidence and organizational elements configured to organize the claims and evidence. Annotations of the annotated text distinguish the organizational elements from the claims and evidence. A rule set or a feature set is identified from the annotated text, where the rule set or the feature set includes textual patterns or word frequency features related to the organizational elements of the annotated text. A model is built based on the annotations and on the rule set or the feature set. The model is configured to identify organizational elements in a new text. The model is applied to the new text.

Claims

1. A computer-implemented method for identifying organizational elements in argumentative or persuasive discourse, the method comprising:

receiving, using a processing system comprising one or more data processors, text that has been annotated, the annotated text including argumentative or persuasive discourse in one or more annotation data structures that includes;

claims and evidence, and

organizational elements configured to organize the claims and evidence, wherein annotations of the annotated text distinguish the organizational elements from the claims and evidence;

identifying a rule set or a feature set from the annotated text and storing the rule set or the feature set in a rule set data structure or a feature set data structure, the rule set or the feature set including textual patterns or word frequency features related to the organizational elements of the annotated text;

building a model based on the annotations and the rule set or the feature set, the model being configured to identify organizational elements in a new text, the model being stored in a model data structure; and

applying the model to the new text, wherein the applying includes accessing the model data structure and determining whether a marginal probability for a word w_jⁱ exceeds a threshold λ

.

2. The method of claim 1, further comprising:

receiving the annotated text including the organizational elements, wherein the organizational elements refer to the claims and evidence and provide an organizational framework for the argumentative or persuasive discourse, and wherein the organizational elements;

declare claims of a writer, speaker, or author;

restate claims of an opponent of the writer, speaker, or author;

evaluate the claims of the opponent;

or

present evidence and relate the evidence to particular claims.

3. The method of claim 1, further comprising:

building the model based on the feature set, wherein the feature set includes features based on the rule set and the textual patterns.

4. The method of claim 1, further comprising:

identifying the rule set from the annotated text, wherein the identifying includes;

identifying an n-gram in the annotated text that includes organizational elements, wherein the n-gram is a contiguous sequence of n items from the annotated text that is identified based on the annotations;

determining a textual pattern that describes the n-gram, wherein the textual pattern is an abstraction of the n-gram that includes parts of speech or word identifiers in a specific sequence; and

storing the textual pattern in the rule set.

5. The method of claim 4, further comprising:

determining the textual pattern that describes the n-gram, wherein the textual pattern describes more than one n-gram in the annotated text.

6. The method of claim 4, further comprising:

determining the textual pattern that describes the n-gram, wherein the parts of speech or word identifiers include a modal verb, an agree verb, an author noun, a verb, a noun, a pronoun, an adjective, an adverb, a proposition, a conjunction, or an interjection.

7. The method of claim 1, further comprising:

building the model based on the annotations and the rule set or the feature set, wherein the building includes maximizing an objective function, the objective function including terms corresponding to the textual patterns or word frequency features.

8. The method of claim 7, further comprising:

building the model based on the annotations and the rule set or the feature set, wherein the annotated text includes N examples (w, y) indexed by i, including sequences of words wⁱ and sequences of labels yⁱ, with individual words and labels indexed by j, wherein yⁱ is a sequence of binary values indicating whether each word w_jⁱ in the sequence is an organizational element, and wherein the building includes;

determining a parameter vector θ

that maximizes the objective function, wherein the objective function is defined as;

9. The method of claim 1, further comprising:

identifying the feature set from the annotated text, wherein the word frequency features include;

a relative frequency for a first word of the annotated text in a text corpus;

a first binary value identifying whether the relative frequency meets or exceeds a first threshold;

a number of different essay prompts in which the first word has appeared;

or

a second binary value identifying whether the number of different essay prompts meets or exceeds a second threshold.

10. The method of claim 9, further comprising:

identifying the feature set from the annotated text, wherein features of the feature set include;

a third binary value identifying whether the first word includes only letters;

a fourth binary value identifying whether a rule-based system predicts the first word as being an organizational element;

a fifth binary value identifying whether the rule-based system predicts a second word as being an organizational element, wherein the second word immediately precedes the first word in the annotated text;

a sixth binary value identifying whether the first word occurs at the beginning of a sentence;

a seventh binary value identifying whether the first word occurs at the end of the sentence;

or

an eighth binary value identifying a transition between a label applied to the first word and a label applied to the second word, wherein the first and the second labels indicate whether the first and the second words, respectively, are organizational elements.

11. The method of claim 1, further comprising:

applying the model to the new text, wherein the applying includes finding a most likely sequence of labels y given a sequence of words wⁱ of the new text, wherein the most likely sequence of labels y is a vector including binary values indicating whether words of the sequence of words wⁱ are organizational elements, and wherein the most likely sequence of labels y is determined using Viterbi decoding.

12. The method of claim 11, further comprising:

determining the most likely sequence of labels y using the Viterbi decoding according to a following equation;

13. The method of claim 1, further comprising:

determining whether the marginal probability for the word w_jⁱ exceeds the threshold λ

to determine a binary value ŷ

_jⁱ, wherein the binary value ŷ

_jⁱ indicates whether the word w_jⁱ is an organizational element, and wherein the binary value ŷ

_jⁱ is determined according to an equation;

14. The method of claim 1, further comprising:

evaluating the model, the evaluating including;

applying the model to the new text, wherein the applying includes generating a set of predictions using the model, and wherein the predictions identify organizational elements of the new text;

comparing the predictions to labels generated by humans, wherein the labels identify organizational elements of the new text; and

based on the comparing, evaluating the model using precision, recall, or F₁ measures.

15. The method of claim 1, further comprising:

applying the model to the new text, wherein the model is applied to generate a score for the new text, and wherein the new text is an essay.

16. The method of claim 15, wherein the model includes a probability distribution based on the rule set or the feature set.

17. A system for identifying organizational elements in argumentative or persuasive discourse, the system comprising:

a data processor; and

computer-readable memory in communication with the data processor encoded with instructions for commanding the data processor to execute steps comprising;

receiving text that has been annotated, the annotated text including argumentative or persuasive discourse in one or more annotation data structures that includes;

claims and evidence, and

organizational elements configured to organize the claims and evidence, wherein annotations of the annotated text distinguish the organizational elements from the claims and evidence;

identifying a rule set or a feature set from the annotated text and storing the rule set or the feature set in a rule set data structure or a feature set data structure, the rule set or the feature set including textual patterns or word frequency features related to the organizational elements of the annotated text;

building a model based on the annotations and the rule set or the feature set, the model being configured to identify organizational elements in a new text, the model being stored in a model data structure; and

applying the model to the new text, wherein the applying includes accessing the model data structure and determining whether a marginal probability for a word w_jⁱ exceeds a threshold λ

.

18. The system of claim 17, wherein the steps further comprise:

applying the model to the new text, wherein the model is applied to generate a score for the new text, and wherein the new text is an essay.

19. The system of claim 18, wherein the model includes a probability distribution based on the rule set or the feature set.

20. The system of claim 17, wherein the organizational elements:

declare claims of a writer, speaker, or author;

restate claims of an opponent of the writer, speaker, or author;

evaluate the claims of the opponent;

or

present evidence and relate the evidence to particular claims.

21. The system of claim 17, wherein the steps further include:

building the model based on the feature set, wherein the feature set includes features based on the rule set and the textual patterns.

22. The system of claim 17, wherein the steps further include:

identifying the rule set from the annotated text, wherein the identifying includes;

identifying an n-gram in the annotated text that includes organizational elements, wherein the n-gram is a contiguous sequence of n items from the annotated text that is identified based on the annotations;

determining a textual pattern that describes the n-gram, wherein the textual pattern is an abstraction of the n-gram that includes parts of speech or word identifiers in a specific sequence; and

storing the textual pattern in the rule set.

23. The system of claim 22, wherein the steps further include:

determining the textual pattern that describes the n-gram, wherein the textual pattern describes more than one n-gram in the annotated text.

24. The system of claim 22, wherein the steps further include:

determining the textual pattern that describes the n-gram, wherein the parts of speech or word identifiers include a modal verb, an agree verb, an author noun, a verb, a noun, a pronoun, an adjective, an adverb, a proposition, a conjunction, or an interjection.

25. The system of claim 17, wherein the steps further include:

building the model based on the annotations and the rule set or the feature set, wherein the building includes maximizing an objective function, the objective function including terms corresponding to the textual patterns or word frequency features.

26. The system of claim 25, wherein the steps further include:

building the model based on the annotations and the rule set or the feature set, wherein the annotated text includes N examples (w, y) indexed by i, including sequences of words wⁱ and sequences of labels yⁱ, with individual words and labels indexed by j, wherein yⁱ is a sequence of binary values indicating whether each word w_jⁱ in the sequence is an organizational element, and wherein the building includes;

determining a parameter vector θ

that maximizes the objective function, wherein the objective function is defined as;

27. The system of claim 17, wherein the steps further include:

identifying the feature set from the annotated text, wherein the word frequency features include;

a relative frequency for a first word of the annotated text in a text corpus;

a first binary value identifying whether the relative frequency meets or exceeds a first threshold;

a number of different essay prompts in which the first word has appeared;

or

a second binary value identifying whether the number of different essay prompts meets or exceeds a second threshold.

28. The system of claim 27, wherein the steps further include:

identifying the feature set from the annotated text, wherein features of the feature set include;

a third binary value identifying whether the first word includes only letters;

a fourth binary value identifying whether a rule-based system predicts the first word as being an organizational element;

a fifth binary value identifying whether the rule-based system predicts a second word as being an organizational element, wherein the second word immediately precedes the first word in the annotated text;

a sixth binary value identifying whether the first word occurs at the beginning of a sentence;

a seventh binary value identifying whether the first word occurs at the end of the sentence;

or

an eighth binary value identifying a transition between a label applied to the first word and a label applied to the second word, wherein the first and the second labels indicate whether the first and the second words, respectively, are organizational elements.

29. The system of claim 17, wherein the steps further include:

applying the model to the new text, wherein the applying includes finding a most likely sequence of labels y given a sequence of words wⁱ of the new text, wherein the most likely sequence of labels y is a vector including binary values indicating whether words of the sequence of words wⁱ are organizational elements, and wherein the most likely sequence of labels y is determined using Viterbi decoding.

30. The system of claim 29, wherein the steps further include:

determining the most likely sequence of labels y using the Viterbi decoding according to a following equation;

31. The system of claim 17, wherein the steps further include:

determining whether the marginal probability for the word w_jⁱ exceeds the threshold λ

to determine a binary value ŷ

_jⁱ, wherein the binary value ŷ

_jⁱ indicates whether the word w_jⁱ is an organizational element, and wherein the binary value ŷ

_jⁱ is determined according to an equation;

32. The system of claim 17, wherein the steps further include:

evaluating the model, the evaluating including;

applying the model to the new text, wherein the applying includes generating a set of predictions using the model, and wherein the predictions identify organizational elements of the new text;

comparing the predictions to labels generated by humans, wherein the labels identify organizational elements of the new text; and

based on the comparing, evaluating the model using precision, recall, or F₁ measures.

33. A non-transitory computer-readable storage medium for identifying organizational elements in argumentative or persuasive discourse, the computer-readable medium comprising computer executable instructions which, when executed, cause the computer system to execute steps comprising:

receiving, using a processing system comprising one or more data processors, text that has been annotated, the annotated text including argumentative or persuasive discourse in one or more annotation data structures that includes;

claims and evidence, and

organizational elements configured to organize the claims and evidence, wherein annotations of the annotated text distinguish the organizational elements from the claims and evidence;

identifying a rule set or a feature set from the annotated text and storing the rule set or the feature set in a rule set data structure or a feature set data structure, the rule set or the feature set including textual patterns or word frequency features related to the organizational elements of the annotated text;

building a model based on the annotations and the rule set or the feature set, the model being configured to identify organizational elements in a new text, the model being stored in a model data structure; and

applying the model to the new text, wherein the applying includes accessing the model data structure and determining whether a marginal probability for a word w_jⁱ exceeds a threshold λ

.

34. The storage medium of claim 33, wherein the steps further include:

applying the model to the new text, wherein the model is applied to generate a score for the new text, and wherein the new text is an essay.

35. The storage medium of claim 34, wherein the model includes a probability distribution based on the rule set or the feature set.

36. The storage medium of claim 33, wherein the steps further include:

building the model based on the feature set, wherein the feature set includes features based on the rule set and the textual patterns.

37. The storage medium of claim 33, wherein the steps further include:

identifying the rule set from the annotated text, wherein the identifying includes;

identifying an n-gram in the annotated text that includes organizational elements, wherein the n-gram is a contiguous sequence of n items from the annotated text that is identified based on the annotations;

determining a textual pattern that describes the n-gram, wherein the textual pattern is an abstraction of the n-gram that includes parts of speech or word identifiers in a specific sequence; and

storing the textual pattern in the rule set.

38. The storage medium of claim 37, wherein the steps further include:

determining the textual pattern that describes the n-gram, wherein the textual pattern describes more than one n-gram in the annotated text.

39. The storage medium of claim 37, wherein the steps further include:

determining the textual pattern that describes the n-gram, wherein the parts of speech or word identifiers include a modal verb, an agree verb, an author noun, a verb, a noun, a pronoun, an adjective, an adverb, a proposition, a conjunction, or an interjection.

40. The storage medium of claim 33, wherein the steps further include:

building the model based on the annotations and the rule set or the feature set, wherein the building includes maximizing an objective function, the objective function including terms corresponding to the textual patterns or word frequency features.

41. The storage medium of claim 40, wherein the steps further include:

building the model based on the annotations and the rule set or the feature set, wherein the annotated text includes N examples (w, y) indexed by i, including sequences of words wⁱ and sequences of labels yⁱ, with individual words and labels indexed by j, wherein yⁱ is a sequence of binary values indicating whether each word w_jⁱ in the sequence is an organizational element, and wherein the building includes;

determining a parameter vector θ

that maximizes the objective function, wherein the objective function is defined as;

42. The storage medium of claim 33, wherein the steps further include:

identifying the feature set from the annotated text, wherein the word frequency features include;

a relative frequency for a first word of the annotated text in a text corpus;

a first binary value identifying whether the relative frequency meets or exceeds a first threshold;

a number of different essay prompts in which the first word has appeared;

or

a second binary value identifying whether the number of different essay prompts meets or exceeds a second threshold.

43. The storage medium of claim 42, wherein the steps further include:

identifying the feature set from the annotated text, wherein features of the feature set include;

a third binary value identifying whether the first word includes only letters;

a fourth binary value identifying whether a rule-based system predicts the first word as being an organizational element;

a fifth binary value identifying whether the rule-based system predicts a second word as being an organizational element, wherein the second word immediately precedes the first word in the annotated text;

a sixth binary value identifying whether the first word occurs at the beginning of a sentence;

a seventh binary value identifying whether the first word occurs at the end of the sentence;

or

an eighth binary value identifying a transition between a label applied to the first word and a label applied to the second word, wherein the first and the second labels indicate whether the first and the second words, respectively, are organizational elements.

44. The storage medium of claim 33, wherein the steps further include:

applying the model to the new text, wherein the applying includes finding a most likely sequence of labels y given a sequence of words wⁱ of the new text, wherein the most likely sequence of labels y is a vector including binary values indicating whether words of the sequence of words wⁱ are organizational elements, and wherein the most likely sequence of labels y is determined using Viterbi decoding.

45. The storage medium of claim 44, wherein the steps further include:

determining the most likely sequence of labels y using the Viterbi decoding according to a following equation;

46. The storage medium of claim 33, wherein the steps further include:

determining whether the marginal probability for the word w_jⁱ exceeds the threshold λ

to determine a binary value ŷ

_jⁱ, wherein the binary value ŷ

_jⁱ indicates whether the word w_jⁱ is an organizational element, and wherein the binary value ŷ

_jⁱ is determined according to an equation;

47. The storage medium of claim 33, wherein the steps further include:

evaluating the model, the evaluating including;

applying the model to the new text, wherein the applying includes generating a set of predictions using the model, and wherein the predictions identify organizational elements of the new text;

comparing the predictions to labels generated by humans, wherein the labels identify organizational elements of the new text; and

based on the comparing, evaluating the model using precision, recall, or F₁ measures.