Method and system for extracting opinions from text documents

• US 20050091038A1
• Filed: 10/22/2003
• Published: 04/28/2005
• Est. Priority Date: 10/22/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Yi, Jeonghee, Nasukawa, Tetsuya
• Granted Patent: US 8,200,477 B2
• Time in Patent Office: Days
• US Class Current: 704/10
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: Y10S 707/939 Idea solicitation

Abstract

A method and system for extracting opinions about a subject of interest from a text document in which each sentence is analyzed individually to identify the opinions. The most relevant feature terms related to the subject are extracted from the document based on their relevancy scores. Candidate feature terms are definite noun phrases at the beginning of the sentences. For each sentence that refers to the subject or a feature term, the invention determines whether the sentence includes an opinion polarity about the subject or the feature term. The opinion polarity is detected by identifying opinion terms in the sentence using an opinion dictionary or an opinion rule base, parsing the sentence with an English parser to identify grammatical components in the sentence and their relationships, and finding a matching entry in the dictionary or the rule base.

Claims

1. A method for extracting opinions about a subject of interest from a text document having a plurality of sentences, the subject associated with a plurality of features, the method comprising the steps of:

extracting from the document feature terms related to the features most relevant to the subject;

for each sentence referring to a feature term, determining whether the sentence includes an opinion polarity about the feature term; and

for each sentence referring to the subject, determining whether the sentence includes an opinion polarity about the subject.

2. The method as recited in claim 1 further comprising the step of determining an opinion skeleton for each opinion to provide support details for said opinion.

7. The method as recited in claim 1, wherein the step of extracting the feature terms includes the steps of:

determining a plurality of candidate feature terms associated the subject;

calculating a relevance score for each candidate feature term; and

identifying the most relevant feature terms from the candidate feature terms based on the relevance scores.

8. The method as recited in claim 7, wherein each candidate feature term is a definite noun phrase located at the beginning of a sentence.

10. The method as recited in claim 1, wherein the step of determining whether the sentence includes an opinion polarity includes the steps of:

identifying opinion terms in the sentence using an opinion dictionary, each entry in the dictionary having an opinion term, a part-of-speech tag, and an associated opinion polarity;

for each sentence having a feature term and an opinion term, parsing the sentence with an English parser to identify grammatical components in the sentence and relationships between said components; and

identifying an opinion polarity associated with said feature term using the opinion dictionary.

11. The method as in claim 10, wherein:

the grammatical components include verb phrases, subject phrases, object phrases, complements, and prepositional phrases;

each feature term may have a modifier describing the feature term; and

the step of identifying an opinion polarity based on the opinion dictionary includes the steps of;

for each sentence having a verb phrase where the verb phrase has no matching entry in the opinion dictionary, assigning an opinion polarity of the modifier of the feature term to the feature term, the opinion polarity of the modifier being defined in the opinion dictionary.

12. The method as recited in claim 10, wherein the opinion polarity associated with the feature term is identified based on an opinion rule.

13. The method as recited in claim 12, wherein:

the grammatical components include verb phrases, subject phrases, object phrases, complements, and prepositional phrases; and

the step of identifying an opinion polarity based on the opinion rule base includes the steps of;

for a sentence having a verb phrase, finding a matching entry in the rule base; and

if there is a matching entry, assigning an opinion polarity to the feature term as defined by the matching entry.

14. The method as recited in claim 12, wherein the rule base comprises a plurality of rules each having a relationship term, a target of the opinion, and a polarity of the opinion.

15. The method as recited in claim 12, wherein the rule base comprises a plurality of rules each having a relationship term, a source of the opinion, and a target of the opinion.

16. The method as recited in claim 15, wherein the target of the opinion is a component of the sentence to which the opinion is to be assigned.

17. The method as recited in claim 15, wherein the source of the opinion is a component of the sentence of which opinion polarity is to be assigned to the target.

18. The method as recited in claim 1, wherein each feature term has a “

part-of”

relationship with the subject.

19. The method as recited in claim 1, wherein each feature term has an “

attribute-of”

relationship with the subject.

20. The method as recited in claim 1, wherein each feature term has an “

attribute-of”

relationship with the associated feature.

3. The method as recited in claim 3, wherein the opinion skeleton includes a feature term and an opinion term referring to said feature term.

4. The method as recited in claim 3, wherein the opinion skeleton includes the subject and an opinion term referring to said subject.

5. The method as recited in claim 3, wherein the opinion skeleton includes a feature term, an opinion term, and a relationship between the feature term and the opinion term.

6. The method as recited in claim 3, wherein the opinion skeleton includes the subject, an opinion term, and a relationship between the subject and the opinion term.

9. The method as recited in claim 6, wherein the step of calculating a relevance score is based on a likelihood ratio test.

21. A system for extracting opinions about a subject of interest from a text document having a plurality of sentences, the subject associated with a plurality of features, the system comprising:

means for extracting from the document feature terms related to the features most relevant to the subject;

for each sentence referring to a feature term, means for determining whether the sentence includes an opinion polarity about the feature term; and

for each sentence referring to the subject, means for determining whether the sentence includes an opinion polarity about the subject.

22. A computer-program product for use with a computer for extracting opinions about a subject of interest from a text document having a plurality of sentences, the subject associated with a plurality of features, the computer-program product comprising:

a computer-readable medium;

means, provided on the computer-readable medium, for extracting from the document feature terms related to the features most relevant to the subject;

means, provided on the computer-readable medium, for each sentence referring to a feature term, for determining whether the sentence includes an opinion polarity about the feature term; and

means, provided on the computer-readable medium, for each sentence referring to the subject, for determining whether the sentence includes an opinion polarity about the subject.