System and method for identifying special word usage in a document

• US 7,269,544 B2
• Filed: 05/20/2003
• Issued: 09/11/2007
• Est. Priority Date: 05/20/2003
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Simske, Steven J.
• Primary Examiner(s): HARPER, V PAUL
• Application Number:
  US10/441,993
• Publication Number:
  US 20040236566A1
• Time in Patent Office: 1,575 Days
• Field of Search: None
• US Class Current: 704/4
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique

Abstract

A method of identifying potential novel word usage in a document comprises determining a part-of-speech assignment for each word in the document using a first part-of-speech tagger, determining a part-of-speech assignment for each word in the document using a second part-of-speech tagger different from the first part-of-speech tagger, and comparing the part-of-speech assignment of the first and second part-of-speech taggers. The method then generates a differential word set having words with different part-of-speech assignment by the first and second part-of-speech taggers. The words in the differential word set are candidates of words of novel usage.

Claims

1. A method of identifying potential novel word usage in a document, comprising:

determining a part-of-speech assignment for each word in the document using a first part-of-speech tagger;

determining a part-of-speech assignment for each word in the document using a second part-of-speech tagger different from the first part-of-speech tagger;

comparing the part-of-speech assignment of the first and second part-of-speech taggers;

generating a differential word set having words with different part-of-speech assignment by the first and second part-of-speech taggers, the words in the differential word set being candidates of words of novel usage; and

determining a weight to each word in the differential word set in response to the part-of-speech assignment of the word by the first part-of-speech tagger.

2. The method, as set forth in claim 1, wherein determining a weight to each word further comprises determining a weight in response to how frequently the word occurs in the document.

3. The method, as set forth in claim 1, wherein determining a weight to each word further comprises determining a weight in response to how frequently the word occurs in a document set comprising the document.

4. The method, as set forth in claim 1, wherein determining a weight to each word further comprises determining a weight by determining:

W=S_d*W_{POS(first POS tagger)}*FR,

where S_d is a difference sum that reflects the part-of-speech usage deviation from an expected part-of-speech usage of the word, W_{POS(first POS tagger)} is a weight based on the part-of-speech assignment for the word determined by the first part-of-speech tagger, and FR is a ratio of occurrence of the word in a document set comprising the document to a document corpus upon which the second part-of-speech tagger is based.

5. The method, as set forth in claim 1, further comprising selecting a subset of words from the differential word set in response to the weight determined for each word.

6. A method of identifying potential novel word usage in a document, comprising:

determining a part-of-speech assignment for each word in the document using a first part-of-speech tagger;

determining a part-of-speech assignment for each word in the document using a second part-of-speech tagger different from the first part-of-speech tagger;

comparing the part-of-speech assignment of the first and second part-of-speech taggers;

generating a differential word set having words with different part-of-speech assignment by the first and second part-of-speech taggers, the words in the differential word set being candidates of words of novel usage; and

determining a weight to each word in the differential word set, wherein determining a weight to each word comprises determining a weight in response to a deviation from an expected part-of-speech usage of the word.

7. The method, as set forth in claim 6, further comprising determining the weight based on a frequency of use of the word in the document.

8. The method, as set forth in claim 6, further comprising determining the weight based on a frequency of use of the word in a document set comprising the document.

9. The method, as set forth in claim 6, further comprising selecting a subset of words from the differential word set in response to the weight determined for each word.

10. A computer-readable article encoded with a computer-executable process, the process comprising:

assigning a first part-of-speech tag to words in at least one document according to a first part-of-speech tagging method;

assigning a second part-of-speech tag for words in the at least one document according to a second part-of-speech tagging method more simplistic than the first part-of-speech tagging method;

comparing the first and second part-of-speech tags;

generating a differential word set having words with different first and second part-of-speech tags; and

determining a weight to each word in the differential word set in response to the first part-of-speech tag of the word.

11. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight in response to a deviation from an expected part-of-speech usage of the word.

12. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight in response to how frequently the word occurs in the document.

13. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight in response to how frequently the word occurs in a document set comprising the document.

14. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight by determining:

W=S_d*W_{POS(first POS tagging method)}*FR,

where S_d is a difference sum that reflects the part-of-speech usage deviation from an expected part-of-speech usage of the word, W_{POS(first POS tagging method)} is a weight based on the first part-of-speech tag for the word, and FR is a ratio of occurrence of the word in a document set comprising the document to a document corpus upon which the second part-of-speech tagging method is based.

15. The article, as set forth in claim 10, further comprising selecting words of novel usage from the differential word set.

16. The article, as set forth in claim 15, wherein selecting words of novel usage comprises selecting words meeting a predetermined weight criteria.

17. A system for identifying potential novel word usage in a document set comprising:

a microprocessor; and

a series of computer instructions comprising a method of;

assigning a first part-of-speech tag to words in at least one document according to a first part-of-speech tagging method;

assigning a second part-of-speech tag for words in at least one document according to a second part-of-speech tagging method more simplistic than the first part-of-speech tagging method;

comparing the first and second part-of-speech tags;

generating a differential word set having words with different first and second part-of-speech tags; and

selecting words of novel usage from the differential word set meeting a predetermined weight criteria.

18. The system, as set forth in claim 17, wherein the method further comprises determining a weight to each word in the differential word set.

19. The system, as set forth in claim 17, wherein the method further comprises determining a weight to each word by determining:

W=S_d*W_{POS(first POS tagging method)}*FR,

where S_d is a difference sum that reflects the part-of-speech usage deviation from an expected part-of-speech usage of the word, W_{POS(first POS tagging method)} is a weight based on the first part-of-speech tag for the word, and FR is a ratio of occurrence of the word in a document set comprising the document to a document corpus upon which the second part-of-speech tagging method is based.