System and method for identifying special word usage in a document

• US 20040236566A1
• Filed: 05/20/2003
• Published: 11/25/2004
• Est. Priority Date: 05/20/2003
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Simske, Steven J.
• Granted Patent: US 7,269,544 B2
• Time in Patent Office: Days
• US Class Current: 704/4
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique

Abstract

A method of identifying potential novel word usage in a document comprises determining a part-of-speech assignment for each word in the document using a first part-of-speech tagger, determining a part-of-speech assignment for each word in the document using a second part-of-speech tagger different from the first part-of-speech tagger, and comparing the part-of-speech assignment of the first and second part-of-speech taggers. The method then generates a differential word set having words with different part-of-speech assignment by the first and second part-of-speech taggers. The words in the differential word set are candidates of words of novel usage.

Claims

1. A method of identifying potential novel word usage in a document, comprising:

determining a part-of-speech assignment for each word in the document using a first part-of-speech tagger;

determining a part-of-speech assignment for each word in the document using a second part-of-speech tagger different from the first part-of-speech tagger;

comparing the part-of-speech assignment of the first and second part-of-speech taggers; and

generating a differential word set having words with different part-of-speech assignment by the first and second part-of-speech taggers, the words in the differential word set being candidates of words of novel usage.

2. The method, as set forth in claim 1, further comprising determining a weight to each word in the differential word set.

3. The method, as set forth in claim 2, wherein determining a weight to each word comprises determining a weight in response to the part-of-speech assignment of the word by the first part-of-speech tagger.

4. The method, as set forth in claim 2, wherein determining a weight to each word comprises determining a weight in response to a deviation from an expected part-of-speech usage of the word.

5. The method, as set forth in claim 2, wherein determining a weight to each word comprises determining a weight in response to how frequently the word occurs in the document.

6. The method, as set forth in claim 2, wherein determining a weight to each word comprises determining a weight in response to how frequently the word occurs in a document set comprising the document.

7. The method, as set forth in claim 2, wherein determining a weight to each word comprises determining a weight by determining:

8. The method, as set forth in claim 2, further comprising selecting a subset of words from the differential word set in response to the weight determined for each word.

9. A computer-readable article encoded with a computer-executable process, the process comprising:

assigning a first part-of-speech tag to words in at least one document according to a first part-of-speech tagging method;

assigning a second part-of-speech tag for words in the at least one document according to a second part-of-speech tagging method more simplistic than the first part-of-speech tagging method;

comparing the first and second part-of-speech tags; and

generating a differential word set having words with different first and second part-of-speech tags.

10. The article, as set forth in claim 9, further comprising determining a weight to each word in the differential word set.

11. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight in response to the first part-of-speech tag of the word.

12. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight in response to a deviation from an expected part-of-speech usage of the word.

13. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight in response to how frequently the word occurs in the document.

14. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight in response to how frequently the word occurs in a document set comprising the document.

15. The article, as set forth in claim 10, wherein determining a weight to each word comprises determining a weight by determining:

16. The article, as set forth in claim 10, wherein selecting words of novel usage comprises selecting words meeting a predetermined weight criteria.

17. The article, as set forth in claim 9, further comprising selecting words of novel usage from the differential word set.

18. A system for identifying potential novel word usage in a document set comprising:

a microprocessor; and

a series of computer instructions comprising a method of;

assigning a first part-of-speech tag to words in at least one document according to a first part-of-speech tagging method;

assigning a second part-of-speech tag for words in at least one document according to a second part-of-speech tagging method more simplistic than the first part-of-speech tagging method;

comparing the first and second part-of-speech tags; and

generating a differential word set having words with different first and second part-of-speech tags.

19. The system, as set forth in claim 18, wherein the method further comprises determining a weight to each word in the differential word set.

20. The system, as set forth in claim 18, wherein the method further comprises determining a weight to each word by determining:

21. The system, as set forth in claim 18, further comprising selecting words of novel usage from the differential word set meeting a predetermined weight criteria.