System and Method for Identifying Phrases in Text

• US 20130158979A1
• Filed: 12/14/2011
• Published: 06/20/2013
• Est. Priority Date: 12/14/2011
• Status: Active Grant

• Current Assignee: Brainspace Corporation (Reveal Data Corp.)
• Original Assignee: PureDiscovery Corporation (Reveal Data Corp.)
• Inventors: Jakubik, Paul A.
• Granted Patent: US 8,949,111 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

A method includes accessing text that includes a plurality of words, tagging each of the plurality of words with one of a plurality of parts of speech (POS) tags, and creating a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag. The method further includes clustering one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens, and forming a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.

Claims

1. A system, comprising:

one or more memory units; and

one or more processing units operable to;

access text comprising a plurality of words;

tag each of the plurality of words with one of a plurality of parts of speech (POS) tags;

create a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag;

cluster one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens; and

form a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.

2. The system of claim 1, wherein the plurality of POS tags comprises:

a POS tag for a proper noun that is a geographic name;

a POS tag for a proper noun that is a name of a person; and

a POS tag for a proper noun that is can be both a geographic name and a name of a person.

3. The system of claim 1, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises utilizing a lexical analyzer to determine whether each of the plurality of words is certain type of entity, the certain type of entity chose from a plurality of entities comprising:

an acronym;

a plurality of initials;

a uniform resource locator (URL); and

an email address.

4. The system of claim 1, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises searching a lexicon for each of the plurality of words, the lexicon mapping a known set of words to POS tags.

5. The system of claim 1, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises applying a set of heuristics.

6. The system of claim 1, the one or more processing units further operable to provide a count of the number of the words in the text, the count based on the plurality of created tokens.

7. The system of claim 1, wherein clustering one or more of the created tokens into the chunk of tokens comprises clustering tokens having the same POS tags into the chunk of tokens.

8. A computer-implemented method, comprising:

accessing text by a processing system, the text comprising a plurality of words;

tagging, by the processing system, each of the plurality of words with one of a plurality of parts of speech (POS) tags;

creating, by the processing system, a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag;

clustering, by the processing system, one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens; and

forming, by the processing system, a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.

9. The computer-implemented method of claim 8, wherein the plurality of POS tags comprises:

a POS tag for a proper noun that is a geographic name;

a POS tag for a proper noun that is a name of a person; and

a POS tag for a proper noun that is can be both a geographic name and a name of a person.

10. The computer-implemented method of claim 8, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises utilizing a lexical analyzer to determine whether each of the plurality of words is certain type of entity, the certain type of entity chose from a plurality of entities comprising:

an acronym;

a plurality of initials;

a uniform resource locator (URL); and

an email address.

11. The computer-implemented method of claim 8, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises searching a lexicon for each of the plurality of words, the lexicon mapping a known set of words to POS tags.

12. The computer-implemented method of claim 8, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises applying a set of heuristics.

13. The computer-implemented method of claim 8, further comprising providing a count of the number of the words in the text, the count based on the plurality of created tokens.

14. The computer-implemented method of claim 8, wherein clustering one or more of the created tokens into the chunk of tokens comprises clustering tokens having the same POS tags into the chunk of tokens.

15. A non-transitory computer-readable medium comprising software, the software when executed by one or more processing units operable to perform operations comprising:

accessing text comprising a plurality of words;

tagging each of the plurality of words with one of a plurality of parts of speech (POS) tags;

creating a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag;

clustering one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens; and

forming a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.

16. The non-transitory computer-readable medium of claim 15, wherein the plurality of POS tags comprises:

a POS tag for a proper noun that is a geographic name;

a POS tag for a proper noun that is a name of a person; and

a POS tag for a proper noun that is can be both a geographic name and a name of a person.

17. The non-transitory computer-readable medium of claim 15, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises searching a lexicon for each of the plurality of words, the lexicon mapping a known set of words to POS tags.

18. The non-transitory computer-readable medium of claim 15, wherein tagging each of the plurality of words with one of the plurality of POS tags comprises applying a set of heuristics.

19. The non-transitory computer-readable medium of claim 15, further comprising providing a count of the number of the words in the text, the count based on the plurality of created tokens.

20. The non-transitory computer-readable medium of claim 15, wherein clustering one or more of the created tokens into the chunk of tokens comprises clustering tokens having the same POS tags into the chunk of tokens.