Identification and Extraction of New Terms in Documents

• US 20130246045A1
• Filed: 03/14/2012
• Published: 09/19/2013
• Est. Priority Date: 03/14/2012
• Status: Abandoned Application

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Ulanov, Alexander, Simanovsky, Andrey
• Application Number:
  US13/420,149
• Publication Number:
  US 20130246045A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking

Abstract

A method and apparatus that can extract new terms from documents for inclusion in a vocabulary collection is disclosed. A document may be parsed to obtain an n-gram phrase indicative of a new term. The phrase may include a plurality of words. The n-gram phrase may be decomposed into a series of bi-gram phrases each including a first and a second phrase part. The first and second phrase parts each include at least one word. It may then be determined whether the first or second phrase part is in a vocabulary collection. If not, it may be estimated as to the probability that the bi-gram phrase should be in the vocabulary collection. The bi-gram phrase may be added to the vocabulary collection if the probability that the bi-gram phrase should be in the vocabulary collection exceeds a minimum threshold level.

Claims

1. A method comprising:

parsing a document to obtain an n-gram phrase indicative of a new term, the phrase comprised of a plurality of words;

breaking the n-gram phrase into a bi-gram phrase comprised of a first and a second phrase part, the first and second phrase part including at least one word;

determining whether the first or second phrase part is in a vocabulary collection;

estimating the probability that the bi-gram phrase should be in the vocabulary collection if it is not; and

adding the bi-gram phrase to the vocabulary collection if the probability that the bi-gram phrase should be in the vocabulary collection exceeds a minimum threshold level.

2. The method of claim 1, the breaking the n-gram phrase into a bi-gram phrase comprising:

decomposing the n-gram phrase into all possible unique bi-gram phrases to create multiple first and second phrase part combinations.

3. The method of claim 2, the determining whether the first or second phrase part is in a vocabulary collection comprising:

for each first and second phrase part combination;

searching the vocabulary collection for vocabulary collection phrases that include the first or second phrase part; and

restricting the search to vocabulary collection phrases that are similar to the first or second phrase part based on a similarity function.

4. The method of claim 3, the estimating the probability that the bi-gram phrase should be in the vocabulary collection comprising:

performing a probability calculation taking into consideration a similarity strength and a collocation strength between the first and second phrase part.

5. The method of claim 1, the vocabulary collection comprising a thesaurus.

6. The method of claim 1, the vocabulary collection comprising a dictionary.

7. The method of claim 1, the vocabulary collection comprising a glossary.

8. An apparatus comprising:

a processor circuit;

a memory;

a parsing module stored in the memory and executable by the processor circuit, the parsing module to obtain an n-gram phrase indicative of a new term, the phrase comprised of a plurality of words;

a decomposition module stored in the memory and executable by the processor circuit, the decomposition module to break the n-gram phrase into a bi-gram phrase comprised of a first and a second phrase part, the first and second phrase part including at least one word;

a phrase determination module stored in the memory and executable by the processor circuit, the phrase determination module to determine whether the first or second phrase part is in a vocabulary collection; and

a probability module stored in the memory and executable by the processor circuit, the probability module to estimate the probability that the bi-gram phrase should be in the vocabulary collection if it is not; and

add the bi-gram phrase to the vocabulary collection if the probability that the bi-gram phrase should be in the vocabulary collection exceeds a minimum threshold level.

9. The apparatus of claim 8,

the decomposition module to decompose the n-gram phrase into all possible unique bi-gram phrases to create multiple first and second phrase part combinations;

and

the phrase determination module to;

search the vocabulary collection for vocabulary collection phrases that include the first or second phrase part for each first and second phrase part combination; and

restrict the search to vocabulary collection phrases that are similar to the first or second phrase part based on a similarity function.

10. The apparatus of claim 9, the probability module to perform a probability calculation taking into consideration a similarity strength and a collocation strength between the first and second phrase part.

11. The apparatus of claim 9, the vocabulary collection comprising one of a thesaurus, a dictionary, or a glossary.

12. An article of manufacture comprising a non-transitory computer-readable storage medium containing instructions that if executed enable a system to:

parse a document to obtain an n-gram phrase indicative of a new term, the phrase comprised of a plurality of words;

break the n-gram phrase into a bi-gram phrase comprised of a first and a second phrase part, the first and second phrase part including at least one word;

determine whether the first or second phrase part is in a vocabulary collection;

estimate the probability that the bi-gram phrase should be in the vocabulary collection if it is not; and

add the bi-gram phrase to the vocabulary collection if the probability that the bi-gram phrase should be in the vocabulary collection exceeds a minimum threshold level.

13. The article of claim 12, further comprising instructions that if executed enable the system to:

decompose the n-gram phrase into all possible unique bi-gram phrases to create multiple first and second phrase part combinations; and

for each first and second phrase part combination;

searching the vocabulary collection for vocabulary collection phrases that include the first or second phrase part; and

restricting the search to vocabulary collection phrases that are similar to the first or second phrase part based on a similarity function.

14. The article of claim 13, further comprising instructions that if executed enable the system to perform a probability calculation taking into consideration a similarity strength and a collocation strength between the first and second phrase part.

15. The article of claim 14, the vocabulary collection comprising one of a thesaurus, a dictionary, or a glossary.