Methods and systems for extracting keyphrases from natural text for search engine indexing

• US 9,390,161 B2
• Filed: 09/24/2010
• Issued: 07/12/2016
• Est. Priority Date: 09/25/2009
• Status: Active Grant

• Current Assignee: Fakhri Karray, Mohamed Salem Kamel, Shady Shehata
• Original Assignee: Fakhri Karray, Mohamed Salem Kamel, Shady Shehata
• Inventors: Shehata, Shady, Karray, Fakhri, Kamel, Mohamed Salem
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Seck, Ababacar
• Application Number:
  US13/497,982
• Publication Number:
  US 20120246100A1
• Time in Patent Office: 2,118 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

The present invention is a method and system for the extraction of keyphrases from natural text. For the purpose of this document, keyphrases are text segments that represent the main topic of a text. The method of the present invention may facilitate keyphrase extraction from any length of text. The text may be of several varieties, such as, for example a sentence, paragraph, document or collection of documents. Phrase separator methods may be applied to the text to extract phrases from the text. From these phrases the present invention may identify the one or more phrases that are integral to the meaning of the text and these may be identified as the keyphrases of the text. The text may be indexed using the keyphrases so that a search based upon any of the keyphrases will cause search engines and/or text retrieval means to retrieve the text.

Claims

1. A computer implemented method for extracting keyphrases from natural text, characterized in that it comprises:

(a) generating one or more phrases in the natural text based on one or more phrase separators in the natural text, wherein each of the one or more phrase separators comprises one or more words from the natural text;

(b) assigning a weight to each of the one or more phrases in the natural text based on its frequency in the semantic frames of one or more sentences of the natural text, wherein each of the one or more sentences is divided into one or more sub-texts by the one or more phrase separators, and the assigned weight is calculated for each of the one or more phrases based on a frequency of the respective phrase within the one or more sub-texts of each sentence; and

(c) ranking the one or more phrases based on their weights to extract one or more keyphrases having the highest ranks.

2. The method of claim 1, further comprising generating the one or more phrase separators utilizing an intelligent classifier, wherein the intelligent classifier generates the one or more phrase separators from the natural text by identifying portions of the natural text as either phrase separators or not phrase separators.

3. The method of claim 2, characterized in that it further comprises training the intelligent classifier using one or more training texts, whereby the intelligent classifier is adapted to learn to recognize phrase separators.

4. The method of claim 3, characterized in that it further comprises teaching the intelligent identifier to identify phrase separators in training texts based on one or more of word position, part of speech tagging, word type, and features or parts of text.

5. The method of claim 4, characterized in that it further comprises utilizing a part of speech tagging model including a knowledge-based dictionary to identify phrase separators based on part of speech tags allotted within the natural text.

6. The method of claim 5, characterized in that it further comprises utilizing the knowledge-based dictionary to build a hash table for each word in the natural text.

7. The method of claim 6, characterized in that it further comprises evaluating each word in accordance with a lookup table as to whether the word has one or more part of speech tags.

8. The method of claim 1, characterized in that it further comprises:

applying the one or more phrase separators to split the natural text based on heuristic rules; and

generating the one or more phrases.

9. The method of claim 8, characterized in that it further comprises calculating the frequency of each of the one or more phrases as it occurs within the one or more generated phrases.

10. The method of claim 9, characterized in that it further comprises calculating the frequency of each of the one or more phrases based on words that appear in each phrase.

11. The method of claim 9, characterized in that the frequency of a phrase p with a scope is calculated by

o weight=freq(p,scope), where scope={sent,parag,doc,collec}.

12. The method of claim 10, characterized in that the average frequency of phrase p that appears in the natural text is calculated by

13. The method of claim 1, characterized in that it further comprises weighting the phrases based on context of the phrase to the natural text, whereby a probability value is assigned to each phrase based on the frequency of the phrase in the text, and the assigned probability value is used to generate a score for each phrase.

14. The method of claim 13, characterized in that it further comprises ranking the weighted phrases in accordance with their weights to extract keyphrases from the phrases.

15. The method of claim 1, characterized in that it further comprises applying an intelligent summarizer to highlight the positions of keyphrases in the natural text and produce a summary of the natural text based on the positions of the keyphrases.

16. The method of claim 15, characterized in that it further comprises extracting text segments containing keyphrases with a high score as a text summary of the natural text.

17. The method of claim 1, characterized in that it further comprises incorporating an intelligent indexer to extract keyphrases related to one or more natural texts to build a phrase-based index.

18. The method of claim 17, characterized in that it further comprises utilizing one or more text retrieval means to provide higher scores to query words that appear in the phrase-based index.

19. The method of claim 18, characterized in that the text retrieval means is a search engine and the query is a search term or search string.

20. The method of claim 19, characterized in that it further comprises conducting a search of the phrase-based index for matches or near-matches between keyphrases and the query.

21. A computer implemented method for extracting keyphrases from natural text, characterized in that it comprises:

(a) generating one or more phrases in the natural text based on one or more phrase separators in the natural text, wherein each of the one or more phrase separators comprises one or more words from the natural text;

(b) identifying semantic frames that are associated with the one or more phrase separators and analyzing the semantic frames so as to associate with one another phrases that have a related meaning;

(c) assigning a weight to each of the one or more phrases in the natural text based on its frequency in the semantic frames of one or more sentences of the natural text and also based on the associations between each phrase and other phrases based on related meaning, wherein each of the one or more sentences is divided into one or more sub-texts by the one or more phrase separators, and the assigned weight is calculated for each of the one or more phrases based on a frequency of the respective phrase within the one or more sub-texts of each sentence; and

(d) ranking the one or more phrases based on their weights to extract one or more keyphrases having the highest ranks.

22. A system having a processor and memory adapted to perform a method comprising the steps of:

generating one or more phrases in the natural text based on an identification of one or more phrase separators in the natural text, wherein each of the one or more phrase separators comprises one or more words from the natural text;

identifying semantic frames that are associated with the one or more phrase separators and analyzing the semantic frames so as to associate with one another phrases that have a related meaning;

assigning a weight to each of the one or more phrases in the natural text based on its frequency in the semantic frames of one or more sentences of the natural text and also based on the associations between each phrase and other phrases based on related meaning, wherein each of the one or more sentences is divided into one or more sub-texts by the one or more phrase separators, and the assigned weight is calculated for each of the one or more phrases based on a frequency of the respective phrase within the one or more sub-texts of each sentence; and

ranking the one or more phrases based on their weights to extract one or more keyphrases having the highest ranks.

23. A non-transitory computer readable storage medium storing a set of computer program instructions, which when executed by a processor, causes a computer device to perform a method comprising the steps of:

generating one or more phrases in the natural text based on one or more phrase separators in the natural text, wherein each of the one or more phrase separators comprises one or more words from the natural text;

identifying semantic frames that are associated with the one or more phrase separators and analyzing the semantic frames so as to associate with one another phrases that have a related meaning;

assigning a weight to each of the one or more phrases in the natural text based on its frequency in the semantic frames of one or more sentences of the natural text and also based on the associations between each phrase and other phrases based on related meaning, wherein each of the one or more sentences is divided into one or more sub-texts by the one or more phrase separators, and the assigned weight is calculated for each of the one or more phrases based on a frequency of the respective phrase within the one or more sub-texts of each sentence; and

ranking the one or more phrases based on their weights to extract one or more keyphrases having the highest ranks.