METHOD AND SYSTEM FOR TEXT SUMMARIZATION AND SUMMARY BASED QUERY ANSWERING

• US 20100287162A1
• Filed: 03/28/2009
• Published: 11/11/2010
• Est. Priority Date: 03/28/2008
• Status: Abandoned Application

• Current Assignee: Sanika Shirwadkar
• Original Assignee: Sanika Shirwadkar
• Inventors: Shirwadkar, Sanika
• Application Number:
  US12/413,518
• Publication Number:
  US 20100287162A1
• Time in Patent Office: Days
• US Class Current: 707/740
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A method and system for generating answers to questions based on electronic data summary which is itself derived on context and semantics of a corpus of authoritative documents and its subsequent usage is disclosed. The method and system provides for generating a taxonomy of concepts, assigning unique-identifiers and weights to the taxonomy concepts using a given corpus of electronic data, using the taxonomy to identify the semantics of the document to be summarized, generating an ontology from a summarized authoritative text, having the ontology generation and the summary generation in a feedback loop, selecting text from a given document based on the weights of unique-identifiers in the taxonomy/ontology, sentences as a summary and pruning of the list based upon an entropy threshold, and the presence of a probability distribution, publishing of the summary in a known format on server or any other software/hardware platform with or without monetization for consumption, usage of the summary to generate answers which can be configured using an ontology and thus prevent denial of information/information overload.

Claims

1. A method comprising:

summarizing text(s), and using the generated single-source or multi-source summary as an answer to an input query, where summarization is based on a pre-generated context of a summarized authoritative texts corpus,

whereby said summary or answer prevents ‘

denial of information’ and

information overload when plurality of electronic data is available.

2. The method of claim 1, wherein the generation of said context further comprises:

classifying input document key terms into unique concept groups using a taxonomy, where each concept is represented by a root term, and the associated terms are represented as leaf terms;

creating an ontology of terms and their inter-associations based on selective authoritative text;

calculating term weights based on ontology and taxonomy;

putting the context generation process in a feedback loop with the ontology generation process, resulting in term weights'"'"' refinement in each round of summarization, until an optimal weights'"'"' stability is achieved; and

summarizing the documents in the corpus and storing it for further use.

3. The method of claim 2, wherein the said Ontology creation process comprises:

cleaning the corpus;

parsing the corpus using Natural Language Parsing techniques such as part of speech tagging;

extracting and categorizing the key terms;

concepts identification;

relationships identification and terms association using techniques such as discovery of relations using existing databases such as Wordnet, association rule mining, terms'"'"' co-occurrence frequency based associations, frequency based discovery and associations for compound terms or phrases, relating compound nouns using the head noun term, etc.; and

ontology construction and storage.

4. The method of claim 2, wherein the calculation of term weights comprises:

leaf term global weights calculation using the pre-calculated weights of the associated root terms along with the leaf term'"'"'s ontology;

using calculated leaf term weights to refine the global weights of the associated root terms;

calculating global weights for terms not present in the taxonomy using only ontology; and

calculating local weights of the constituent terms in a document by using the local frequency and global weights of the terms.

5. The method of claim 1, wherein said summary generation further comprises of:

generating a list of sentences with corresponding weights that are calculated by using the constituent words'"'"' weights;

pruning sentences based on sentence entropy that is calculated using factors such as number of weighted words in the sentence, total number of words in the sentence and the sentence weight;

pruning of said sentences based on the document structure and the sentence position in the document;

pruning of the said sentences based on a given probability distribution;

selection of unique identifiers in a sentence that at least are also present in adjacent sentences and belong to the same semantic-group whereby adjacent sentences have related meaning;

parsing the sentences using Natural Language Parsing techniques to identify their grammatical content for making similar sentences adjacent;

replacing the sentences by similar sentences from the context database based on multitude of factors and, adding them together to form a summary.

6. The method of claim 1, wherein the said context is used for;

summarizing a new document that is not present in the corpus; and

creating multi-document summary as an answer to a user query.

7. The method of claim 5, wherein the answer generation comprises of:

using a variety of methods including Vector Space and Clustering and choosing the sentences based on a voting mechanism;

refining answers by using an available ontology of concepts, and by improving the answer-generation mechanism that in turn uses optimization techniques on the metrics provided by tools that compare user summaries with machine generated summaries;

using the top n summaries that match the question where n is smaller than the total number of retrieved results (N) for creating an answer.

8. The method of claim 5 wherein said summary and answer can vary depending upon the social network to which the initiating user belongs;

and the language of the user.

9. The method of claim 1, wherein the storage, retrieval and usage of said summary further comprises:

storing said summary in multitude of ways that at least include storage in a database against a Universal Resource Identifier and embedding in the structure of said electronic data to be summarized;

indexing the summaries using a search engine;

re-formatting of said summary for suitable access by resource locators/search engines/browsers, and for display on request whereby saving user time and computing resources;

displaying summary near the corresponding URI (Uniform Resource Identifier);

displaying said summary based on user preferences at least comprising of means of traversal from simpler summaries to complex summaries and from one summary to another using associated Uniform Resource Identifiers;

storing said summary on a server or any other software/hardware platform and using it in a multitude of ways that at least include searching summaries based on input key text;

monetizing publishing/consuming of summaries;

modifying or ranking summary based on user input and publishing as per user request; and

displaying similar summaries of different electronic data by resource locators/search engines/browsers;

combining said summary with at least summary of similar electronic data and user provided summary to create a topic-summary.

10. A system comprising:

Means adapted for summarizing text(s), and using the generated summary as an answer to an input query, where summarization is based on a pre-generated context of a summarized authoritative texts corpus,

whereby said summary or answer prevents ‘

denial of information’ and

information overload when plurality of electronic data is available.

11. The system of claim 10, wherein the generation of said context further comprises:

classifying input document key terms into unique concept groups using a taxonomy, where each concept is represented by a root term, and the associated terms are represented as leaf terms;

creating an ontology of terms and their inter-associations based on selective authoritative text;

calculating term weights based on ontology and taxonomy;

putting the context generation process in a feedback loop with the ontology generation process, resulting in term weights'"'"' refinement in each round of summarization, until an optimal weights'"'"' stability is achieved; and

summarizing the documents in the corpus and storing it for further use.

12. The system of claim 11, wherein the said Ontology creation process comprises:

means adapted for cleaning the corpus;

means adapted for parsing the corpus using Natural Language Parsing techniques such as part of speech tagging;

means adapted for extracting and categorizing the key terms;

means adapted for concepts identification;

means adapted for relationships identification and terms association using techniques such as discovery of relations using existing databases such as Wordnet, association rule mining, terms'"'"' co-occurrence frequency based associations, frequency based discovery and associations for compound terms or phrases, relating compound nouns using the head noun term, etc.; and

ontology construction and storage.

13. The system of claim 11, wherein said term weights'"'"' calculation comprises of:

means adapted for leaf term global weights calculation using the pre-calculated weights of the associated root terms along with the leaf term'"'"'s ontology;

means adapted for using calculated leaf term weights to refine the global weights of the associated root terms;

means adapted for calculating global weights for terms not present in the taxonomy using only ontology; and

means adapted for calculating local weights of the constituent terms in a document by using the local frequency and global weights of the terms.

14. The system of claim 10, wherein said summary generation means further comprises of:

means adapted for generating a list of sentences with corresponding weights that are calculated by using the constituent words'"'"' weights;

means adapted for pruning sentences based on sentence entropy that is calculated using number of weighted words in the sentence, total number of words in the sentence and the sentence weight;

means adapted for pruning of said sentences based on the document structure and the sentence position in the document;

means adapted for pruning of the said sentences based on a given probability distribution;

means adapted for selecting unique identifiers in a sentence that at least are also present in adjacent sentences and belong to the same semantic-group whereby adjacent sentences have related meaning;

means adapted for parsing the sentences using Natural Language Parsing techniques to identify their grammatical content for making similar sentences adjacent;

means adapted for replacing the sentences by similar sentences from the context database based on multitude of factors and, adding them together to form a summary.

15. The system of claim 10, wherein the usage means of the said context comprises:

means adapted for summarizing a new document that is not present in the corpus; and

means adapted for creating multi-document summary as an answer to a user query.

16. The system of claim 15, wherein the answer generation comprises of:

means adapted for using a variety of methods including Vector Space and Clustering and choosing the sentences based on a voting mechanism;

means adapted for refining answers by using an available ontology of concepts, and by improving the answer-generation mechanism that in turn uses optimization techniques on the metrics provided by tools that compare user summaries with machine generated summaries;

means adapted for using the top n summaries that match the question where n is smaller than the total number of retrieved results (N) for creating an answer.

17. The system of claim 15, wherein said summary and answer can vary depending upon the social network to which the initiating user belongs;

and the language of the user.

18. The system of claim 10, wherein the storage, retrieval and usage of said summary further comprises:

means adapted for storing said summary in multitude of ways that at least include storage in a database against a Universal Resource Identifier and embedding in the structure of said electronic data to be summarized;

means adapted for indexing the summaries using a search engine ;

means adapted for re-formatting of said summary for suitable access by resource locators/search engines/browsers, and for display on request whereby saving user time and computing resources;

means adapted for displaying summary near the corresponding URI (Uniform Resource Identifier);

means adapted for displaying said summary based on user preferences at least comprising of means of traversal from simpler summaries to complex summaries and from one summary to another using associated Uniform Resource Identifiers;

storing said summary on a server or any other software/hardware platform and using it in a multitude of ways that at least include searching summaries based on input key text;

monetizing publishing/consuming of summaries;

modifying or ranking summary based on user input and publishing as per user request; and

displaying similar summaries of different electronic data by resource locators/search engines/browsers;

means adapted for combining said summary with at least summary of similar electronic data and user provided summary to create a topic-summary.

19. A computer readable medium of instructions comprising:

instructions for summarizing text(s), and using the generated single-source or multi-source summary as an answer to an input query, where summarization is based on a pre- generated context of a summarized authoritative texts corpus,

whereby said summary or answer prevents ‘

denial of information’ and

information overload when plurality of electronic data is available.

20. The computer readable medium of instructions of claim 19, wherein the generation of said context further comprises:

instructions for classifying input document key terms into unique concept groups using a taxonomy, where each concept is represented by a root term, and the associated terms are represented as leaf terms;

instructions for creating an ontology of terms and their inter-associations based on selective authoritative text;

calculating term weights based on ontology and taxonomy;

instructions for putting the context generation process in a feedback loop with the ontology generation process resulting in term weight'"'"'s refinement in each round of summarization, until an optimal weights'"'"' stability is achieved; and

instructions for summarizing the documents in the corpus and storing it for further use.

21. The computer readable medium of instructions of claim 20, wherein the said Ontology creation process comprises:

instructions for cleaning the corpus;

instructions for parsing the corpus using Natural Language Parsing techniques such as part of speech tagging;

instructions for extracting and categorizing the key terms;

instructions for concepts identification;

instructions for relationships identification and terms association using techniques such as discovery of relations using existing databases such as Wordnet, association rule mining, terms'"'"' co-occurrence frequency based associations, frequency based discovery and associations for compound terms or phrases, relating compound nouns using the head noun term, etc.; and

ontology construction and storage.

22. The computer readable medium of instructions of claim 20, wherein the calculation of term weights comprises:

instructions for leaf term global weights calculation using the pre-calculated weights of the associated root terms along with the leaf term'"'"'s ontology;

instructions for using calculated leaf term weights to refine the global weights of the associated root terms;

instructions for calculating global weights for terms not present in the taxonomy using only ontology; and

instructions for calculating local weights of the constituent terms in a document by using the local frequency and global weights of the terms.

23. The computer readable medium of instructions of claim 19, wherein said summary generation further comprises of:

instructions for generating a list of sentences with corresponding weights that are calculated by using the constituent words'"'"' weights;

instructions for pruning sentences based on sentence entropy, that is calculated using the number of weighted words in the sentence, total number of words in the sentence and the sentence weight;

instructions for pruning of said sentences based on the document structure and the sentence position in the document;

instructions for pruning of the said sentences based on a given probability distribution;

instructions for selection of unique identifiers in a sentence that at least are also present in adjacent sentences and belong to the same semantic-group whereby adjacent sentences have related meaning;

instructions for parsing the sentences using Natural Language Parsing techniques to identify their grammatical content for making similar sentences adjacent;

replacing the sentences by similar sentences from the context database based on multitude of factors and, instructions for adding them together to form a summary

24. The computer readable medium of instructions of claim 19, wherein the said context is used for;

instructions for summarizing a new document that is not present in the corpus; and

instructions for creating multi-document summary as an answer to a user query.

25. The computer readable medium of instructions of claim 24, wherein the answer generation comprises of:

instructions for using a variety of methods including Vector Space and Clustering and choosing the sentences based on a voting mechanism;

instructions for refining answers by using an available ontology of concepts, and by improving the answer-generation mechanism that in turn uses optimization techniques on the metrics provided by tools that compare user summaries with machine generated summaries;

instructions for using the top n summaries that match the question where n is smaller than the total number of retrieved results (N) for creating an answer.

26. The computer readable medium of instructions of claim 24, wherein said summary and answer can vary depending upon the social network to which the initiating user belongs;

and the language of the user.

27. The computer readable medium of instructions of claim 19, wherein the storage, retrieval and usage of said summary further comprises:

instructions for storing said summary in multitude of ways that at least include storage in a database against a Universal Resource Identifier and embedding in the structure of said electronic data to be summarized;

instructions for indexing the summaries using a search engine;

instructions for re-formatting of said summary for suitable access by resource locators/search engines/browsers, and for display on request whereby saving user time and computing resources;

instructions for displaying summary near the corresponding URI (Uniform Resource Identifier);

instructions for displaying said summary based on user preferences at least comprising of means of traversal from simpler summaries to complex summaries and from one summary to another using associated Uniform Resource Identifiers;

instructions for storing said summary on a server or any other software/hardware platform and using it in a multitude of ways that at least include searching summaries based on input key text;

monetizing publishing/consuming of summaries;

instructions for modifying or ranking summary based on user input and publishing as per user request; and

displaying similar summaries of different electronic data by resource locators/search engines/browsers;

combining said summary with at least summary of similar electronic data and user provided summary to create a topic-summary.