System and method of textual information analytics

• US 20060277465A1
• Filed: 07/18/2005
• Published: 12/07/2006
• Est. Priority Date: 06/07/2005
• Status: Active Grant

• Current Assignee: Textual Analytics Solutions
• Original Assignee: Textual Analytics Solutions
• Inventors: Deshmukh, Sanjay, Bhatia, Vaibhav, Pandit, Madan
• Granted Patent: US 7,689,557 B2
• Time in Patent Office: Days
• US Class Current: 715/234
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

This invention provides a method and system for analyzing and deriving analytical insights from textual information. The information structuration process determines the structure in which the text information is rendered. A cyclical extraction process using the parameters of co-frequency, co-dependency, and co-relatedness and parts of speech, determines various textual aspects and their subcomponents such as themes and dimensions. Using the subcomponents of the textual aspects, relationship maps are built, disambiguated and ranked. A text analytics and decision support matrix is created using the ranked relations, thereby providing a highly relevant result set to the user'"'"'s information need. A multidimensional navigation matrix is created that helps a user navigate across dimensions.

Claims

1. A method of analyzing and deriving analytical insights from a given textual information database, comprising the steps of:

conducting information structuration for deriving information structures;

deriving phrase chunks through shallow text parsing, wherein the step of shallow text parsing further comprises parts of speech tagging and the application of said information structures;

conducting cyclical processing for deriving the structured, unstructured and binding textual aspects from said phrase chunks;

constructing relationship maps connecting said structured, unstructured and binding textual aspects;

ranking textual aspects using said relationship maps; and

creating text analytics and decision support matrix using the relevancy ranked textual aspects.

2. The method of claim 1, wherein the information structure types are further divided into dense text types, non-dense text types, uniform resource locators URL patterns, title patterns, text structure patterns and presentation patters.

3. The method of claim 1, wherein the step of cyclical processing further comprises cyclically processing relatedness, frequency, parts of speech, qualifier qualifieds, dependency and dimension pattern.

4. The method of claim 1, wherein the step of deriving the structured, unstructured and binding textual aspects from the phrase chunks, further comprises the steps of:

measuring the precision value of the dimensions extracted;

editing the rules and reapplying them to generate a new set of dimensions if the precision or recall value is below a required prescribed value; and

processing the textual aspects by a set of processing elements cyclically until a minimum output resolution is achieved.

5. The method of claim 1, wherein the step of determining structured, unstructured and binding structural aspects comprises the step of disambiguating the relations among the final set of structured, unstructured and binding structural aspects.

6. The method of claim 5, where in the step of disambiguating comprises the determination of space tags, time tags, qualifier tag and action tags.

7. The method of claim 1, wherein the step of determining the highest ranked textual aspects comprises:

determining the agentive dimension, and attributive dimension from the structural textual aspects;

determining the themes, dynamic dimension, actions, qualifiers and binding textual aspects from unstructured textual aspects;

building relationship maps for said agentive dimension, dynamic dimension, attributive dimension, themes, actions and qualifiers.

8. The method of claim 1, wherein the step of determining a set of structured, unstructured and binding structural aspects comprises:

generating seed rules for dimension extraction and generating seed textual structural aspects, seed unstructured textual aspects and seed binding structural aspects; and

applying said seed rules and determining a larger set of the seeded textual aspects;

conducting a wider and deeper crawl by using the ranked textual aspects; and

repeating the process of identifying a set of textual aspects.

9. The method of claim 1, wherein the step of determine the relationship maps between textual themes, textual actions and textual dimensions comprises determining the co-frequency, co-occurrence, co-dependency, co-relatedness measures of said textual themes, textual actions and textual dimensions either within a document or across documents in a textual database of document corpus.

10. The method of claim 1, wherein the step of creating seed dimensions comprises:

templatizing and automatically capturing human annotation;

generating humanly created rules; and

combining said humanly created rules and semi-automated computer generated predictive rules for extracting and processing textual dimensions from a document or a document corpus.

11. The method of claim 1, wherein textual dimensions are time, number, location, event, people roles, organizations, brand names and products, scientific and technology names and terminologies, quotations and attribution, explanations, arguments, opinions, suggestions, findings, assumptions, relations, associations, negations, emotions, sentiments, oppositions, contradictions, contrasts, qualities, attributes, commands, directives, order, rulings, measurements including scientific, technical and commercial and general measures, names of natural and man-made living beings like plants, animals, genes, clones, names of natural and manmade things, substances like chemicals, drugs, monuments, names of arts, culture, sports, and other terms and terminologies, causes, membership, substance, fact, non-facts, type and kind, entailment, part whole, contrast, metaphor, metonymy, similarity and compare.

12. The method of claim 1, wherein the determination of the measure of co-relatedness further comprises the steps of:

detecting word form for allowing alternative forms of a word to be included to determine relatedness of themes;

detecting qualification positioning, wherein one or two words qualify one or more words for determining the closeness of textual themes;

detecting the variation in sense form, wherein two different dictionary senses of words determine the closeness of textual themes.

13. The method of claim 9, wherein the step of measuring co-dependency comprises:

measuring intra-dimensional dependency, wherein a particular dimension may have a dominant form or a dependent form, further comprising one of more of the following steps of;

splitting a dimension into a shorter form;

mutating a dimension by fusing it with another dimension;

extending a dimension based on its procedural occurrence;

measuring inter-dimensional dependency, wherein dependency between two or more textual dimensions is determined and extracted; and

measuring contextual dependency, wherein dependency between two dimensions within a single dimension is determined by their contextual, grammatical and positional relationships.

14. The method of claim 9, wherein the measure of co-frequency comprises the steps of:

measuring the co-frequency of textual segments that are frequent within a document and frequent across documents;

measuring the co-frequency of textual segments that are frequent within a document but not frequent across documents;

measuring the co-frequency of textual segments that are not frequent within a document but are frequent across documents; and

measuring the co-frequency of textual segments that are not frequent within a document and are not frequent across documents.

15. The method of claim 1, wherein the step of determining the most relevant textual aspects by ranking, comprises the steps of:

determining the count of relations that each textual aspect forms with every other textual aspect;

determining the summation of the strength of each individual relation, wherein the strength is based on a measure of co-occurrence, co-relatedness and co-dependency at a clause, sentence, tri-sentence, paragraph and document and document corpus level;

counting the number of qualifiers associated with each textual aspect; and

measuring the frequency of textual aspects within and across boundaries of sentence, tri-sentence, paragraph, document and document corpus level.

16. The method of claim 1, wherein the step of ranking the textual aspects further includes the steps of ranking themes, dimensions, qualifiers, actions, sentences, clauses, tri-sentences, paragraphs, document and document corpus level.

17. A method of navigating, comprising:

navigating across a multidimensional matrix, wherein dimensions include one or more of the following;

time, number, location, event, people roles, organizations, brand names and products, scientific and technology names and terminologies, quotations and attribution, explanations, arguments, opinions, suggestions, findings, assumptions, relations, associations, negations, emotions, sentiments, oppositions, contradictions, contrasts, qualities, attributes, commands, directives, order, rulings, measurements including scientific, technical and commercial and general measures, names of natural and manmade living beings like plants, animals, genes, clones, names of natural and manmade things, substances like chemicals, drugs, monuments, names of arts, culture, sports, and other terms and terminologies, causes, membership, substance, fact, non facts, type and kind, entailment, part whole, contrast, metaphor, metonymy, similarity, and compare;

and wherein the multi-dimensional matrix is ranked to facilitate a user to select the most effective dimension set to navigate with.

18. A system for analyzing and deriving analytical insights from a given textual information database, comprising:

visual information extraction and structuration engine for conducting information structuration;

cyclical extractor for conducting cyclical processing and deriving phrase chunks from said information structures and deriving the structured, unstructured and binding textual aspects from the phrase chunks;

relationship builder for constructing relationship maps connecting said structured, unstructured and binding textual aspects;

ranking module for ranking textual aspects using said relationship maps; and

text analytics and decision support creator for creating text analytics and decision support matrix using the relevancy ranked textual aspects.