System and method of textual information analytics

• US 7,689,557 B2
• Filed: 07/18/2005
• Issued: 03/30/2010
• Est. Priority Date: 06/07/2005
• Status: Active Grant

• Current Assignee: Textual Analytics Solutions
• Original Assignee: Textual Analytics Solutions
• Inventors: Deshmukh, Sanjay, Bhatia, Vaibhav, Pandit, Madan
• Primary Examiner(s): Vital; Pierre M
• Assistant Examiner(s): Mitiku; Berhanu
• Application Number:
  US11/183,657
• Publication Number:
  US 20060277465A1
• Time in Patent Office: 1,716 Days
• Field of Search: None
• US Class Current: 707/771
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

This invention provides a method and system for analyzing and deriving analytical insights from textual information. The information structuration process determines the structure in which the text information is rendered. A cyclical extraction process using the parameters of co-frequency, co-dependency, and co-relatedness and parts of speech, determines various textual aspects and their subcomponents such as themes and dimensions. Using the subcomponents of the textual aspects, relationship maps are built, disambiguated and ranked. A text analytics and decision support matrix is created using the ranked relations, thereby providing a highly relevant result set to the user'"'"'s information need. A multidimensional navigation matrix is created that helps a user navigate across dimensions.

Claims

1. A method of creating a text analytics and decision support matrix by analyzing and deriving analytical insights from a given textual information database using a textual information processor, said textual information processor performing the steps of:

conducting information structuration for deriving information structure types, wherein said information structure types enable classification of said information structure into dense text and non-dense text;

deriving phrase chunks through shallow text parsing, wherein the step of shallow text parsing further comprises parts of speech tagging of said dense text and the application of said information structure types;

conducting cyclical processing for deriving a plurality of textual aspects from said derived phrase chunks, comprising;

deriving structured textual aspects, wherein said derived structured textual aspects communicate meaning unambiguously and are context independent;

deriving unstructured textual aspects, wherein said derived unstructured structural aspects are one or both of context dependent and domain dependent; and

deriving binding textual aspects, wherein said binding textual aspects bind together either the textual aspects with each other or the textual aspects with textual relations;

constructing relationship maps connecting said structured, unstructured and binding textual aspects;

determining a plurality of tags for defining relationship between said structured, unstructured, and binding textual aspects, wherein said determined tags comprise one or more space tags, one or more time tags, and one or more action tags;

disambiguating relationship between said structured, unstructured, and binding textual aspects using one or more qualifiers, one or more relation elements, and one or more of said determined tags;

ranking textual aspects using said relationship maps; and

creating said text analytics and decision support matrix using the relevancy ranked textual aspects.

2. The method of claim 1, wherein the information structure types are

further divided into dense text types, non-dense text types, uniform resource locator URL patterns, title patterns, text structure patterns and presentation patterns.

3. The method of claim 1, wherein the step of cyclical processing further

comprises tagging parts of speech, extracting textual dimensions, determining co-frequency of textual segments, determining co-dependency of textual segments, and determining co-relatedness of textual segments.

4. The method of claim 1, wherein the step of deriving the structured, unstructured and binding textual aspects from the phrase chunks, further comprises the steps of:

generating individual, combinatory, complex, and predictive dimension extraction rules;

sequencing and ordering of said dimension extraction rules;

applying the dimension extraction rules for extracting dimensions;

measuring the precision value of the dimensions extracted;

editing the rules and reapplying them to generate a new set of dimensions if the precision or recall value is below a required prescribed value; and

processing the textual aspects by a set of processing elements cyclically until a minimum output resolution is achieved.

5. The method of claim 1, wherein the step of ranking textual aspects comprises:

determining the agentive dimension, and attributive dimension from the structural textual aspects;

determining the themes, dynamic dimension, actions, qualifiers and binding textual aspects from unstructured textual aspects; and

building relationship maps for said agentive dimension, dynamic dimension, attributive dimension, themes, actions and qualifiers.

6. The method of claim 1, wherein the step of extracting dimensions in the step of cyclical processing further comprises:

bootstrapping the dimension extraction using predictive rules built using a plurality of seed rules and seed information obtained as an output of dimension extraction comprising the steps of;

defining a dimension to be extracted and choosing a representative sample of said dimension as a seed dimension for processing a focused crawl;

applying a plurality of seed rules on previously extracted dimensions;

mapping relationship of said dimension with other dimensions using one individual dimension type as a seed at a time, wherein said mapped relationship is stored;

mining and mapping the seed dimension with respect to other dimensions for obtaining the individual dimension rule patterns, combinatory dimension rule patterns, part of speech pattern, co-frequency, co-dependency and co-relatedness;

creating rules for disambiguating one dimension from the rest and repeating the process of disambiguating one dimension with respect to another dimension and storing in a unique set of patterns until all possible alternatives of writing the dimension are explored;

processing a plurality of disambiguation rules cyclically for disambiguating the dimension from the rest and repeating the process for all dimensions; and

building a predictive model based on syntactic and lexical pattern analysis for identifying the presence of certain types of dimensions based on the presence or absence of dimensions and pattern recognition.

7. The method of claim 1, wherein the step of determine the relationship maps between textual themes, textual actions and textual dimensions comprises determining the co-frequency, co-occurrence, co-dependency, co-relatedness measures of said textual themes, textual actions and textual dimensions either within a document or across documents in a textual database of a document corpus.

8. The method of claim 1, wherein the step of creating seed dimensions comprises:

templatizing and automatically capturing human annotation;

generating humanly created rules; and

combining said humanly created rules and semi-automated computer generated predictive rules for extracting and processing textual dimensions from a document or a document corpus.

9. The method of claim 1, wherein textual dimensions are time, number, location, event, people roles, organizations, brand names and products, scientific and technology names and terminologies, quotations and attribution, explanations, arguments, opinions, suggestions, findings, assumptions, relations, associations, negations, emotions, sentiments, oppositions, contradictions, contrasts, qualities, attributes, commands, directives, order, rulings, measurements including scientific, technical and commercial and general measures, names of natural and man-made living beings like plants, animals, genes, clones, names of natural and manmade things, substances like chemicals, drugs, monuments, names of arts, culture, sports, and other terms and terminologies, causes, membership, substance, fact, non-facts, type and kind, entailment, part whole, contrast, metaphor, metonymy, similarity and compare.

10. The method of claim 1, wherein the step of conducting cyclical processing comprises determination of the measure of co-relatedness, comprising the steps of:

detecting word form for allowing alternative forms of a word to be included to determine relatedness of themes;

detecting qualification positioning, wherein one or two words qualify one or more words for determining the closeness of textual themes; and

detecting the variation in sense form, wherein two different dictionary senses of words determine the closeness of textual themes.

11. The method of claim 7, wherein the step of measuring co-dependency comprises:

measuring intra-dimensional dependency, wherein a particular dimension may have a dominant form or a dependent form, further comprising one of more of the following steps of;

splitting a dimension into a shorter form;

mutating a dimension by fusing it with another dimension;

extending a dimension based on its procedural occurrence;

measuring inter-dimensional dependency, wherein dependency between two or more textual dimensions is determined and extracted; and

measuring contextual dependency, wherein dependency between two dimensions within a single dimension is determined by their contextual, grammatical and positional relationships.

12. The method of claim 7, wherein the measure of co-frequency comprises the steps of:

measuring the co-frequency of textual segments that are frequent within a document and frequent across documents;

measuring the co-frequency of textual segments that are frequent within a document but not frequent across documents;

measuring the co-frequency of textual segments that are not frequent within a document but are frequent across documents; and

measuring the co-frequency of textual segments that are not frequent within a document and are not frequent across documents.

13. The method of claim 1, wherein the step of determining the most relevant textual aspects by ranking, comprises:

determining the count of relations that each textual aspect forms with every other textual aspect;

determining the summation of the strength of each individual relation, wherein the strength is based on a measure of co-occurrence, co-relatedness and co-dependency at a clause, sentence, tri-sentence, paragraph and document and document corpus level;

counting the number of qualifiers associated with each textual aspect; and

measuring the frequency of textual aspects within and across boundaries of sentence, tri-sentence, paragraph, document and document corpus level.

14. The method of claim 1, wherein the step of ranking the textual aspects further includes the steps of ranking themes, dimensions, qualifiers, actions, sentences, clauses, tri-sentences, paragraphs, document and document corpus level.

18. The method of claim 1, wherein said space tags define relationship between the textual aspects based on location of the textual aspects and said time tags define relationship between textual aspects based on a given time range, wherein said time tags further validate and find existence of the relationship between the textual aspects for said given time range.

19. The method of claim 1, wherein said action tags define relationship based on action associated with the textual aspects.

15. A system, comprising of a processor and a storage device, configured for creating a text analysis and decision support matrix by analyzing and deriving analytical insights, and further comprising:

a visual information extraction and structuration engine for conducting information structuration for deriving information structure types;

cyclical extractor for conducting cyclical processing and deriving phrase chunks from said information structures and deriving structured, unstructured and binding textual aspects from the phrase chunks;

relationship builder for identifying the relation between said structured, unstructured and binding textual aspects and constructing relationship maps;

ranking module for ranking textual aspects using said relationship maps;

text analytics and decision support matrix creator for creating text analytics and decision support matrix using the relevancy ranked textual aspects;

a model enhancer for generating automated web queries and conducting wide web crawl for enhancing the product;

a textual information database for storing data; and

a user interface for presenting the text analysis and decision support matrix in usable form.

16. The system of claim 15, wherein said text analytics and decision support matrix comprises matrix variables customizable to cater to specific needs of a user.

17. The system of claim 15, wherein said cyclical extractor further comprises:

a dimension extraction engine for extracting dimensions from the textual aspects comprising;

a rule generation module for generating dimension extraction rules;

a validation clean up and conflict resolution module for dimension extraction rule sequence and ordering, dimension results validation sequencing and ordering, dimension results clean up sequencing and ordering and conflict resolution through cyclical processing;

a precision recall and measurement module for modifying a plurality of rules for dimension extraction and creating a new set of rules; and

a result editing module for determining a sample set of results for various dimensions generated based on dimension specific sampling rules.