Decision-support expert system and methods for real-time exploitation of documents in non-english languages

• US 8,078,551 B2
• Filed: 08/31/2006
• Issued: 12/13/2011
• Est. Priority Date: 08/31/2005
• Status: Active Grant

• Current Assignee: IntuView Ltd.
• Original Assignee: IntuView Ltd.
• Inventors: Bar, Shmuel
• Primary Examiner(s): VINCENT, DAVID ROBERT
• Application Number:
  US11/991,002
• Publication Number:
  US 20090157382A1
• Time in Patent Office: 1,930 Days
• Field of Search: 704/231, 704/270, 706/12, 706/45, 706/62
• US Class Current: 706/12
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232

Abstract

A method for real-time exploitation of documents in non-English languages includes processing an input document in into a processed input document, extracting ontology elements from the processed input document to obtain a document digest (DD), statistically scoring each DD to obtain a DD with category scores, refining the DD and the category scores to obtain a summary of each document in the form of a refined DD with refined category scores. The summary allows a user to estimate in real-time if the input document warrants added attention.

Claims

1. A method for automatic real-time summarization and information extraction from one or more documents in a source language in order to present their content in a summary in a chosen target language and to input the content as categorized entities, the method comprising:

a. selecting said source language;

b. processing an input document into a textual format document while preserving visual information;

c. performing linguistic analysis based on models of a linguistic-register associated with the analyzed document;

d. extracting lexical instances corresponding to at least one of a pre-determined domain-specific lexicon and run-time lexical instances built according to pre-determined syntactic rules;

e. extracting ontology elements from the lexical instances to obtain a set of ontological instances comprising instances from a pre-determined ontology and refined ontological instances, said refined ontological instances comprising a combination of pre-determined ontological instances;

f. creating a document digest (DD) from the set of ontological instances, the DD being presented in a relationary map which retains information on features, context and linguistic origin of the components of the DD;

g. using domain-specific statistical models relating to categorization clusters and categories within said categorization clusters to determine the most likely categorization of the document in relation to a spectrum of categories by comparing the updated DD to each category model;

wherein the step of using domain-specific models includes;

i. creating a vector representing each DD using a pre-categorized corpus of domain-specific documents;

ii. analyzing those documents to extract a model of a document digest which corresponds to or contradicts each category of document in the domain;

iii. using statistical algorithms to create the models and creating mixed models of the different results which are specific to each category;

iv. creating a mixed model of the results of analysis of each text;

v. calculating a similarity result between each of a plurality of statistical models obtained offline of said categories and the vector generated by the DD of the input text and determining the most likely category in each cluster;

vi. adding ontological instances to the DD based on the input of the categorization upon which rules from the rule base are applied; and

vii. analyzing the winner model to extract a confidence score related to the DD;

h. creating in said target language a document summary for each of said one or more documents based on said DD and summarization rules.

2. The method of claim 1, wherein the step of identifying the source language is performed using linguistic features selected from the group consisting of an alphabet, unique or typical diacritics, frequency analysis of letter combinations and a combination thereof.

3. The method of claim 1, wherein the step of extraction of lexical instances includes:

i. identifying sets of linguistic styles which constitute distinct linguistic registers;

ii. linguistic tagging corpuses of texts corresponding to those registers;

iii. building of statistical models for use in future linguistic analysis of text outside of the corpus;

iv. using said statistical models to perform morphological and syntactic analysis on the input text to determine whether morphological variants of lexical instances in the lexicon which appear in the text are to be accepted as corresponding to those lexical entries.

4. The method of claim 1 wherein the step of extracting ontology elements includes linking lexical instances to ontological instances contained in a domain-specific ontology by performing one or more of the following:

i. corresponding one lexical instance to more than one ontological instance;

ii. a number of lexical instances generating together at least one ontological instance;

iii. creating disambiguation of ontological instances by the same lexical instances is done by rules.

5. The method of claim 1 wherein the step of creating a document digest (DD) includes setting all ontological instances in relation to each other in a manner which corresponds to:

i. their relative location in the source text; and

ii. implicit semantic features which are expressed by their hierarchal place in the ontology and features represented by functional relations with other ontological instances.

6. The method of claim 1 further comprising, after step g:

i. creating virtual ontological entities from the ontological instances in said set of ontological instances comprising determining their association to one or more ontological concepts, their features representing the contextual and implicit information which describes their nature and relationship with other entities and their linguistic representations and inputting them into a database;

ii. matching virtual ontological entities to pre-existing ontological entities and saving separately matching and non-matching virtual ontological entities;

iii. updating the DD with said ontological entities and virtual ontological entities to create an updated DD; and

iv. repeating steps (i) to (iii) to generate new ontological instances based on the categorization of the document in step (i) and on application of categorization rules.

7. The method of claim 6 wherein the creation of said virtual entities is performed by:

i. identifying named entities in the DD;

ii. comparing the identified named entities to named entities in the ontology;

iii. identifying possible relations between ontology instances and the named entities;

iv. identifying named entities which are not in the ontology and determining their category or type based on contextual information and information implicit in the names;

v. “

reverse engineering”

of transliterated names into the original source language determining the original form or possible forms of the entity, using the aggregated attributes of the entity to choose between alternative outputs or to provide confidence to them;

vi. parsing the named entity and performs cultural-linguistic sensitive analysis of its components;

vii. using the analysis of these components for further validation or reassessment of the categorization of the type of the entity;

viii. filling the attribute slots of the virtual entity to provide further qualities for identification and matching;

ix. aggregation of all the variants and aliases referring to an entity within the different inputs, while maintaining their source identity for possible regression;

x. finding relations between identified entities; and

xi. generation of a virtual “

identity card”

of an entity with all the aggregated information that is collected in the inputs about it.

8. The method of claim 1, wherein the creating said vector representing each DD includes:

i. forming a vector in ontology elements space, wherein an i-th cell of the vector represents an ontology element through the value of a respective term frequency multiplied by an inverse document frequency;

ii. normalizing the vector into a normalized vector, and

iii. multiplying the normalized vector by a manually given score of the respective DD.

9. The method of claim 1 wherein the statistically scoring each DD to obtain a DD with associated category scores includes:

i. creating a vector representing each DD;

ii. calculating a similarity result between each of a plurality of centroids obtained offline and the vector and determining a winner centroid, and

iii. analyzing the winner centroid to extract a confidence score related to the DD.

10. The method of claim 9, wherein the creating a vector representing each DD includes forming a vector in ontology elements space, wherein an i-th cell of the vector represents an ontology element through the value of a respective term frequency multiplied by an inverse document frequency, normalizing the vector into a normalized vector, and multiplying the normalized vector by a manually given score of the respective DD.

11. The method of claim 9, wherein the statistically scoring each DD based on predefined categories included in a knowledge base includes adjusting values in the knowledge base based on input of new materials.

12. The method of claim 11, wherein the adjusting is manual.

13. The method of claim 11, wherein the adjusting is automatic.

14. The method of claim 13, wherein the automatic adjusting includes automatic training.