Information data retrieval, where the data is organized in terms, documents and document corpora

• US 7,593,932 B2
• Filed: 01/13/2003
• Issued: 09/22/2009
• Est. Priority Date: 01/16/2002
• Status: Active Grant

• Current Assignee: Elucidon Group Ltd.
• Original Assignee: Elucidon Group Ltd.
• Inventors: Lindh, Per, Londahl, Bjorn
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): XUE, BIN
• Application Number:
  US10/501,397
• Publication Number:
  US 20050149494A1
• Time in Patent Office: 2,444 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/289 Object oriented databases
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

The invention relates to improved solutions for information retrieval, wherein the information is represented by digitized text data. This data is further presumed to be organized in terms (431-438), documents and document corpora, where each document contains at least one term (431-438) and each document corpus contains at least one document. Based on a concept vector (420-424), which conceptually classifies the contents of each document, a term-to-concept vector is generated for each term (431-438) in the document corpus. The term-to-concept vector describes a relationship between the term (431) and each of the concept vectors (420-424). On basis of the term-to-concept vectors for the document corpus, a term-term matrix is generated which describes a term-to-term relationship between all the terms (431-438) in the document corpus. The term-term matrix may then be processed and used for retrieving information from the document corpus, such as the fact that a first term (431) is related to a second term (436).

Claims

1. A method of processing digitized textual information in a computerized database system, the information being organized in terms, documents and document corpora, where each document contains at least one term and each document corpus contains at least one document, the method comprising:

generating, by using a computer, a concept vector for each document in a document corpus wherein the concept vector conceptually classifying contents of the document on a relatively compact format,

generating, for each term in the document corpus, a term-to-concept vector describing a relationship between the term and each of the concept vectors wherein the term-to-concept vectors being generated on basis of the concept vectors, comprises;

receiving the term-to-concept vectors for the document corpus and on basis thereof generating a term-term matrix describing a term-to-term relationship between the terms in the document corpus, wherein the generation of the term-term matrix comprises;

retrieving, for each term in each combination of two unique terms in the document corpus, a respective term-to-concept vector, generating a relation vector describing the relationship between the terms in the each combination of two unique terms, each component in the relation vector being equal to a lowest component value of corresponding component values in the term-to-concept vectors, generating a relationship value for the each combination of two unique terms as the sum of all component values in the corresponding relation vector, and generating a matrix containing the relationship values of all combinations of two unique terms in the document corpus,

processing the term-term matrix into processed textual information and displaying the processed textual information via a user output interface, and

displaying the processed textual information as a distance graph in which each term constitutes a node wherein the node representing a first term is connected to one or more other nodes representing secondary terms to which the first term has a conceptual relationship of at least a specific strength, and a relevance measure between the first term and at least one second term is represented by a minimum number of node hops between the first term and the at least one second term.

2. The method according to claim 1, wherein each document in the document corpus being associated with a document-concept matrix representing at least one concept element whose relevance with respect to the document is described by a weight factor, the generation of each term-to-concept vector comprises:

identifying a term-relevant set of documents in the document corpus, each document in the term-relevant set containing at least one occurrence of the term,

calculating a term weight for the term in each of the documents in the term-relevant set,

retrieving a respective concept vector being associated with each document in the term-relevant set where the term weight exceeds a first threshold value,

selecting a relevant set of concept vectors including any concept vector in which at least one concept component exceeds a second threshold value,

calculating a non-normalized term-to-concept vector as the sum of all concept vectors in the relevant set, and

normalizing the non-normalized term-to-concept vector.

3. The method according to claim 1 wherein the method further comprises the steps of:

calculating a statistical co-occurrence value between the each combination of two unique terms in the document corpus, the statistical co-occurrence value describing a dependent probability that a certain second term exists in a document provided that a certain first term exists in the document, and

incorporating the statistical co-occurrence values into the term-term matrix to represent lexical relationships between the terms in the document corpus.

4. The method according to claim 1 wherein the method further comprises the step of:

displaying the processed textual information on a format being adapted for human comprehension.

5. The method according to claim 4, wherein the displaying step further comprises involving presentation of:

at least one document identifier specifying a document being relevant with respect at least one term in a query,

at least one term being related to a term in the query, and

a conceptual distribution representing a conceptual relationship between two or more terms in the document corpus, the conceptual distribution being based on shared concepts which are common to said terms.

6. The method according to claim 4 wherein the displaying step further comprises involving presentation of at least one document identifier specifying a document being relevant with respect to at least one term in a query in combination with at least one user specified concept.

7. The method according to claim 5 wherein the method further comprises the step of:

selecting the at least one user specified concept from the shared concepts in the conceptual distribution.

8. The method according to claim 4 wherein the method further comprises the step of:

illustrating the conceptual relationship between the first term and the at least one second term by means of a respective relevance measure being associated with the at least one second term in respect of the first term.

9. The method according to claim 8, wherein the method further comprises the step of:

displaying the processed textual information on a graphical format which visualizes the strength in the conceptual relationship between at least two terms.

10. The method according to claim 8 wherein the method further comprises the step of:

displaying the processed textual information as a distance graph in which each term constitutes a node wherein the node representing the first term is connected to one or more other nodes representing secondary terms to which the first term has a conceptual relationship, each connection is associated with an edge weight representing the strength of a conceptual relationship between the first term and a particular secondary term, and the relevance measure between the first term and the particular secondary term is represented by an accumulation of the edge weights being associated with the connections constituting a minimum number node hops between the first term and the particular secondary term.

11. The method according to claim 1, wherein each term further comprises:

a single word,

a proper name,

a phrase, and

a compound of single words.

12. The method according to claim 1 further comprises the step of updating the document corpus with added data in form of at least one new document by means of

identifying any added terms in the new document which lack a representation in the document corpus,

identifying any existing terms in the new document which were represented in the document corpus before adding the at least one new document,

retrieving, for each of the existing terms, a corresponding concept vector,

generating a new concept vector with respect to the at least one new document as a sum of the corresponding concept vectors,

normalizing the new concept vector into a normalized new concept vector, and

assigning the normalized new concept vector to each of the added terms in the new document.

13. A computer-implemented search engine, embedded on a computer readable storage medium, for processing an amount of digitized textual information and extracting data there from, the information being organized in terms, documents and document corpora, where each document contains at least one term and each document corpus contains at least one document, comprising:

an interface configured to receive a query from a user, and

a processing unit configured to process a document corpus on basis of the query and return processed textual information being relevant to the query said process involving

generating a concept vector for each document in the document corpus, the concept vector conceptually classifying contents of the document on a relatively compact format, and

generating, for each term in the document corpus, a term-to-concept vector describing a relationship between the term and each of the concept vectors, wherein the processing unit in turn comprises;

a processing module configured to receive the term-to-concept vectors for the document corpus and on basis thereof generate a term-term matrix describing a term-to-term relation-ship between the terms in the document corpus, wherein the generation of the term-term matrix comprises;

retrieving, for each term in each combination of two unique terms in the document corpus, a respective term-to-concept vector, generating a relation vector describing the relationship between the terms in the each combination of two unique terms, each component in the relation vector being equal to a lowest component value of corresponding component values in the term-to-concept vectors, generating a relationship value for the each combination of two unique terms as the sum of all component values in the corresponding relation vector, and generating a matrix containing the relationship values of all combinations of two unique terms in the document corpus,

an exploring module configured to receive the query and the term-term matrix, and on basis of the query process the term-term matrix into the processed textual information, and

a display module configured to display the processed textual information as a distance graph in which each term constitutes a node wherein the node representing a first term is connected to one or more other nodes representing secondary terms to which the first term has a conceptual relationship of at least a specific strength, and a relevance measure between the first term and at least one second term is represented by a minimum number of node hops between the first term and the at least one second term.

14. A computer-implemented database system comprising:

a processor;

memory holding an amount of digitized textual information being organized in terms, documents and document corpora, wherein each document contains at least one term and each document corpus contains at least one document, wherein each document in a document corpus being associated with concept vector which conceptually classifies contents of the document on a relatively compact format, and wherein each term in the document corpus being associated with a term-to-concept vector describing a relationship between the term and each of the concept vectors, delivering the term-to-concept vectors to a search engine for processing an amount of digitized textual information and extracting data there from, the information being organized in terms, documents and document corpora, where each document contains at least one term and each document corpus contains at least one document, and

computer program instructions implementing;

an interface configured to receive a query from a user, and

a processing unit configured to process a document corpus on basis of the query and return processed textual information being relevant to the query said process involving

generating a concept vector for each document in the document corpus, the concept vector conceptually classifying the contents of the document on a relatively compact format, and

generating, for each term in the document corpus, a term-to-concept vector describing a relationship between the term and each of the concept vectors, wherein the processing unit in turn comprises;

a processing module configured to receive the term-to-concept vectors for the document corpus and on basis thereof generate a term-term matrix describing a term-to-term relation-ship between the terms in the document corpus, wherein the generation of the term-term matrix comprises;

retrieving, for each term in each combination of two unique terms in the document corpus, a respective term-to-concept vector, generating a relation vector describing the reIationship between the terms in the each combination of two unique terms, each component in the relation vector being equal to a lowest component value of corresponding component values in the term-to-concept vectors, generating a relationship value for the each combination of two unique terms as the sum of all component values in the corresponding relation vector, and generating a matrix containing the relationship values of all combinations of two unique terms in the document corpus,

an exploring module configured to receive the query and the term-term matrix, and on basis of the query process the term-term matrix into the processed textual information, and

a display module configured to display the processed textual information as a distance graph in which each term constitutes a node wherein the node representing a first term is connected to one or more other nodes representing secondary terms to which the first term has a conceptual relationship of at least a specific strength, and a relevance measure between the first term and at least one second term is represented by a minimum number of node hops between the first term and the at least one second term.

15. The computer-implemented database system according to claim 14 further comprising an iterative term-to-concept engine configured to receive fresh digitized textual information added to the database and on basis of this information

generate concept vectors for any added document, and

generate a term-to-concept vector describing a relationship between any added term and each of the concept vectors.

16. A server computer system for providing data processing services in respect of digitized textual information, wherein the server comprises:

a processor;

memory for storing computer program instructions and data; and

computer pogram instructions stored in the memory for implementing;

a search engine for processing an amount of digitized textual information and extracting data there from, the information being organized in terms, documents and document corpora, where each document contains at least one term and each document corpus contains at least one document, comprising an interface configured to receive a query from a user, and a processing unit configured to process a document corpus on basis of the query and return processed textual information being relevant to the query said process involving generating a concept vector for each document in the document corpus, the concept vector conceptually classifying contents of the document on a relatively compact format, and generating, for each term in the document corpus, a term-to-concept vector describing a relationship between the term and each of the concept vectors, wherein the processing unit in turn comprises a processing module configured to receive the term-to-concept vectors for the document corpus and on basis thereof generate a term-term matrix describing a term-to-term relation-ship between the terms in the document corpus, wherein the generation of the term-term matrix comprises;

retrieving, for each term in each combination of two unique terms in the document corpus, a respective term-to-concept vector, generating a relation vector describing the relationship between the terms in the each combination of two unique terms, each component in the relation vector being equal to a lowest component value of corresponding component values in the term-to-concept vectors, generating a relationship value for the each combination of two unique terms as the sum of all component values in the corresponding relation vector, and generating a matrix containing the relationship values of all combinations of two unique terms in the document corpus, an exploring module configured to receive the query and the term-term matrix, and on basis of the query process the term-term matrix into the processed textual information, a display module configured to display the processed textual information as a distance graph in which each term constitutes a node wherein the node representing a first term is connected to one or more other nodes representing secondary terms to which the first term has a conceptual relationship of at least a specific strength, and a relevance measure between the first term and at least one second term is represented by a minimum number of node hops between the first term and the at least one second term, and

a communication interface towards a database system holding an amount of digitized textual information and configured to deliver the term-to concept vectors to the search engine.

17. A computer system comprising:

a processor for executing computer program instructions, a memory for storing computer program instructions and computer program instructions comprising software for processing digitized textual information, the information being organized in terms, documents and document corpora, where each document contains at least one term and each document corpus contains at least one document, the digitized textual information processed by;

generating a concept vector for each document in a document corpus wherein the concept vector conceptually classifying the contents of the document on a relatively compact format,

generating, for each term in the document corpus, a term-to-concept vector describing a relationship between the term and each of the concept vectors wherein the term-to-concept vectors being generated on basis of the concept vectors,

receiving the term-to-concept vectors for the document corpus and on basis thereof generating a term-term matrix describing a term-to-term relationship between the terms in the document corpus, wherein the generation of the term-term matrix comprises;

retrieving, for each term in each combination of two unique terms in the document corpus, a respective term-to-concept vector, generating a relation vector describing the relationship between the terms in the each combination of two unique terms, each component in the relation vector being equal to a lowest component value of corresponding component values in the term-to-concept vectors, generating a relationship value for the each combination of two unique terms as the sum of all component values in the corresponding relation vector, and generating a matrix containing the relationship values of all combinations of two unique terms in the document corpus,

processing the term-term matrix into processed textual information and displaying the processed textual information via a user output interface, and

displaying the processed textual information as a distance graph in which each term constitutes a node wherein the node representing a first term is connected to one or more other nodes representing secondary terms to which the first term has a conceptual relationship of at least a specific strength, and a relevance measure between the first term and at least one second term is represented by a minimum number of node hops between the first term and the at least one second term.

18. A computer program product stored in a computer readable storage medium, the computer program product comprising:

computer program instructions recorded thereon for causing a computer to process digitized textual information, the information being organized in terms, documents and document corpora, where each document contains at least one term and each document corpus contains at least one document, the digitized textual information processed by;

generating a concept vector for each document in a document corpus wherein the concept vector conceptually classifying contents of the document on a relatively compact format,

generating, for each term in the document corpus, a term-to-concept vector describing a relationship between the term and each of the concept vectors wherein the term-to-concept vectors being generated on basis of the concept vectors,

receiving the term-to-concept vectors for the document corpus and on basis thereof generating a term-term matrix describing a term-to-term relationship between the terms in the document corpus, wherein the generation of the term-term matrix comprises;

retrieving, for each term in each combination of two unique terms in the document corpus, a respective term-to-concept vector, generating a relation vector describing the relationship between the terms in the each combination of two unique terms, each component in the relation vector being equal to a lowest component value of corresponding component values in the term-to-concept vectors, generating a relationship value for the each combination of two unique terms as the sum of all component values in the corresponding relation vector, and generating a matrix containing the relationship values of all combinations of two unique terms in the document corpus,

processing the term-term matrix into processed textual information and displaying the processed textual information via a user output interface, and

displaying the processed textual information as a distance graph in which each term constitutes a node wherein the node representing a first term is connected to one or more other nodes representing secondary terms to which the first term has a conceptual relationship of at least a specific strength, and a relevance measure between the first term and at least one second term is represented by a minimum number of node hops between the first term and the at least one second term.