Probabilistic information retrieval based on differential latent semantic space

• US 20030050921A1
• Filed: 05/08/2001
• Published: 03/13/2003
• Est. Priority Date: 05/08/2001
• Status: Active Grant

• Current Assignee: SunFlare Co., Ltd.
• Original Assignee: SunFlare Co., Ltd.
• Inventors: Sasai, Hiroyuki, Chen, Liang, Tokuda, Naoyuki
• Granted Patent: US 6,654,740 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: Y10S 707/917 Text
• CPC Class Codes: Y10S 707/955 Object-oriented
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A computer-based information search and retrieval system and method for retrieving textual digital objects that makes full use of the projections of the documents onto both the reduced document space characterized by the singular value decomposition-based latent semantic structure and its orthogonal space. The resulting system and method has increased robustness, improving the instability of the traditional keyword search engine due to synonymy and/or polysemy of a natural language, and therefore is particularly suitable for web document searching over a distributed computer network such as the Internet.

Claims

1. A method for setting up an information retrieval system and retrieving text information, comprising the steps of:

preprocessing text including word, noun phrase and stop word identification;

constructing system terms including setting up a term list and global weights;

setting up and normalizing document vectors of all collected documents;

constructing an interior differential term-document matrix D_I^mxn₁ such that each column in said interior differential term-document matrix is an interior differential document vector;

decomposing, using SVD algorithm, D_I, such that D_I=USV^T, then with a proper k₁, defining the D_I,k1=U_k1S_k1V_k1^T to approximate D_I;

defining an interior document likelihood function, P(x|D_I);

constructing an exterior differential term-document matrix D_E^mxn₁, such that each column in said exterior differential term-document matrix is an exterior differential document vector;

decomposing, using SVD algorithm, D_E, such that D_E=USV^T, then with a proper value of k₂, defining the D_E,k2=U_k2S_k2V_k2^T to approximate D_E;

defining an exterior document likelihood function, P(x|D_E); and

defining a posteriori function

$P<br><br>\left(D_I,x\right)=\frac{P<br><br>\left(x<br><br>D_I\right)<br><br>P<br><br>\left(D_I\right)}{P<br><br>\left(x<br><br>D_I\right)<br><br>P<br><br>\left(D_I\right)+P<br><br>\left(x<br><br>D_E\right)<br><br>P<br><br>\left(D_E\right)},$

where P(D_I) is set to be an average number of recalls divided by the number of documents in the data base and P(D_E) is set to be 1-P(D_I).

2. The method as set forth in claim 1, wherein the interior document likelihood function, P(x|D_I) is,

3. The method as set forth in claim 2, wherein, ρ

₁ is chosen as δ

_ki+1²/2, and r₁ is n₁.

4. The method as set forth in claim 1, wherein the exterior document likelihood function, P(x|D_E) is,

5. The method as set forth in claim 4, wherein ρ

₂ is chosen as δ

_k2+1²/2, and r₂ is n₂.

6. The method as set forth in claim 1, further comprising the steps of:

setting up a document vector for a query by generating terms as well as frequency of term occurrence, and thereby obtaining a normalized document vector for the query;

given the query, constructing a differential document vector x;

calculating the interior document likelihood function P(x|D_I) and the exterior document likelihood function P(x|D_E) for the document;

calculating the posteriori probability function P(D_I|x); and

selecting documents according to one of P(D_I|x) exceeding a given threshold or N best documents with largest P(D_I|x), those values of P(D_I|x) being shown as scores to rank a match.

7. A method for setting up an information retrieval system and retrieving text information, comprising the steps of:

preprocessing text;

constructing system terms;

setting up and normalizing document vectors of all collected documents;

constructing an interior differential term-document matrix D_I^mxn₁ such that each column in said interior differential term-document matrix is an interior differential document vector;

decomposing D_I, such that D_I=USV^T, then with a proper k₁, defining the D_I,k1=U_k1S_k1V_k1^T to approximate D_I;

defining an interior document likelihood function, P(x|D_I);

constructing an exterior differential term-document matrix D_E^mxn₂, such that each column in said exterior differential term-document matrix is an exterior differential document vector;

decomposing D_E, such that D_E=USV^T, then with a proper value of k₂, defining the D_E,k2=U_k2S_k2V_k2^T to approximate D_E;

defining an exterior document likelihood function, P(x|D_E);

defining a posteriori function

$P<br><br>\left(D_I,x\right)=\frac{P<br><br>\left(x<br><br>D_I\right)<br><br>P<br><br>\left(D_I\right)}{P<br><br>\left(x<br><br>D_I\right)<br><br>P<br><br>\left(D_I\right)+P<br><br>\left(x<br><br>D_E\right)<br><br>P<br><br>\left(D_E\right)},$

where P(D_I) is set to be an average number of recalls divided by the number of documents in the data base and P(D_E) is set to be 1-P(D_I);

setting up a document vector for a query by generating terms as well as frequency of term occurrence, and thereby obtaining a normalized document vector for the query;

given the query, constructing a differential document vector x;

calculating the interior document likelihood function P(x|D_I) and the exterior document likelihood function P(x|D_E) for the document;

calculating the posteriori probability function P(D_I|x); and

selecting documents according to one of P(D_I|x) exceeding a given threshold or N best documents with largest P(D_I|x), those values of P(D_I|x) being shown as scores to rank a match.

8. The method as set forth in claim 7, wherein the interior document likelihood function, P(x|D_I) is,

9. The method as set forth in claim 8, wherein, ρ

₁ is chosen as δ

_ki+1²/2, and r₁ is n₁.

10. The method as set forth in claim 7, wherein the exterior document likelihood function, P(x|D_E) is,

11. The method as set forth in claim 10, wherein ρ

₂ is chosen as δ

_k2+1² /2, and r₂ is n₂.