System, method and computer program product for performing unstructured information management and automatic text analysis, including a search operator functioning as a Weighted AND (WAND)

• US 8,280,903 B2
• Filed: 06/13/2008
• Issued: 10/02/2012
• Est. Priority Date: 05/30/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Broder, Andrei Z, Carmel, David, Herscovici, Michael, Soffer, Aya, Zien, Jason
• Primary Examiner(s): ROBINSON, GRETA LEE
• Application Number:
  US12/138,857
• Publication Number:
  US 20090222441A1
• Time in Patent Office: 1,572 Days
• Field of Search: 707/706, 707/723, 707726-730, 707/735, 707/737, 707/739, 707748-750, 707/754, 707/758, 707/769, 707/771, 707/779, 707/802, 707/999.004, 707/999.005, 707/999.101
• US Class Current: 707/769
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Disclosed is a system architecture, components and a searching technique for an Unstructured Information Management System (UIMS). The UIMS may be provided as middleware for the effective management and interchange of unstructured information over a wide array of information sources. The architecture generally includes a search engine, data storage, analysis engines containing pipelined document annotators and various adapters. The searching technique makes use of a two-level searching technique. A search query includes a search operator containing of a plurality of search sub-expressions each having an associated weight value. The search engine returns a document or documents having a weight value sum that exceeds a threshold weight value sum. The search operator is implemented as a Boolean predicate that functions as a Weighted AND (WAND).

Claims

1. A data processing system for processing stored data, comprising:

data storage for storing a collection of data units; and

coupled to the data storage, a search engine responsive to a query for retrieving at least one data unit from said data storage;

where

the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum, where at least one of the weight values and threshold weight value sum are variable during a search.

2. A data processing system as in claim 1, where said data units comprise documents.

3. A data processing system as in claim 1, wherein the search engine is configured to assimilate views of two levels.

4. A data processing system as in claim 3, wherein the two levels treats multiple views of a document as completely separate documents unrelated to each other unless they focus on a same document text.

5. A data processing system as in claim 3, wherein in the two levels treats the search engine is configured to enable queries to span multiple views of a document.

6. A data processing system as in claim 1, wherein the data processing system is configured to perform at least one analysis function and at least one access function.

7. A data processing system as in claim 6, wherein the at least one analysis function is configured to perform document-level analysis and collection-level analysis.

8. A data processing system as in claim 6, wherein the at least one access function is configured to perform semantic search access and structured knowledge access.

9. A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising:

a computer program segment for storing a collection of data units; and

a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;

where

the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum, where at least one of the weight values and threshold weight value sum are variable during a search.

10. A computer program product as in claim 9, where said data units comprise documents.

11. A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising :

a computer program segment for storing a collection of data units; and

a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;

where

the query comprises a search operator comprised of a plurality of search sub-expression each having an associated weight value, and where said search engine returns a data having a weight value sum that exceeds a threshold weight value sum, where the data unit having a weight value sum that exceeds a threshold weight value sum undergoes a detailed evaluation and where a value associated with the detailed evaluation is compared to minimum value.

12. A computer program product as in claim 11, where if the value associated with the detailed evaluation is above the minimum value, at least one of the associated weight values and threshold are modified.

13. A computer program product as in claim 11, where if the value associated with the detailed evaluation is above the minimum value, the data unit is added to a heap.

14. A computer program product as in claim 13, wherein the heap represents top scored data units.

15. A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising:

a computer program segment for storing a collection of data units; and

a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;

where the at least one data unit is stored in a heap if the heap is not full and where the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum, where, if the heap is full, the data unit having a weight value sum that exceeds the threshold weight value sum replaces a data unit with the least weight value sum from the heap.