System, Method and Computer Program Product for Performing Unstructured Information Management and Automatic Text Analysis, Including a Search Operator Functioning as a Weighted And (WAND)

• US 20090222441A1
• Filed: 06/13/2008
• Published: 09/03/2009
• Est. Priority Date: 05/30/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: ZIEN, Jason, SOFFER, Aya, BRODER, Andrei Z., HERSCOVICI, Michael, CARMEL, David
• Granted Patent: US 8,280,903 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

Disclosed is a system architecture, components and a searching technique for an Unstructured Information Management System (UIMS). The UIMS may be provided as middleware for the effective management and interchange of unstructured information over a wide array of information sources. The architecture generally includes a search engine, data storage, analysis engines containing pipelined document annotators and various adapters. The searching technique makes use of a two-level searching technique. A search query includes a search operator containing of a plurality of search sub-expressions each having an associated weight value. The search engine returns a document or documents having a weight value sum that exceeds a threshold weight value sum. The search operator is implemented as a Boolean predicate that functions as a Weighted AND (WAND).

Claims

1. A data processing system for processing stored data, comprising:

data storage for storing a collection of data units; and

coupled to the data storage, a search engine responsive to a query for retrieving at least one data unit from said data storage;

where

the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum.

2. A data processing system as in claim 1, where said data units comprise documents.

3. A data processing system as in claim 1, where at least one of the weight values and threshold weight value are variable during a search.

4. (canceled)

5. (canceled)

6. (canceled)

7. (canceled)

8. (canceled)

9. (canceled)

10. (canceled)

11. (canceled)

12. (canceled)

13. (canceled)

14. (canceled)

15. (canceled)

16. (canceled)

17. (canceled)

18. (canceled)

19. A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising:

a computer program segment for storing a collection of data units; and

a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;

where

the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum.

20. A computer program product as in claim 19, where said data units comprise documents.

21. A computer program product as in claim 19, where at least one of the weight values and threshold weight value are variable during a search.

33. A computer program product as in claim 19, where the data unit having a weight value sum that exceeds a threshold weight value sum undergoes a detailed evaluation.

34. A computer program product as in claim 33, where a value associated with the detailed evaluation is compared to minimum value.

35. A computer program product as in claim 34, where if the value associated with the detailed evaluation is above the minimum value, at least one of the associated weight values and threshold are modified.

36. A computer program product as in claim 34, where if the value associated with the detailed evaluation is above the minimum value, the data unit is added to a heap.

37. A computer program product as in claim 36, wherein the heap represents top scored data units.

22. (canceled)

23. (canceled)

24. (canceled)

25. (canceled)

26. (canceled)

27. (canceled)

28. (canceled)

29. (canceled)

30. (canceled)

31. (canceled)

32. (canceled)

38. A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising:

a computer program segment for storing a collection of data units; and

a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;

where the at least one data unit is stored in a heap if the heap is not full and where the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value.

39. A computer program product as in claim 38, where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum.

40. A computer program product as in claim 39, where, if the heap is full, the data unit having a weight value sum that exceeds the threshold weight value sum replaces a data unit with the least weight value sum from the heap.