System and method for measuring the quality of document sets

• US 8,219,593 B2
• Filed: 06/25/2008
• Issued: 07/10/2012
• Est. Priority Date: 06/26/2007
• Status: Active Grant

• Current Assignee: Oracle OTC Subsidiary LLC (Oracle Corporation)
• Original Assignee: Endeca Technologies Incorporated (Oracle Corporation)
• Inventors: Tunkelang, Daniel, Wang, Joyce Jeanpin, Zelevinsky, Vladimir, Kapell, Joshua William
• Primary Examiner(s): Breene, John E
• Assistant Examiner(s): PHILLIPS, III, ALBERT M
• Application Number:
  US12/146,223
• Publication Number:
  US 20090006438A1
• Time in Patent Office: 1,476 Days
• Field of Search: 707/776, 707/803
• US Class Current: 707/803
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/3331 Query processing

Abstract

Systems and methods are described that calculate the interestingness of a set of one or more records in a database, either absolutely (i.e., compared to an overall collection of records) or relative to some other set of records. In one embodiment, the measure is a relative entropy value that has been normalized. Various applications of the measure are described in the context of an information retrieval system. These applications include, for example, guiding query interpretation, guiding view selection and summarization, intelligent ranges, event detection, concept triggers and interpreting user actions, hierarchy discovery, and adaptive data mining.

Claims

1. A method for organizing a database, the method comprising:

accessing, by a computer system, a set comprising a plurality of database records stored in a database on a computer system, wherein the plurality of database records comprise at least one facet, and wherein the database is organized based on an existing structure;

establishing, by the computer system, at least one identifying characteristic for the set comprising the plurality of database records and a respective at least one facet;

analyzing, by the computer system, the database to determine a statistical distribution of the at least one identifying characteristic;

generating, by the computer system, a measurement of distinctiveness for the set, based on the statistical distribution of the at least one identifying characteristic;

identifying, by the computer system, at least one distinctive group of database records within the database based on the measurement of distinctiveness;

generating, by the computer system, a descriptor associated with the identified at least one distinctive group of database records; and

organizing, by the computer system, the database based on the descriptor, wherein the act of organizing the database based on the descriptor includes modifying the existing structure of the database to include the organization of the database based on the descriptor.

2. The method according to claim 1, wherein the act of analyzing the database further comprises an act of generating at least one subset of the set comprising a plurality of database records.

3. The method according to claim 1, further comprising an act of normalizing the measurement of distinctiveness, wherein the act of normalizing the measurement of distinctiveness includes an act of correcting the measurement of distinctiveness of the set to account for a size of the set, based on a measurement of distinctiveness of the at least one generated subset.

4. The method according to claim 2, further comprising an act of manipulating a size of the generated subset within the database.

5. The method according to claim 1, further comprising an act of comparing the statistical distribution of the at least one identifying characteristic against a baseline statistical distribution.

6. The method according to claim 1, further comprising an act of generating a hierarchy of organization based on the measurement of distinctiveness.

7. The method according to claim 6, further comprising an act of identifying a relationship between database records of the database based, at least in part, on the statistical distribution of at least one identifying characteristic.

8. The method according to claim 7, further comprising an act of determining a relationship based on at least one identifying characteristic in common, and a measurement of distinctiveness.

9. The method according to claim 7, wherein the relationship identifies at least one of a parent, child, and sibling database record within the database.

10. The method according to claim 6, further comprising an act of updating the hierarchy of organization based on review of the database.

11. The method according to claim 10, wherein review of the database comprises at least one of access to the database, navigation of the database, at least one query run against the database, at least one search in the database, a filtering operation on the database, a ranking operation on the database, and a data mining operation on the database.

12. The method according to claim 1, wherein the act of modifying the existing structure of the database to include the organization of the database based on the descriptor, comprises an act of caching information associated with the generation of the measurement of distinctiveness.

13. The method according to claim 12, wherein the caching information comprises at least one of age of the measurement of distinctiveness, age of any of the underlying calculations used to generate the measurement of distinctiveness, intermediate computation values, partial computation values, computational expense associated with generation of the measurement of distinctiveness, resource usage to maintain the measurement of distinctiveness, and resource usage to maintain the group.

14. The method according to claim 1, further comprising an act of modifying the at least one descriptor in response to review of the database.

15. The method according to claim 1, wherein the act of modifying the existing structure of the database to include the organization of the database based on the descriptor, comprises an act of indexing the database based, at least in part, the at least one descriptor.

16. The method according to claim 1, wherein the at least one identifying characteristic comprises at least one element of review of the database.

17. The method according to claim 16, wherein the acts of generating the descriptor and organizing are repeated for subsequent review of the database.

18. The method according to claim 16, further comprising an act of generating an additional identifying characteristic based on review of the database.

19. The method according to claim 1, further comprising an act of generating at least one additional descriptor in response to review of the database.

20. The method according to claim 19, wherein the act of organizing is further based on the at least one additional descriptor.

21. The method according to claim 1, wherein the act of establishing at least one identifying characteristic from the database records and a respective at least one facet, further comprises an act of determining the at least one identifying characteristic based on review of the database by a host computer system.

22. The method according to claim 21, wherein review of the database comprises at least one of access to the database, navigation of the database, at least one query run against the database, at least one search in the database, a filtering operation on the database, a ranking operation on the database, and a data mining operation on the database.

23. The method according to claim 1, wherein the act of organizing the database includes an act of persisting the modified structure for subsequent access to the database by a plurality of host computer systems.

24. The method according to claim 23, wherein the act of persisting comprises at least one of storing the descriptor in the database records of the at least one distinctive group of database records and storing the modified structure for the subsequent access to the database by the plurality of host computer systems.

25. The method according to claim 1, wherein the at least one identifying characteristic for the set is established automatically by the computer system.

26. A non-transitory computer-readable medium having computer-readable instructions stored thereon that define instructions that, as a result of being executed by a computer, instruct the computer to perform a method for organizing a database, the method comprising the acts of:

accessing a set comprising a plurality of database records stored in a database on a computer system, wherein the plurality of database records comprise at least one facet, and wherein the database is organized based on an existing structure;

establishing at least one identifying characteristic for the set comprising the plurality of database records and a respective at least one facet;

analyzing the database to determine a statistical distribution of the at least one identifying characteristic;

generating a measurement of distinctiveness for the set, based on the statistical distribution of the at least one identifying characteristic;

identifying at least one distinctive group of database records within the database based on the measurement of distinctiveness;

generating a descriptor associated with the identified at least one distinctive group of database records; and

organizing the database based on the descriptor, wherein the act of organizing the database based on the descriptor includes modifying the existing structure of the database to include the organization of the database based on the descriptor.

27. A system for organizing a database, the system comprising:

at least one processor operatively connected to a memory for executing system components;

an access component adapted to access a set comprising a plurality of database records in a database stored in the memory, wherein the plurality of database records comprise at least one facet, and wherein the database is organized based on an existing structure;

an analysis component adapted to;

establish at least one identifying characteristic for the set comprising the plurality of database records and a respective at least one facet,

determine a statistical distribution of the at least one identifying characteristic, and

determine a measurement of distinctiveness for the set based on a the statistical distribution of at least one identifying characteristic;

a generation component adapted to generate a descriptor for at least one distinctive group of database records of the database based on the measurement of distinctiveness; and

an organization component adapted to organize the database based on the descriptor, wherein the organization component is further adapted to modify the existing structure of the database based on the descriptor.

28. The system according to claim 27, further comprising a normalization component adapted to normalize the measurement of distinctiveness.

29. The system according to claim 27, wherein the analysis component is further adapted to generate at least one subset from the database.

30. The system according to claim 29, wherein the analysis component is further adapted to manipulate a size of the generated subset.

31. The system according to claim 27, wherein the analysis component is further adapted to compare the statistical distribution of the at least one identifying characteristic against a baseline statistical distribution.

32. The system according to claim 27, wherein the organization component is further adapted to generate a hierarchy of organization based on the measurement of distinctiveness.

33. The system according to claim 32, wherein the organization component is further adapted to identify a relationship between database records of the database based, at least in part, on the statistical distribution of at least one identifying characteristic.

34. The system according to claim 33, wherein the organization component is further adapted to determine a relationship based on at least one identifying characteristic in common, and a measurement of distinctiveness.

35. The system according to claim 33, wherein the organization component is further adapted to identify at least one of a parent, child, and sibling database record within the database.

36. The system according to claim 32, wherein the organization component is further adapted to update the hierarchy of organization based on review of the database.

37. The system according to claim 36, wherein review of the database comprises at least one of access to the database, navigation of the database, at least one query run against the database, at least one search in the database, a filtering operation on the database, a ranking operation on the database, and a data mining operation on the database.

38. The system according to claim 27, further comprising a storage component adapted to cache information associated with the measurement of distinctiveness.

39. The system according to claim 38, wherein the caching information comprises at least one of age of the measurement of distinctiveness, age of any of the underlying calculations used to generate the measurement of distinctiveness, intermediate computation values, partial computation values, computational expense associated with generation of the measurement of distinctiveness, resource usage to maintain the measurement of distinctiveness, and resource usage to maintain the group.

40. The system according to claim 27, wherein the generation component is further adapted to modify the at least one descriptor in response to review of the database.

41. The system according to claim 27, wherein the organization component is further adapted to store the modified structure of the database by generating an index for the database based, at least in part, the descriptor.

42. The system according to claim 27, wherein the at least one identifying characteristic comprises at least one element of review of the database.

43. The system according to claim 42, wherein the analysis component is further adapted to generate an additional identifying characteristic based on review of the database.

44. The system according to claim 27, wherein the generation component is further adapted to generate at least one additional descriptor in response to review of the database.

45. The system according to claim 44, wherein the organization component is further adapted to organize the database based on the at least one additional descriptor.

46. The system according to claim 27, wherein the analysis component is further adapted to determine the at least one identifying characteristic based on review of the database.

47. The system according to claim 27, further comprising a storage component adapted to store the modified structure for subsequent access to the database by a plurality of host computer systems.