Analyzing the Ability to Find Textual Content

• US 20080033971A1
• Filed: 08/01/2006
• Published: 02/07/2008
• Est. Priority Date: 08/01/2006
• Status: Active Grant

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Fine, Shai, Pelleg, Dan, Carmel, David, Yom-Tov, Elad, Darlow, Adam
• Granted Patent: US 7,792,830 B2
• Time in Patent Office: Days
• US Class Current: 707/101
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence

Abstract

A method and system for analyzing a document set (202, 420) are provided. The method includes determining a set of terms (312) from the terms of the document set that minimizes a distance measurement (405) from the given set of documents (420). The method includes using a greedy algorithm to build the set of terms incrementally, at each stage finding a single word that is closest to the document set (202, 420). The set of terms is evaluated to assess the ability to find the document set (202, 420). The set of terms are compared with expected terms to evaluate the ability to find the document set (202, 420). A measure of the ability to find a document set (202, 420) is provided by computing a distance measure (403) between a document set and an entire collection.

Claims

1. A method for analyzing a document set, comprising:

providing a document set;

determining a set of terms from the terms of the document set that minimizes a distance measurement from the given document set.

2. A method as claimed in claim 1, wherein the document set and the set of terms are modelled by probability distributions of the terms in the sets and a distance measurement generated between the probability distributions.

3. A method as claimed in claim 1, wherein the terms are words, parts of words, or word combinations.

4. A method as claimed in claim 1, including using a greedy algorithm to build the set of terms incrementally, at each stage finding a single term that is closest to the document set.

5. A method as claimed in claim 1, including evaluating the set of terms to assess the ability to find the document set.

6. A method as claimed in claim 5, including comparing the set of terms with expected terms to evaluate the ability to find the document set.

7. A method as claimed in claim 1, including:

creating a sequence of queries from the set of terms;

executing the sequence of queries in a search engine;

computing the average precision for each query; and

analyzing the average precision against the number of terms.

8. A method as claimed in claim 7, wherein analyzing the average precision against the number of terms includes clustering the results into categories of behaviour.

9. A method as claimed in claim 8, wherein the categories of behaviour include:

easily findable document sets,

document sets requiring long queries to be located, and

document sets which are not findable.

10. A method as claimed in claim 1, including simulating changes to the text of the document set to improve the ability to find the document set.

11. A method as claimed in claim 1, including determining the distance measurement of a specific term to the document set.

12. A method as claimed in claim 1, including:

comparing a set of terms for a first document set with sets of terms for one or more other document sets; and

determining similar document sets which may be confusable.

13. A method as claimed in claim 11, wherein comparing is carried out by measuring the overlap of the sets of terms.

14. A method for analyzing a document set, comprising:

computing a distance measure between a document set and an entire collection; and

using the distance measure as a prediction of the ability to find the document set in the collection.

15. A method as claimed in claim 14, wherein the document set and the entire collection are modelled by probability distributions of the terms and a distance measurement generated between the probability distributions.

16. A method as claimed in claim 14, wherein the distance measurement is the Jensen-Shannon divergence.

17. A computer program product stored on a computer readable storage medium, comprising computer readable program code means for performing the steps of:

determining a set of terms from the terms of the document set that minimizes a distance measurement from the given document set.

18. A system for analyzing a document set, comprising:

a server computer providing a document set for access by client computers across a network; and

a document set analyzer for determining a set of terms from the terms of the document set that minimizes a distance measurement from the given document set.

19. A system as claimed in claim 18, including a greedy algorithm means to build the set of terms incrementally, at each stage finding a single term that is closest to the document set.

20. A system as claimed in claim 18, including means for evaluating the set of terms to assess the ability to find the document set.