System and method for measuring the quality of information retrieval

• US 7,376,620 B2
• Filed: 07/23/2001
• Issued: 05/20/2008
• Est. Priority Date: 07/23/2001
• Status: Active Grant

• Current Assignee: Avolin LLC
• Original Assignee: Consona CRM Incorporated (Aptean Superior Holdings, Inc.)
• Inventors: Mohammed, Shafi, Lynch, Denis, Kay, David B., Angel, Mark, Wormington, Catherine
• Primary Examiner(s): Poinvil; Frantzy
• Assistant Examiner(s): GRAHAM, CLEMENT B
• Application Number:
  US09/911,839
• Publication Number:
  US 20030018626A1
• Time in Patent Office: 2,493 Days
• Field of Search: 707/2, 707/3, 707/5, 705/10, 705/3, 705/37, 705/34, 705/36, 705/38, 705/39, 714/4, 714/5, 709/239
• US Class Current: 705/40
• CPC Class Codes: G06Q 20/102 Bill distribution or payments
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes

Abstract

In an information retrieval application, a system and method for detecting content holes. A content body is parsed into a plurality of concepts nodes, including a first concept node. A percentage of successful service interactions is determined as a function of concept node and, if the percentage of successful service interactions at the first concept node is below a predefined threshold, a content hole is flagged.

Claims

1. A computer-assisted method for detecting content holes, comprising:

accessing a content body organized into a plurality of concept nodes to which content is selectively pre-tagged before receiving a user query as part of a service interaction,

wherein the content body includes a first concept node;

determining a percentage of successful service interactions as a function of concept nodes; and

if the percentage of successful service interactions related to the first concept node is below a predefined threshold, flagging a content hole and presenting an indication of the flagged content hole to a user or a computerized process.

8. The computer-assisted method of claim 1, wherein each concept node represents a concept in the content body.

9. The computer-assisted method of claim 1, wherein the successful service interaction comprises a query from a user for which returned content matches that user'"'"'s intent.

2. A computer-readable medium having instructions that, when executed in a computer, detects content holes-by:

accessing a content body organized into a plurality of concept nodes to which content is selectively pre-tagged before receiving;

a user query as part of a service interaction,

wherein the content body includes a first concept node;

determining a percentage of successful service interactions as a function of concept nodes; and

if the percentage of successful service interactions related to the first concept node is below a predefined threshold, flagging a content hole and presenting an indication of the flagged content hole to a user or a computerized process.

3. A computer-assisted method of charging for services, comprising:

determining a percentage of successful service interactions in a first information retrieval system;

determining a percentage of successful service interactions for services provided in a second information retrieval system; and

billing as a function of the difference between the percentage of successful service interactions in the first information retrieval system and the percentage of successful service interactions for services provided in the second information retrieval system.

4. The computer-assisted method according to claim 3, wherein determining a percentage of successful service interactions for services provided in the second information retrieval system includes:

accessing a content body organized into a plurality of concept nodes to which content is selectively pre-tagged before receiving a user query as part of a service interaction, wherein the content body includes a first concept node;

determining a percentage of successful service interactions as a function of concept nodes; and

wherein billing as a function of the difference between the percentage of successful service interactions in the first information retrieval system and the percentage of successful service interactions for services provided in the second information retrieval system includes weighting successful interactions as a function of concept nodes.

10. The computer-assisted method of claim 3, wherein the successful service interaction comprises a query from a user for which returned content matches that user'"'"'s intent.

11. The computer-assisted method of claim 4, wherein each concept node represents a concept in the content body.

12. The computer-assisted method of claim 4, wherein the successful service interaction comprises a query from a user for which returned content matches that user'"'"'s intent.

5. A computer-readable medium having instructions when executed in a computer, charges for services by:

accessing a content body organized into a plurality of concept nodes to which content is selectively pre-tagged before receiving a user query as part of a service interaction, wherein the content body includes a first concept node;

determining a percentage of successful service interactions in a first information retrieval system;

determining a percentage of successful service interactions for services provided in a second information retrieval system; and

billing as a function of the difference between the percentage of successful service interactions in the first information retrieval system and the percentage of successful service interactions for services provided in the second information retrieval system, wherein the billing includes weighting successful interactions as a function of concept nodes.

6. A computer-assisted method for detecting content holes, comprising:

(a) accessing a content body organized into a plurality of concept nodes to which content is selectively pre-tagged before receiving a user query as part of a service interaction, wherein the content body includes a first concept node;

(b) determining a percentage of successful service interactions (SSIs) as a function of concept nodes;

(c) determining a percentage of queries as a function of concept nodes;

(d) determining a percentage of documents as a function of concept nodes;

(e) computing a content hole score for the first concept node as a function of at least one of (b), (c), and (d); and

(f) flagging a content hole if the content hole score is below a predefined threshold and and presenting an indication of the flagged content hole to a user or a computerized process.

13. The computer-assisted method of claim 6, wherein each concept node represents a concept in the content body.

14. The computer-assisted method of claim 6, wherein the successful service interaction comprises a query from a user for which returned content matches that user'"'"'s intent.

16. The computer-assisted method of claim 6, wherein the acts (a)-(f) are performed in the order presented in claim 6.

7. A computer-assisted method of charging for services, comprising:

determining a percentage of successful service interactions in an information retrieval system over a period of time; and

billing as a function of the percentage of successful service interactions in the information retrieval system over the period of time.

15. The computer-assisted method of claim 7, wherein the successful service interaction comprises a query from a user for which returned content matches that user'"'"'s intent.