Data independent relevance evaluation utilizing cognitive concept relationship

• US 20070136343A1
• Filed: 12/14/2005
• Published: 06/14/2007
• Est. Priority Date: 12/14/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Ying, Wang, Lee, Queva, Pascale
• Granted Patent: US 7,660,786 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/08 Auctions
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A measurable means to evaluate a given relevancy verification process to a human decision process is provided. For example, a cognitive concept relationship (CCR) system can be utilized to provide a relevancy verification evaluation that is independent of the relevancy quality of the test data employed to train the relevancy verification algorithm under test. This provides a means to evaluate relevancy verification of, for example, keyword/item pairs for any number of business applications with different relevance standards and/or changes in relevance standards over time, without the need to manually re-label the test data and/or re-measure the algorithmic relevancy.

Claims

1. A system that facilitates relevancy verification evaluation of data item pairs, comprising:

a receiving component that obtains an input data item pair for relevancy evaluation; and

an evaluation component that employs an evaluation system to facilitate in evaluation of a relevancy verification process and in determination of relevancy of the input data item pair independent of test data utilized to train the relevancy verification process.

2. The system of claim 1, the input data item pair associated with an advertising keyword and an advertising target and/or entity.

3. The system of claim 1, the evaluation component utilizes a cognitive concept relationship (CCR) system to facilitate in evaluating the relevancy verification process.

4. The system of claim 3, the CCR system employs human classification of test cases.

5. The system of claim 3, the evaluation component provides a measurement of algorithmic relevancy accuracy.

6. The system of claim 1 further comprising:

a business model component that utilizes the relevancy determination to adaptively determine relevance of a keyword in relation to a business model.

7. The system of claim 6 further comprising:

an advertising component that determines a business advertising decision for a keyword based on the business model relevance.

8. The system of claim 1, the evaluation component supports multiple business relevancy standards concurrently and/or supports business relevancy standards that change over time without retraining the evaluation model.

9. An advertising keyword auction system employing, at least in part, the system of claim 1.

20. A device employing the system of claim 1 comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device.

10. A method for facilitating relevancy verification evaluation of a data item pair, comprising:

obtaining an input data item pair to be evaluated for relevance; and

evaluating relevancy of the input data item pair independent of algorithmic relevance verification training data via employing an evaluation model that utilizes a cognitive concept relationship (CCR) system.

11. The method of claim 10, the input data item pair associated with an advertising keyword and an advertising entity and/or target.

12. The method of claim 10 further comprising:

employing test cases for the CCR system that are classified by humans;

breaking set boundaries of data in the test cases and enabling test data independent relevance evaluation.

13. The method of claim 10 further comprising:

utilizing the CCR system to measure algorithmic accuracy for relevance.

14. The method of claim 10 further comprising:

mapping algorithmic outputs to a business standard directly to support business applications with different business relevance standards, changes in a business relevance standard over time, and/or concurrent multiple business relevance standards.

15. An online Internet auction method that employs, at least in part, the method of claim 10 to facilitate in auctioning keywords automatically.

19. A device employing the method of claim 10 comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device.

16. A method for evaluating advertising keyword auctions, comprising:

obtaining a keyword desired by a business entity to be associated with its advertising; and

utilizing an adaptive business relevancy decision making process to evaluate if the keyword is relevant to the business.

17. The method of claim 16 further comprising:

determining an acceptance of a bid by a business based on the relevancy of the keyword to the business.

18. The method of claim 16, the adaptive business relevancy decision making process is based on, in part, a cognitive concept relationship (CCR) technique.